{"id":22087958,"url":"https://github.com/bayoadejare/pipeline-edtech","last_synced_at":"2026-04-15T14:36:47.070Z","repository":{"id":265781811,"uuid":"861634757","full_name":"BayoAdejare/pipeline-edtech","owner":"BayoAdejare","description":"Edtech ADF Pipeline Project","archived":false,"fork":false,"pushed_at":"2024-10-09T00:24:15.000Z","size":13,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-19T20:06:53.018Z","etag":null,"topics":["adf","azure","azure-sql","contextual-analysis","data-engineer","data-engineering-pipeline","data-warehouse","database","databricks","machine-learning","pandas","power-bi","powerbi","python3","synapse"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BayoAdejare.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-23T09:00:27.000Z","updated_at":"2024-10-09T05:46:03.000Z","dependencies_parsed_at":"2024-11-30T23:22:18.112Z","dependency_job_id":null,"html_url":"https://github.com/BayoAdejare/pipeline-edtech","commit_stats":null,"previous_names":["bayoadejare/pipeline-edtech"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Fpipeline-edtech","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Fpipeline-edtech/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Fpipeline-edtech/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BayoAdejare%2Fpipeline-edtech/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BayoAdejare","download_url":"https://codeload.github.com/BayoAdejare/pipeline-edtech/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245179484,"owners_count":20573474,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adf","azure","azure-sql","contextual-analysis","data-engineer","data-engineering-pipeline","data-warehouse","database","databricks","machine-learning","pandas","power-bi","powerbi","python3","synapse"],"created_at":"2024-12-01T02:07:04.477Z","updated_at":"2026-04-15T14:36:42.007Z","avatar_url":"https://github.com/BayoAdejare.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# EdTech Azure Data Factory Pipeline\n\nWelcome to the EdTech Azure Data Factory Pipeline project! This advanced system processes and analyzes educational data from multiple sources to provide comprehensive insights into student performance, content effectiveness, and learning patterns using Azure Data Factory and related Azure services.\n\n## Table of Contents\n- [Project Overview](#project-overview)\n- [Data Sources](#data-sources)\n- [Azure Architecture](#azure-architecture)\n- [Project Structure](#project-structure)\n- [Setup and Configuration](#setup-and-configuration)\n- [Usage](#usage)\n- [Example: Student Performance Analysis](#example-student-performance-analysis)\n- [Example: Content Effectiveness Evaluation](#example-content-effectiveness-evaluation)\n- [Example: Educational Research Integration](#example-educational-research-integration)\n- [CI/CD with Azure DevOps](#cicd-with-azure-devops)\n- [License](#license)\n\n## Project Overview\n\nOur EdTech Azure Data Factory Pipeline is designed to handle large-scale educational data processing from various sources. It includes data ingestion from internal systems and external educational datasets, processing, analysis, and visualization components to enhance learning experiences and provide valuable insights for educators and administrators.\n\nKey features:\n- Integration with Learning Management Systems (LMS) and Student Information Systems (SIS)\n- Integration with high-quality educational research databases and public datasets\n- Real-time student activity tracking and processing\n- Scalable data processing using Azure Data Factory and Azure Databricks\n- Machine learning models for personalized learning path recommendations\n- Student performance analytics and early intervention systems\n- Content effectiveness analysis and improvement suggestions\n- Integration with Azure Cognitive Services for natural language processing of student feedback\n\n## Data Sources\n\n### Internal Data Sources\n1. **Learning Management Systems**\n   - Canvas LMS API\n   - Moodle Web Services\n   - Blackboard Learn REST API\n\n2. **Student Information Systems**\n   - PowerSchool API\n   - Ellucian Banner API\n\n### External Data Sources\n1. **Educational Statistics**\n   - **National Center for Education Statistics (NCES)**: Comprehensive education data\n     - API: [NCES RestAPI](https://nces.ed.gov/developer)\n     - Datasets: Enrollment, achievement, demographics\n     - Use for: Benchmarking and contextual analysis\n\n2. **Academic Research**\n   - **Education Resources Information Center (ERIC)**\n     - API Documentation: [ERIC API](https://eric.ed.gov/?api)\n     - Content: Research papers, teaching methodologies\n     - Use for: Evidence-based teaching strategies\n\n3. **Open Educational Resources**\n   - **OER Commons API**: Access to open educational resources\n     - API Documentation: [OER Commons API](https://www.oercommons.org/api-docs)\n     - Use for: Supplementary content recommendations\n\n4. **Cognitive Skills Research**\n   - **NIH Cognitive Atlas**: Standardized cognitive concepts\n     - API: [Cognitive Atlas API](https://www.cognitiveatlas.org/api/)\n     - Use for: Aligning content with cognitive development stages\n\n5. **Labor Market Data**\n   - **O*NET Web Services**: Occupational information network\n     - API Documentation: [O*NET API](https://services.onetcenter.org/reference)\n     - Use for: Career pathway alignment and guidance\n\n### Data Integration Examples\n\n```python\n# Example: Integrating NCES data for contextual analysis\nfrom nces_api import NCESClient\nimport pandas as pd\n\ndef enrich_student_data_with_nces():\n    nces_client = NCESClient(api_key=os.environ[\"NCES_API_KEY\"])\n    \n    # Fetch national achievement data\n    national_data = nces_client.get_achievement_data(\n        subject=\"mathematics\",\n        grade_level=\"8th\",\n        year=\"2024\"\n    )\n    \n    # Convert to Spark DataFrame\n    national_df = spark.createDataFrame(pd.DataFrame(national_data))\n    \n    # Read local student performance data\n    local_df = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/student_performance/\")\n    \n    # Perform comparative analysis\n    comparison = local_df.join(\n        national_df,\n        [\"subject\", \"grade_level\"]\n    ).select(\n        \"subject\",\n        local_df.avg_score.alias(\"local_avg\"),\n        national_df.avg_score.alias(\"national_avg\")\n    )\n    \n    return comparison\n\n# Example: Integrating ERIC research for content enhancement\ndef enhance_content_with_research():\n    eric_client = ERICClient(api_key=os.environ[\"ERIC_API_KEY\"])\n    \n    # Fetch relevant research papers\n    research_data = eric_client.search(\n        keywords=[\"active learning\", \"student engagement\"],\n        publication_date_gte=\"2023-01-01\"\n    )\n    \n    # Extract teaching methodologies\n    methodologies = extract_methodologies(research_data)\n    \n    # Enhance content recommendations\n    enhanced_recommendations = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/content_recommendations/\") \\\n        .join(\n            spark.createDataFrame(methodologies),\n            \"subject\"\n        )\n    \n    return enhanced_recommendations\n\ndef extract_methodologies(research_data):\n    # Use Azure Cognitive Services to extract teaching methodologies\n    text_analytics_client = TextAnalyticsClient(\n        endpoint=os.environ[\"COGNITIVE_SERVICES_ENDPOINT\"],\n        credential=os.environ[\"COGNITIVE_SERVICES_KEY\"]\n    )\n    \n    methodologies = []\n    for paper in research_data:\n        key_phrases = text_analytics_client.extract_key_phrases([paper.abstract])[0]\n        methodologies.extend(key_phrases)\n    \n    return methodologies\n```\n\n[Rest of the original sections remain unchanged: Azure Architecture, Project Structure, Setup and Configuration, Usage, original examples, and CI/CD with Azure DevOps]\n\n## Example: Educational Research Integration\n\nHere's an example of how to integrate educational research data to enhance content recommendations:\n\n```python\n# In Azure Databricks notebook\n\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col, explode\nfrom azure.ai.textanalytics import TextAnalyticsClient\nfrom azure.core.credentials import AzureKeyCredential\n\ndef integrate_research_insights():\n    # Initialize clients\n    spark = SparkSession.builder.appName(\"ResearchIntegration\").getOrCreate()\n    text_analytics_client = TextAnalyticsClient(\n        endpoint=os.environ[\"COGNITIVE_SERVICES_ENDPOINT\"],\n        credential=AzureKeyCredential(os.environ[\"COGNITIVE_SERVICES_KEY\"])\n    )\n    eric_client = ERICClient(api_key=os.environ[\"ERIC_API_KEY\"])\n\n    # Read current content data\n    content_df = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/course_content/\")\n\n    # For each subject area, fetch and analyze relevant research\n    for subject in content_df.select(\"subject\").distinct().collect():\n        # Fetch related research papers\n        papers = eric_client.search(\n            keywords=[subject.subject],\n            publication_date_gte=\"2023-01-01\"\n        )\n        \n        # Extract key insights using Azure Cognitive Services\n        research_insights = []\n        for paper in papers:\n            response = text_analytics_client.extract_key_phrases([paper.abstract])[0]\n            research_insights.extend(response.key_phrases)\n        \n        # Create DataFrame with research insights\n        research_df = spark.createDataFrame(\n            [(subject.subject, insight) for insight in research_insights],\n            [\"subject\", \"research_insight\"]\n        )\n        \n        # Join with content data and save enriched content\n        enriched_content = content_df \\\n            .join(research_df, \"subject\") \\\n            .groupBy(\"content_id\", \"subject\", \"title\") \\\n            .agg(collect_list(\"research_insight\").alias(\"research_insights\"))\n        \n        # Save enriched content\n        enriched_content.write \\\n            .mode(\"overwrite\") \\\n            .parquet(f\"abfss://processed-data@yourdatalake.dfs.core.windows.net/enriched_content/{subject.subject}\")\n\n# Execute the integration\nintegrate_research_insights()\n```\n\nThis example demonstrates how to:\n1. Fetch relevant research papers from ERIC based on subject areas\n2. Extract key insights using Azure Cognitive Services\n3. Enrich existing course content with research-backed insights\n4. Save the enriched content for use in recommendations and content development\n\n## Azure Architecture\n\nOur pipeline utilizes the following Azure services:\n\n- Azure Data Factory: Orchestrates and automates the data movement and transformation\n- Azure Blob Storage: Stores raw and processed data\n- Azure Databricks: Performs complex data processing and runs machine learning models\n- Azure SQL Database: Stores structured data and analysis results\n- Azure Analysis Services: Creates semantic models for reporting\n- Power BI: Provides interactive dashboards and reports\n- Azure Key Vault: Securely stores secrets and access keys\n- Azure Monitor: Monitors pipeline performance and health\n\n## Project Structure\n\n```\nedtech-azure-pipeline/\n│\n├── adf/\n│   ├── pipeline/\n│   │   ├── ingest_lms_data.json\n│   │   ├── process_student_performance.json\n│   │   └── analyze_content_effectiveness.json\n│   ├── dataset/\n│   │   ├── lms_data.json\n│   │   ├── sis_data.json\n│   │   └── processed_data.json\n│   └── linkedService/\n│       ├── AzureBlobStorage.json\n│       ├── AzureDataLakeStorage.json\n│       └── AzureDatabricks.json\n│\n├── databricks/\n│   ├── notebooks/\n│   │   ├── student_performance_analysis.py\n│   │   ├── content_effectiveness_evaluation.py\n│   │   └── learning_path_recommendation.py\n│   └── libraries/\n│       └── education_utils.py\n│\n├── sql/\n│   ├── schema/\n│   │   ├── student_performance.sql\n│   │   └── content_metrics.sql\n│   └── stored_procedures/\n│       ├── calculate_student_progress.sql\n│       └── evaluate_content_engagement.sql\n│\n├── power_bi/\n│   ├── StudentPerformanceDashboard.pbix\n│   └── ContentEffectivenessReport.pbix\n│\n├── tests/\n│   ├── unit/\n│   └── integration/\n│\n├── scripts/\n│   ├── setup_azure_resources.sh\n│   └── deploy_adf_pipelines.sh\n│\n├── .azure-pipelines/\n│   ├── ci-pipeline.yml\n│   └── cd-pipeline.yml\n│\n├── requirements.txt\n├── .gitignore\n└── README.md\n```\n\n## Setup and Configuration\n\n1. Clone the repository:\n   ```\n   git clone https://github.com/your-org/edtech-azure-pipeline.git\n   cd edtech-azure-pipeline\n   ```\n\n2. Set up Azure resources:\n   ```\n   ./scripts/setup_azure_resources.sh\n   ```\n\n3. Configure Azure Data Factory pipelines:\n   ```\n   ./scripts/deploy_adf_pipelines.sh\n   ```\n\n4. Set up Azure Databricks workspace and upload notebooks from the `databricks/notebooks/` directory.\n\n5. Create Azure SQL Database schema and stored procedures using scripts in the `sql/` directory.\n\n6. Import Power BI reports from the `power_bi/` directory and configure data sources.\n\n## Usage\n\n1. Monitor and manage Azure Data Factory pipelines through the Azure portal or using Azure Data Factory SDK.\n\n2. Schedule pipeline runs or trigger them manually based on your requirements.\n\n3. Access Databricks notebooks for custom analysis and model training.\n\n4. View reports and dashboards in Power BI for insights into student performance and content effectiveness.\n\n## Example: Student Performance Analysis\n\nHere's an example of how to use Azure Databricks to analyze student performance:\n\n```python\n# In Azure Databricks notebook\n\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import avg, count\n\n# Initialize Spark session\nspark = SparkSession.builder.appName(\"StudentPerformanceAnalysis\").getOrCreate()\n\n# Read student performance data from Azure Data Lake\nperformance_data = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/student_performance/\")\n\n# Calculate average scores by subject\navg_scores = performance_data.groupBy(\"subject\").agg(\n    avg(\"score\").alias(\"average_score\"),\n    count(\"student_id\").alias(\"student_count\")\n)\n\n# Identify subjects that need attention (average score \u003c 70)\nsubjects_needing_attention = avg_scores.filter(avg_scores.average_score \u003c 70)\n\n# Display results\nsubjects_needing_attention.show()\n\n# Write results back to Azure SQL Database\nsubjects_needing_attention.write \\\n    .format(\"jdbc\") \\\n    .option(\"url\", \"jdbc:sqlserver://yourserver.database.windows.net:1433;database=yourdatabase\") \\\n    .option(\"dbtable\", \"subjects_needing_attention\") \\\n    .option(\"user\", \"yourusername\") \\\n    .option(\"password\", \"yourpassword\") \\\n    .mode(\"overwrite\") \\\n    .save()\n```\n\nThis example demonstrates how to:\n1. Read processed student performance data from Azure Data Lake\n2. Calculate average scores by subject\n3. Identify subjects that need attention based on average scores\n4. Write the results back to Azure SQL Database for reporting\n\n## Example: Content Effectiveness Evaluation\n\nHere's an example of how to evaluate content effectiveness using Azure Data Factory and Azure Databricks:\n\n```python\n# In Azure Databricks notebook\n\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import col, datediff, avg\n\n# Initialize Spark session\nspark = SparkSession.builder.appName(\"ContentEffectivenessEvaluation\").getOrCreate()\n\n# Read content interaction data and assessment results\ncontent_data = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/content_interactions/\")\nassessment_data = spark.read.parquet(\"abfss://processed-data@yourdatalake.dfs.core.windows.net/assessment_results/\")\n\n# Join content interaction data with assessment results\ncombined_data = content_data.join(assessment_data, \"student_id\")\n\n# Calculate content effectiveness metrics\neffectiveness_metrics = combined_data.groupBy(\"content_id\").agg(\n    avg(\"time_spent\").alias(\"avg_time_spent\"),\n    avg(\"assessment_score\").alias(\"avg_assessment_score\"),\n    avg(datediff(col(\"assessment_date\"), col(\"interaction_date\"))).alias(\"avg_days_to_assessment\")\n)\n\n# Identify highly effective content (high assessment scores, reasonable time spent)\nhighly_effective_content = effectiveness_metrics.filter(\n    (effectiveness_metrics.avg_assessment_score \u003e 80) \u0026\n    (effectiveness_metrics.avg_time_spent \u003c 60)  # Assuming time spent is in minutes\n)\n\n# Display results\nhighly_effective_content.show()\n\n# Write results to Azure SQL Database\nhighly_effective_content.write \\\n    .format(\"jdbc\") \\\n    .option(\"url\", \"jdbc:sqlserver://yourserver.database.windows.net:1433;database=yourdatabase\") \\\n    .option(\"dbtable\", \"highly_effective_content\") \\\n    .option(\"user\", \"yourusername\") \\\n    .option(\"password\", \"yourpassword\") \\\n    .mode(\"overwrite\") \\\n    .save()\n```\n\nThis example shows how to:\n1. Read content interaction data and assessment results from Azure Data Lake\n2. Join and analyze the data to calculate content effectiveness metrics\n3. Identify highly effective content based on assessment scores and time spent\n4. Write the results to Azure SQL Database for further analysis and reporting\n\n## CI/CD with Azure DevOps\n\nWe use Azure DevOps for continuous integration and deployment. Our pipeline includes:\n\n1. **Continuous Integration (CI)**\n   - Triggered on every push and pull request to the `main` branch\n   - Validates Azure Data Factory pipeline definitions\n   - Runs unit tests for Databricks notebooks and custom modules\n   - Lints SQL scripts and validates database objects\n\n2. **Continuous Deployment (CD)**\n   - Triggered on successful merges to the `main` branch\n   - Deploys Azure Data Factory pipelines to a staging environment\n   - Runs integration tests\n   - Upon approval, deploys to the production environment\n\nTo view and modify these pipelines, check the `.azure-pipelines/` directory.\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayoadejare%2Fpipeline-edtech","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbayoadejare%2Fpipeline-edtech","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbayoadejare%2Fpipeline-edtech/lists"}