{"id":15035581,"url":"https://github.com/aqiftekhar/adfpipelinecognitivesearch","last_synced_at":"2026-03-17T12:44:28.314Z","repository":{"id":245857889,"uuid":"818825130","full_name":"aqiftekhar/ADFPipelineCognitiveSearch","owner":"aqiftekhar","description":"Implemented Azure AI Search, Azure Data Factory Pipelines and Change Data Capture using C# and Azure.","archived":false,"fork":false,"pushed_at":"2024-07-23T19:21:15.000Z","size":38,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-13T12:14:43.982Z","etag":null,"topics":["azure","azure-ai-search","azure-ai-services","azure-blob-container","azure-blob-storage","azure-data-factory","azure-sql-database","change-data-capture","csharp","csharp-code"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aqiftekhar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-23T01:31:53.000Z","updated_at":"2024-07-23T19:21:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"8d91d2ba-ac52-4f86-b3b7-8b0930994e32","html_url":"https://github.com/aqiftekhar/ADFPipelineCognitiveSearch","commit_stats":null,"previous_names":["aqiftekhar/adfpipelinecognitivesearch"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqiftekhar%2FADFPipelineCognitiveSearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqiftekhar%2FADFPipelineCognitiveSearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqiftekhar%2FADFPipelineCognitiveSearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aqiftekhar%2FADFPipelineCognitiveSearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aqiftekhar","download_url":"https://codeload.github.com/aqiftekhar/ADFPipelineCognitiveSearch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243401510,"owners_count":20285058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-ai-search","azure-ai-services","azure-blob-container","azure-blob-storage","azure-data-factory","azure-sql-database","change-data-capture","csharp","csharp-code"],"created_at":"2024-09-24T20:28:58.373Z","updated_at":"2025-12-27T16:03:21.764Z","avatar_url":"https://github.com/aqiftekhar.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Azure AI Search \u0026 Data Factories Integration\n\n## Description\nFollowing tasks are performed in this project:\n- Integration of Azure Cognitive Search to Search and filter Queries.\n- Integration of Azure Data Factory.\n- Trigger ETL pipeline to insert Products using Stored Procedure from C# Code.\n- Implemented Change Data Capture in C#.\n- For Change Data Capture, Copy recently added changes from Products table in Azure Database.\n- For Change Data Capture, Write recently added Products to Destination Data Source (For Demo, I am using JSON File as Destination).\n- Managing and securing Azure resources using environment variables in .env file.\n- Building and running the application using .NET 7.\n\n## Project Configurations\nFollowing are the steps to configure and run the project.\n\n## Azure Data Factory Pipeline\nFollowing is the Azure Data Factory:\n- Pipeline that Wait for 5 seconds.\n- Execute Lookup Activity to Load the data from Excel file.\n- ForEach Activity with nested Stored Procedure to insert Products\n\n![ADF Pipeline](./ecommerceAPP/images/adf_pipeline.png \"ADF Pipeline\")\n\n### Clone the repository\n```bash\ngit clone https://github.com/aqiftekhar/ADFPipelineCognitiveSearch.git\ncd ADFPipelineCognitiveSearch\n```\n\n### Open in VSCode\n```bash\ncode .\n```\n\n### Restore NuGet Packages\n```bash\ndotnet restore\n```\n\n### Add `.env` file with the following environment configurations\n```env\n# Azure configurations\nSUBSCRIPTION_ID=\nAZURE_CLIENT_ID=\nAZURE_TENANT_ID=\nAZURE_CLIENT_SECRET=\nRESOURCE_GROUP_NAME=\n\n# DB connection string\nCONNECTION_STRING=\n\n# Azure AI Search Configurations\nSEARCH_SERVICE_NAME=\nSEARCH_SERVICE_API_KEY=\nSEARCH_SERVICE_URL=\nSEARCH_INDEX_NAME=\n\n# Azure Data Factory Configurations\nDATA_FACTORY_NAME=\nPIPELINE_NAME=\n\n# Azure Blob Storage Configurations\nBLOB_STORAGE_CONNECTION_STRING=\nCONTAINER_NAME=\nCDC_CONTAINER_NAME=\n```\n\n### Build Application using VSCode\n```bash\ndotnet build\n```\n\n### Run Application using VSCode\n```bash\ndotnet run\n```\n\n## Features\nEnter a command:\n1. Search products\n2. Run ETL pipeline\n3. Capture changes\n4. Exit\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqiftekhar%2Fadfpipelinecognitivesearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faqiftekhar%2Fadfpipelinecognitivesearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faqiftekhar%2Fadfpipelinecognitivesearch/lists"}