Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/aqiftekhar/adfpipelinecognitivesearch
Implemented Azure AI Search, Azure Data Factory Pipelines and Change Data Capture using C# and Azure.
https://github.com/aqiftekhar/adfpipelinecognitivesearch
azure azure-ai-search azure-ai-services azure-blob-container azure-blob-storage azure-data-factory azure-sql-database change-data-capture csharp csharp-code
Last synced: 27 days ago
JSON representation
Implemented Azure AI Search, Azure Data Factory Pipelines and Change Data Capture using C# and Azure.
- Host: GitHub
- URL: https://github.com/aqiftekhar/adfpipelinecognitivesearch
- Owner: aqiftekhar
- Created: 2024-06-23T01:31:53.000Z (5 months ago)
- Default Branch: main
- Last Pushed: 2024-07-23T19:21:15.000Z (4 months ago)
- Last Synced: 2024-10-12T14:20:16.680Z (27 days ago)
- Topics: azure, azure-ai-search, azure-ai-services, azure-blob-container, azure-blob-storage, azure-data-factory, azure-sql-database, change-data-capture, csharp, csharp-code
- Language: C#
- Homepage:
- Size: 37.1 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Azure AI Search & Data Factories Integration
## Description
Following tasks are performed in this project:
- Integration of Azure Cognitive Search to Search and filter Queries.
- Integration of Azure Data Factory.
- Trigger ETL pipeline to insert Products using Stored Procedure from C# Code.
- Implemented Change Data Capture in C#.
- For Change Data Capture, Copy recently added changes from Products table in Azure Database.
- For Change Data Capture, Write recently added Products to Destination Data Source (For Demo, I am using JSON File as Destination).
- Managing and securing Azure resources using environment variables in .env file.
- Building and running the application using .NET 7.
## Project Configurations
Following are the steps to configure and run the project.
## Azure Data Factory Pipeline
Following is the Azure Data Factory:
- Pipeline that Wait for 5 seconds.
- Execute Lookup Activity to Load the data from Excel file.
- ForEach Activity with nested Stored Procedure to insert Products
### Clone the repository
```bash
git clone https://github.com/aqiftekhar/ADFPipelineCognitiveSearch.git
cd ADFPipelineCognitiveSearch
```
### Open in VSCode
```bash
code .
```
### Restore NuGet Packages
```bash
dotnet restore
```
### Add `.env` file with the following environment configurations
```env
# Azure configurations
SUBSCRIPTION_ID=
AZURE_CLIENT_ID=
AZURE_TENANT_ID=
AZURE_CLIENT_SECRET=
RESOURCE_GROUP_NAME=
# DB connection string
CONNECTION_STRING=
# Azure AI Search Configurations
SEARCH_SERVICE_NAME=
SEARCH_SERVICE_API_KEY=
SEARCH_SERVICE_URL=
SEARCH_INDEX_NAME=
# Azure Data Factory Configurations
DATA_FACTORY_NAME=
PIPELINE_NAME=
# Azure Blob Storage Configurations
BLOB_STORAGE_CONNECTION_STRING=
CONTAINER_NAME=
CDC_CONTAINER_NAME=
```
### Build Application using VSCode
```bash
dotnet build
```
### Run Application using VSCode
```bash
dotnet run
```
## Features
Enter a command:
1. Search products
2. Run ETL pipeline
3. Capture changes
4. Exit