{"id":30637656,"url":"https://github.com/objectgyan/mdmfileprocessor","last_synced_at":"2025-08-30T23:07:23.812Z","repository":{"id":278759960,"uuid":"936675137","full_name":"objectgyan/MDMFileProcessor","owner":"objectgyan","description":"This project utilizes an Azure Function App to process files and store them in a central repository.","archived":false,"fork":false,"pushed_at":"2025-02-21T14:16:17.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-21T15:24:35.295Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/objectgyan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-21T13:47:51.000Z","updated_at":"2025-02-21T14:16:23.000Z","dependencies_parsed_at":"2025-02-21T15:34:50.873Z","dependency_job_id":null,"html_url":"https://github.com/objectgyan/MDMFileProcessor","commit_stats":null,"previous_names":["objectgyan/mdmfileprocessor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/objectgyan/MDMFileProcessor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/objectgyan%2FMDMFileProcessor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/objectgyan%2FMDMFileProcessor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/objectgyan%2FMDMFileProcessor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/objectgyan%2FMDMFileProcessor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/objectgyan","download_url":"https://codeload.github.com/objectgyan/MDMFileProcessor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/objectgyan%2FMDMFileProcessor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272917735,"owners_count":25014935,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-30T23:07:23.198Z","updated_at":"2025-08-30T23:07:23.806Z","avatar_url":"https://github.com/objectgyan.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# File Processor - Device Management System\n\nA Azure Durable Functions-based solution for processing large CSV files containing device information.\n\n## Features\n\n- Chunk-based processing of large CSV files\n- Validation of device records\n- Failed record handling and storage\n- Automatic database upsert operations\n- Scalable and resilient architecture\n\n## System Components\n\n### Activities\n\n- **ChunkProcessingActivity**: Processes individual chunks of the CSV file\n  - Validates device records\n  - Stores valid records in SQL Database\n  - Tracks failed records separately\n\n## Architecture Overview\n\nThe solution uses Azure Durable Functions to process large MDM data files in chunks, implementing fan-out/fan-in pattern for parallel processing.\n\n### Key Components\n\n1. **Trigger**: Blob trigger that activates when new MDM data files are uploaded\n2. **Orchestrator**: Coordinates the processing of MDM device data chunks\n3. **Activities**: \n   - Device Data Chunk Processing\n   - Summary Storage\n   - Notification Service\n\n### Flow Sequence\n\n1. MDM data file uploaded to Azure Blob Storage\n2. BlobTrigger initiates processing\n3. Orchestrator splits file into manageable chunks\n4. Parallel processing of device data chunks\n5. Results consolidation and summary generation\n6. Notifications sent to stakeholders\n\n### Data Processing Rules\n\n- Validates device data against business rules\n- Stores valid records in SQL Database\n- Tracks failed records separately\n- Generates processing summaries\n- Sends notifications via multiple channels\n\n## Configuration\n\n### Required App Settings\n```json\n{\n  \"AzureWebJobsStorage\": \"\u003cstorage-connection-string\u003e\",\n  \"SqlConnectionString\": \"\u003csql-connection-string\u003e\",\n  \"SendGridKey\": \"\u003csendgrid-api-key\u003e\",\n  \"TeamsWebhookUrl\": \"\u003cteams-webhook-url\u003e\",\n  \"ChunkSize\": \"2000\"\n}\n```\n\n### Database Schema\n```sql\nCREATE TABLE DeviceInventory (\n    DeviceId INT PRIMARY KEY,\n    DeviceName NVARCHAR(100),\n    SerialNumber NVARCHAR(100),\n    OSVersion NVARCHAR(50),\n    LastCheckIn DATETIME,\n    ComplianceStatus NVARCHAR(50)\n)\n```\n\n## Monitoring \u0026 Troubleshooting\n\n- Processing summaries available via REST API\n- Failed records tracking in Azure Table Storage\n- Application Insights integration for monitoring\n- Status notifications via email and Teams\n\n## Sequence Diagram\n\n```mermaid\nsequenceDiagram\n    participant Upload as MDM Data Upload\n    participant Trigger as Blob Trigger\n    participant Orch as MDM Orchestrator\n    participant Activity as Processing Activities\n    participant DB as SQL Database\n    participant Notify as Notification Service\n\n    Upload-\u003e\u003eTrigger: Upload MDM Data File\n    Trigger-\u003e\u003eOrch: Start Processing\n    Orch-\u003e\u003eOrch: Split into Chunks\n    loop Parallel Processing\n        Orch-\u003e\u003eActivity: Process Device Chunk\n        Activity-\u003e\u003eDB: Store Valid Records\n        Activity-\u003e\u003eActivity: Log Invalid Records\n        Activity-\u003e\u003eOrch: Return Results\n    end\n    Orch-\u003e\u003eDB: Store Summary\n    Orch-\u003e\u003eNotify: Send Notifications\n    Notify--\u003e\u003eUpload: Processing Complete\n```\n\n## Error Handling\n\n- Retry policies for transient failures\n- Dead letter queue for failed messages\n- Detailed error logging in Application Insights\n- Error notifications to IT team\n\n## Security\n\n- Azure AD integration\n- Role-based access control\n- Data encryption in transit and at rest\n- Secure connection strings and keys in KeyVault\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobjectgyan%2Fmdmfileprocessor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fobjectgyan%2Fmdmfileprocessor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fobjectgyan%2Fmdmfileprocessor/lists"}