https://github.com/objectgyan/mdmfileprocessor
This project utilizes an Azure Function App to process files and store them in a central repository.
https://github.com/objectgyan/mdmfileprocessor
Last synced: 11 months ago
JSON representation
This project utilizes an Azure Function App to process files and store them in a central repository.
- Host: GitHub
- URL: https://github.com/objectgyan/mdmfileprocessor
- Owner: objectgyan
- Created: 2025-02-21T13:47:51.000Z (over 1 year ago)
- Default Branch: master
- Last Pushed: 2025-02-21T14:16:17.000Z (over 1 year ago)
- Last Synced: 2025-02-21T15:24:35.295Z (over 1 year ago)
- Language: C#
- Size: 19.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# File Processor - Device Management System
A Azure Durable Functions-based solution for processing large CSV files containing device information.
## Features
- Chunk-based processing of large CSV files
- Validation of device records
- Failed record handling and storage
- Automatic database upsert operations
- Scalable and resilient architecture
## System Components
### Activities
- **ChunkProcessingActivity**: Processes individual chunks of the CSV file
- Validates device records
- Stores valid records in SQL Database
- Tracks failed records separately
## Architecture Overview
The solution uses Azure Durable Functions to process large MDM data files in chunks, implementing fan-out/fan-in pattern for parallel processing.
### Key Components
1. **Trigger**: Blob trigger that activates when new MDM data files are uploaded
2. **Orchestrator**: Coordinates the processing of MDM device data chunks
3. **Activities**:
- Device Data Chunk Processing
- Summary Storage
- Notification Service
### Flow Sequence
1. MDM data file uploaded to Azure Blob Storage
2. BlobTrigger initiates processing
3. Orchestrator splits file into manageable chunks
4. Parallel processing of device data chunks
5. Results consolidation and summary generation
6. Notifications sent to stakeholders
### Data Processing Rules
- Validates device data against business rules
- Stores valid records in SQL Database
- Tracks failed records separately
- Generates processing summaries
- Sends notifications via multiple channels
## Configuration
### Required App Settings
```json
{
"AzureWebJobsStorage": "",
"SqlConnectionString": "",
"SendGridKey": "",
"TeamsWebhookUrl": "",
"ChunkSize": "2000"
}
```
### Database Schema
```sql
CREATE TABLE DeviceInventory (
DeviceId INT PRIMARY KEY,
DeviceName NVARCHAR(100),
SerialNumber NVARCHAR(100),
OSVersion NVARCHAR(50),
LastCheckIn DATETIME,
ComplianceStatus NVARCHAR(50)
)
```
## Monitoring & Troubleshooting
- Processing summaries available via REST API
- Failed records tracking in Azure Table Storage
- Application Insights integration for monitoring
- Status notifications via email and Teams
## Sequence Diagram
```mermaid
sequenceDiagram
participant Upload as MDM Data Upload
participant Trigger as Blob Trigger
participant Orch as MDM Orchestrator
participant Activity as Processing Activities
participant DB as SQL Database
participant Notify as Notification Service
Upload->>Trigger: Upload MDM Data File
Trigger->>Orch: Start Processing
Orch->>Orch: Split into Chunks
loop Parallel Processing
Orch->>Activity: Process Device Chunk
Activity->>DB: Store Valid Records
Activity->>Activity: Log Invalid Records
Activity->>Orch: Return Results
end
Orch->>DB: Store Summary
Orch->>Notify: Send Notifications
Notify-->>Upload: Processing Complete
```
## Error Handling
- Retry policies for transient failures
- Dead letter queue for failed messages
- Detailed error logging in Application Insights
- Error notifications to IT team
## Security
- Azure AD integration
- Role-based access control
- Data encryption in transit and at rest
- Secure connection strings and keys in KeyVault