{"id":49769226,"url":"https://github.com/gabegi/dotnet-api-logging-opentelemetry-elasticsearch","last_synced_at":"2026-05-11T12:00:49.499Z","repository":{"id":326372152,"uuid":"1104641979","full_name":"Gabegi/dotnet-api-logging-opentelemetry-elasticsearch","owner":"Gabegi","description":"Best practices for logging in .NET API","archived":false,"fork":false,"pushed_at":"2025-12-08T18:40:44.000Z","size":129,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-11T11:59:44.067Z","etag":null,"topics":["elasticsearch","logging","opentelemetry"],"latest_commit_sha":null,"homepage":"https://medium.com/@codebob75/production-ready-net-api-logging-serilog-elasticsearch-opentelemetry-817a53c98cbe","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Gabegi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-26T13:40:52.000Z","updated_at":"2026-04-09T17:35:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/Gabegi/dotnet-api-logging-opentelemetry-elasticsearch","commit_stats":null,"previous_names":["gabegi/.net_api_logging","gabegi/dotnet-api-logging-opentelemetry-elasticsearch"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Gabegi/dotnet-api-logging-opentelemetry-elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Gabegi","download_url":"https://codeload.github.com/Gabegi/dotnet-api-logging-opentelemetry-elasticsearch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Gabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32894003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-10T13:40:02.631Z","status":"online","status_checked_at":"2026-05-11T02:00:05.975Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","logging","opentelemetry"],"created_at":"2026-05-11T12:00:17.391Z","updated_at":"2026-05-11T12:00:49.490Z","avatar_url":"https://github.com/Gabegi.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# .NET 9 Production Logging Solution\n\nProduction-ready .NET 9 API with structured logging, PII masking, distributed tracing, and Elasticsearch integration.\n\n## Features\n\n- ✅ **Structured Logging** - JSON format with Serilog\n- ✅ **PII Masking** - GDPR/PCI-DSS compliant (emails, credit cards, etc.)\n- ✅ **Distributed Tracing** - OpenTelemetry with correlation IDs\n- ✅ **Source Generators** - High-performance logging (3x faster)\n- ✅ **Multiple Sinks** - Console, File (rolling), Elasticsearch\n- ✅ **Smart Filtering** - Health checks hidden, errors auto-elevated\n- ✅ **Enrichment** - Client IP, User Agent, Machine, Thread, Environment\n\n## Quick Start\n\n### Prerequisites\n- .NET 9 SDK\n- Docker \u0026 Docker Compose (for Elasticsearch/Kibana)\n\n### 1. Start Elasticsearch \u0026 Kibana (Optional)\n```bash\ndocker-compose up -d\n```\n\n### 2. Run the API\n```bash\ncd src/LoggingProduction\ndotnet run\n```\n\nAPI: **http://localhost:5022**\n\n### 3. Run Tests\n```powershell\n.\\test-api.ps1\n```\n\n## API Endpoints\n\n### Products\n```bash\n# List all\nGET /api/products\n\n# Get by ID\nGET /api/products/{id}\n\n# Create\nPOST /api/products\nContent-Type: application/json\n{\"name\":\"Laptop\",\"price\":999.99,\"sku\":\"LAPTOP-001\"}\n\n# Update\nPUT /api/products/{id}\n{\"name\":\"Updated\",\"price\":1299.99,\"sku\":\"LAPTOP-001\"}\n\n# Delete\nDELETE /api/products/{id}\n```\n\n### Orders\n```bash\n# List all\nGET /api/orders\n\n# Get by ID\nGET /api/orders/{id}\n\n# Create\nPOST /api/orders\n{\"customerId\":\"CUST-001\",\"productId\":\"PROD-001\",\"quantity\":5}\n\n# Update\nPUT /api/orders/{id}\n{\"customerId\":\"CUST-001\",\"productId\":\"PROD-001\",\"quantity\":10}\n\n# Delete\nDELETE /api/orders/{id}\n```\n\n### Health Check\n```bash\nGET /health\n# Returns: {\"status\":\"healthy\",\"timestamp\":\"2025-12-05T...\"}\n```\n\n## Correlation IDs\n\nAdd correlation ID for distributed tracing:\n\n```bash\ncurl -H \"X-Correlation-ID: my-trace-id\" http://localhost:5022/api/products\n```\n\nThe API will:\n- Auto-generate ID if not provided\n- Include ID in all logs\n- Return ID in response header\n\n## PII Masking\n\nAutomatically masks sensitive data in logs:\n\n| PII Type | Example | Masked Output |\n|----------|---------|---------------|\n| Email | john@example.com | ***MASKED*** |\n| Credit Card | 4532-1234-5678-9010 | ****-****-****-9010 |\n| Phone | 555-123-4567 | ***-***-4567 |\n| Password | myPassword123 | ***MASKED*** |\n\n**Example:**\n```bash\n# Request with email\nPOST /api/orders\n{\"customerId\":\"john.doe@example.com\",\"items\":[]}\n\n# Log output (email masked)\nCreating order for customer ***MASKED*** with total 0\n```\n\n## Elasticsearch \u0026 Kibana Setup\n\n### Start Services\n```bash\ndocker-compose up -d\n```\n\n### Access Kibana\n1. Open: **http://localhost:5601**\n2. Create data view: `logstash-*`\n3. Go to **Discover** to view logs\n\n### Search Logs\n```\n# By correlation ID\nCorrelationId: \"my-trace-id\"\n\n# By customer\ncustomerId: \"CUST-001\"\n\n# Errors only\nLevel: \"Error\"\n\n# Slow requests\nElapsedMilliseconds: \u003e= 1000\n```\n\n### Stop Services\n```bash\n# Stop but keep data\ndocker-compose stop\n\n# Stop and delete everything\ndocker-compose down -v\n```\n\n## Configuration\n\n### Development (`appsettings.Development.json`)\n- Console + File sinks\n- 7-day log retention\n- Verbose output with properties\n\n### Production (`appsettings.Production.json`)\n- Console + File + Elasticsearch\n- 30-day retention, 100MB file limit\n- Monthly Elasticsearch indices\n- Offline buffering enabled\n\n## Architecture\n\n```\nLoggingProduction/\n├── API/\n│   ├── Endpoints/          # Minimal API routes\n│   └── Middleware/         # CorrelationIdMiddleware\n├── Data/\n│   ├── Models/            # Product, Order entities\n│   └── Repositories/      # In-memory storage\n├── Services/              # Business logic\n├── Telemetry/             # Source-generated loggers\n├── LoggingExtensions/     # Serilog, OpenTelemetry config\n└── Program.cs\n```\n\n## Logging Features\n\n### Smart Log Levels\n- Health checks → Verbose (hidden)\n- Normal requests → Information\n- Client errors (4xx) → Warning\n- Server errors (5xx) → Error\n- Slow requests (\u003e1s) → Warning\n\n### Log Enrichment\nEvery log includes:\n- `CorrelationId` - Request tracing\n- `ClientIP` - Client address\n- `UserAgent` - Client browser/tool\n- `MachineName` - Server name\n- `Environment` - Dev/Prod\n- `ThreadId` - Thread number\n- `Application` - \"LoggingProduction\"\n\n### Source-Generated Logging\nUses `[LoggerMessage]` attributes for high performance:\n\n```csharp\n[LoggerMessage(Level = LogLevel.Information,\n    Message = \"Creating product with name {ProductName}\")]\npublic static partial void LogCreatingProduct(ILogger logger, string productName);\n```\n\n**Performance:** 3x faster than manual logging, zero allocations.\n\n## OpenTelemetry Tracing\n\nDistributed tracing with step-by-step timing:\n\n```\nActivity.TraceId:    624bc726a90c58eb5414aa22319ac7d5\nActivity.SpanId:     c9453b92aea51cab\nActivity.DisplayName: POST /api/products/\nActivity.Duration:   00:00:00.6670350\nActivity.Tags:\n  - http.response.status_code: 201\n  - server.address: localhost\n  - server.port: 5022\n```\n\n**Environment-based export:**\n- Development: Console (for debugging)\n- Production: OTLP to Elastic APM (port 4317)\n\n## Testing\n\nRun the test suite:\n\n```powershell\n.\\test-api.ps1\n```\n\nTests include:\n1. Product creation with PII\n2. Order creation (email masking test)\n3. GET requests with auto-generated correlation IDs\n4. Concurrent requests\n\nCheck logs for:\n- All requests have correlation IDs\n- Emails masked as `***MASKED***`\n- Structured JSON logging\n- OpenTelemetry traces with timing\n\n## Production Readiness\n\n### Security\n- ✅ PII masking (GDPR/PCI-DSS compliant)\n- ✅ No sensitive data in logs\n- ✅ Correlation IDs for audit trails\n\n### Performance\n- ✅ Async sinks (non-blocking)\n- ✅ Source-generated logging (3x faster)\n- ✅ Batch processing for Elasticsearch\n- ✅ File size limits and rotation\n\n### Observability\n- ✅ Structured logging (queryable)\n- ✅ Distributed tracing\n- ✅ Centralized log aggregation\n- ✅ Real-time monitoring in Kibana\n\n### Scalability\n- ✅ Daily rolling files\n- ✅ Monthly Elasticsearch indices\n- ✅ Auto-cleanup (7-30 day retention)\n- ✅ Buffering for offline resilience\n\n## Best Practices Demonstrated\n\n- Minimal API with method injection\n- Service layer abstraction\n- Repository pattern (in-memory)\n- Async/await throughout\n- Clean separation of concerns\n- Environment-specific configuration\n- Source generators for performance\n- PII protection by design\n\n## Troubleshooting\n\n### App Won't Start\n```bash\n# Check port 5022 is available\nnetstat -ano | findstr :5022\n\n# Rebuild\ndotnet clean\ndotnet build\n```\n\n### No Logs in Kibana\n```bash\n# Check Elasticsearch is running\ncurl http://localhost:9200\n\n# Check if indices created\ncurl http://localhost:9200/logstash-*/_stats\n\n# Restart containers\ndocker-compose restart\n```\n\n### PII Not Masked\nCheck logs for `***MASKED***`. If not appearing:\n- Verify `Serilog.Enrichers.Sensitive` package installed\n- Check `SerilogConfiguration.cs` has masking enricher\n- Rebuild: `dotnet build`\n\n## Resources\n\n- [Serilog Documentation](https://serilog.net/)\n- [OpenTelemetry .NET](https://opentelemetry.io/docs/languages/net/)\n- [Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)\n- [Kibana User Guide](https://www.elastic.co/guide/en/kibana/current/index.html)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgabegi%2Fdotnet-api-logging-opentelemetry-elasticsearch/lists"}