{"id":49805939,"url":"https://github.com/thealistairross/sentinel-log-aggregator","last_synced_at":"2026-05-12T21:04:31.821Z","repository":{"id":322003270,"uuid":"1087504247","full_name":"TheAlistairRoss/Sentinel-Log-Aggregator","owner":"TheAlistairRoss","description":"Azure SDK-compliant Python client library for aggregating logs from multiple Microsoft Sentinel workspace","archived":false,"fork":false,"pushed_at":"2025-11-28T19:40:15.000Z","size":41613,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-30T22:51:59.226Z","etag":null,"topics":["azure","azure-monitor","azure-sdk","cybersecurity","kql","log-analytics","microsoft","microsoft-sentinel","python","security-analytics","siem"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TheAlistairRoss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"docs/security-implementation.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-11-01T03:35:57.000Z","updated_at":"2025-11-28T19:37:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator","commit_stats":null,"previous_names":["thealistairross/sentinel-log-aggregator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TheAlistairRoss/Sentinel-Log-Aggregator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheAlistairRoss%2FSentinel-Log-Aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheAlistairRoss%2FSentinel-Log-Aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheAlistairRoss%2FSentinel-Log-Aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheAlistairRoss%2FSentinel-Log-Aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TheAlistairRoss","download_url":"https://codeload.github.com/TheAlistairRoss/Sentinel-Log-Aggregator/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TheAlistairRoss%2FSentinel-Log-Aggregator/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32956855,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-12T09:19:52.626Z","status":"ssl_error","status_checked_at":"2026-05-12T09:17:33.438Z","response_time":102,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-monitor","azure-sdk","cybersecurity","kql","log-analytics","microsoft","microsoft-sentinel","python","security-analytics","siem"],"created_at":"2026-05-12T21:04:23.389Z","updated_at":"2026-05-12T21:04:31.807Z","avatar_url":"https://github.com/TheAlistairRoss.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Microsoft Sentinel Log Aggregator\n\nAn Azure SDK-compliant Python client library for aggregating and processing logs from multiple Microsoft Sentinel workspaces into centralized reporting tables for security analytics and dashboard creation.\n\n## Features\n\n- **Azure SDK Compliance**: Follows Microsoft Azure SDK design guidelines and patterns\n- **Multi-workspace Support**: Query and aggregate data across multiple Sentinel workspaces\n- **Batch Processing**: Configurable time-based batching with concurrent execution\n- **Centralized Reporting**: Transform and normalize data for centralized analytics\n- **Comprehensive Error Handling**: Service-specific exceptions with detailed error information\n- **Distributed Tracing**: Built-in Azure Monitor Application Insights integration\n- **Long-running Operations**: LRO support for batch operations with progress tracking\n- **Health Monitoring**: Built-in health checks and service diagnostics\n- **Standard Authentication**: Azure Identity integration with multiple credential types\n\n## Installation\n\n### Requirements\n- Python 3.11 or higher (tested on Python 3.11-3.14)\n- Azure subscription with Microsoft Sentinel workspaces\n- Appropriate Azure permissions (see Configuration section)\n\n### Install from GitHub\n\n#### Install Latest Release (Recommended)\n```bash\n# Install from latest GitHub release\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git\n\n# Install from specific tag/version\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@v1.0.0\n```\n\n#### Install from Release Package\n```bash\n# Install wheel package from releases\npip install https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator/releases/latest/download/sentinel_log_aggregator-1.0.0-py3-none-any.whl\n\n# Install source distribution from releases\npip install https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator/releases/download/v1.0.0/sentinel-log-aggregator-1.0.0.tar.gz\n```\n\n#### Install from Development Branches\n```bash\n# Install from develop branch (latest development features)\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@develop\n\n# Install from any specific branch\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@feature/your-branch-name\n\n# Install from specific commit\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@abc123f\n```\n\n\u003e **⚠️ Development Branch Warning**: Development branches may contain unstable features and breaking changes. Always use tagged releases for production environments.\n\n#### Development Installation (Editable)\n```bash\n# Clone and install in editable mode for development\ngit clone https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git\ncd Sentinel-Log-Aggregator\n\n# Switch to develop branch (optional)\ngit checkout develop\n\n# Install in editable mode with development dependencies\npip install -e \".[dev,security]\"\n```\n\n## Quick Start\n\n### Simple Installation and Basic Usage\n\nThis section covers the fastest way to get started with basic functionality.\n\n#### Install the Package\n\n```bash\n# Install from GitHub (latest release)\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git\n```\n\n#### Basic Configuration\n\nCreate a simple environment configuration:\n\n```bash\n# Required settings\nexport DCR_LOGS_INGESTION_ENDPOINT=\"https://your-dcr-endpoint.monitor.azure.com\"\nexport DCR_RULE_ID=\"dcr-your-rule-id\"\n\n# Time range options (use one of these methods):\n\n# Option 1: Lookback period (ISO 8601 duration)\nexport LOOKBACK_PERIOD=\"P7D\"              # Look back 7 days\nexport BATCH_TIME_SIZE=\"PT12H\"            # Process in 12-hour batches\n\n# Option 2: Explicit time range\nexport START_TIME=\"2025-01-01T00:00:00Z\"  # Explicit start time (ISO 8601)\nexport END_TIME=\"2025-01-07T00:00:00Z\"    # Explicit end time (ISO 8601)\nexport BATCH_TIME_SIZE=\"PT6H\"             # Process in 6-hour batches\n\n# Option 3: Use last successful runs (requires health logging)\nexport USE_LAST_SUCCESSFUL=true           # Start from last successful completion\nexport HEALTH_LOGGING_ENABLED=true       # Enable health tracking\nexport BATCH_TIME_SIZE=\"PT24H\"           # Process in 24-hour batches\n\n# Optional settings\nexport LOG_LEVEL=INFO\n```\n\n#### Create Your First Query\n\nCreate a simple query file `my_query.yaml`:\n\n```yaml\nname: \"security_events_summary\"\ndestination_stream: \"Custom-SecuritySummary_CL\"\ndescription: \"Basic security events summary\"\n\nparameters:\n  row_level_security_tag:\n    type: \"string\"\n    required: false\n    default: \"\"\n\nquery: |\n  SecurityEvent\n  | where TimeGenerated \u003e ago(7d)\n  | summarize EventCount = count() by Computer\n  | extend WorkspaceTag = \"{row_level_security_tag}\"\n```\n\n#### Create Workspace Configuration\n\nCreate a simple `workspaces.yaml`:\n\n```yaml\nworkspaces:\n  - resource_id: \"/subscriptions/YOUR-SUB-ID/resourcegroups/YOUR-RG/providers/microsoft.operationalinsights/workspaces/YOUR-WORKSPACE\"\n    customer_id: \"YOUR-WORKSPACE-CUSTOMER-ID\"\n    queries_list:\n      - \"my_query.yaml\"\n    parameters:\n      row_level_security_tag: \"main\"\n```\n\n#### Run the Aggregator\n\n```bash\n# Run with your configuration\nsentinel-aggregator run --workspace-config workspaces.yaml\n```\n\n### Simple Programmatic Usage\n\nFor basic programmatic usage without complex configuration:\n\n```python\nimport asyncio\nfrom azure.identity.aio import DefaultAzureCredential\nfrom sentinel_log_aggregator import SentinelAggregatorClient\n\nasync def simple_example():\n    # Auto-configure from environment variables\n    credential = DefaultAzureCredential()\n    \n    # Connection string approach (simplest)\n    client = SentinelAggregatorClient.from_connection_string(\n        \"endpoint=https://your-dcr.monitor.azure.com;dcr_rule_id=dcr-your-rule-id\"\n    )\n    \n    async with client:\n        # Quick health check\n        health = await client.get_service_properties()\n        print(f\"Service status: {health.connectivity_status}\")\n        \n        # Simple query\n        result = await client.query_workspace(\n            workspace_id=\"your-workspace-id\",\n            query=\"SecurityEvent | take 5\"\n        )\n        \n        if result.succeeded:\n            print(f\"Got {result.record_count} records\")\n\n# Run it\nasyncio.run(simple_example())\n```\n\n---\n\n## Advanced Setup and Configuration\n\nThis section covers complex scenarios, enterprise deployments, and advanced configuration options.\n\n### Installation Options\n\n#### Production Installation\n\n```bash\n# Install from specific release tag\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@v0.1.0\n\n# Install from release package (exact version control)\npip install https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator/releases/download/v0.1.0/sentinel-log-aggregator-0.1.0.tar.gz\n```\n\n#### Development Installation\n\n```bash\n# Clone and install in editable mode\ngit clone https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git\ncd Sentinel-Log-Aggregator\ngit checkout develop  # Latest development features\npip install -e \".[dev,security]\"\n```\n\n#### Branch-Specific Installation\n\n```bash\n# Install from development branch (latest features)\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@develop\n\n# Install from specific feature branch\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@feature/your-branch-name\n\n# Install from specific commit\npip install git+https://github.com/TheAlistairRoss/Sentinel-Log-Aggregator.git@abc123f\n```\n\n\u003e **⚠️ Development Branch Warning**: Development branches may contain unstable features and breaking changes. Always use tagged releases for production environments.\n\n### Advanced Environment Configuration\n\n#### Complete Environment Variables\n\n```bash\n# Required settings\nDCR_LOGS_INGESTION_ENDPOINT=https://your-dcr-endpoint.monitor.azure.com\nDCR_RULE_ID=dcr-your-rule-id\n\n# Time range configuration (choose one method)\n# Method 1: Lookback period (recommended for scheduled runs)\nLOOKBACK_PERIOD=P30D                    # ISO 8601 duration: P30D (30 days), PT24H (24 hours), P1DT12H (1.5 days)\n\n# Method 2: Explicit time range (for historical analysis)\nSTART_TIME=2025-01-01T00:00:00Z         # ISO 8601 datetime\nEND_TIME=2025-01-31T23:59:59Z           # ISO 8601 datetime\n\n# Method 3: Continue from last successful run (for incremental processing)\nUSE_LAST_SUCCESSFUL=true               # Boolean: true/false\nHEALTH_LOGGING_ENABLED=true            # Required when using last successful\n\n# Batch processing configuration\nBATCH_TIME_SIZE=PT24H                  # ISO 8601 duration: batch size for time-based processing\n\n# Performance tuning\nMAX_CONCURRENT_QUERIES=5               # Number of concurrent queries\nQUERY_TIMEOUT_SECONDS=300              # Query timeout in seconds\n\n# Logging and monitoring\nLOG_LEVEL=INFO\nCORRELATION_ID_PREFIX=sentinel-agg\n\n# Authentication - choose one method:\n\n# Option 1: Managed Identity (recommended for Azure-hosted environments)\n# No additional configuration needed\n\n# Option 2: Service Principal (CI/CD and automation)\nAZURE_CLIENT_ID=your-client-id\nAZURE_TENANT_ID=your-tenant-id\nAZURE_CLIENT_SECRET=your-client-secret\n\n# Option 3: Azure CLI (development)\n# Run 'az login' first, then no additional config needed\n```\n\n#### YAML Configuration File\n\nCreate a comprehensive `config.yaml`:\n\n```yaml\n# Core configuration\ndcr_logs_ingestion_endpoint: \"https://your-endpoint.monitor.azure.com\"\ndcr_rule_id: \"dcr-your-rule-id\"\n\n# Time range configuration (choose one method)\n# Method 1: Lookback period\nlookback_period: \"P30D\"              # ISO 8601 duration\n\n# Method 2: Explicit time range\nstart_time: \"2025-01-01T00:00:00Z\"   # ISO 8601 datetime\nend_time: \"2025-01-31T23:59:59Z\"     # ISO 8601 datetime\n\n# Method 3: Last successful continuation\nuse_last_successful: true           # Boolean\nhealth_logging_enabled: true        # Required for last successful\n\n# Batch processing\nbatch_time_size: \"PT24H\"            # ISO 8601 duration\n\n# Performance settings\nmax_concurrent_queries: 10\nquery_timeout_seconds: 600\n\n# Monitoring settings\nlog_level: \"INFO\"\nenable_distributed_tracing: true\ncorrelation_id_prefix: \"sentinel-prod\"\n\n# Retry configuration\nmax_retries: 3\nretry_delay_seconds: 5\nexponential_backoff: true\n\n# Security settings\nvalidate_queries: true\nsanitize_parameters: true\n```\n\n### Complex Workspace Configuration\n\nCreate an enterprise-grade `workspaces.yaml`:\n\n```yaml\n# Enterprise Microsoft Sentinel Workspaces Configuration\nmetadata:\n  version: \"2.0\"\n  description: \"Production workspace configuration for security analytics\"\n  last_updated: \"2025-11-03\"\n  environment: \"production\"\n\nworkspaces:\n  # Production Security Operations Center\n  - resource_id: \"/subscriptions/prod-sub-id/resourcegroups/security-rg/providers/microsoft.operationalinsights/workspaces/prod-soc-sentinel\"\n    customer_id: \"12345678-1234-1234-1234-123456789012\"\n    queries_list:\n      - \"queries/security/incident_analysis.yaml\"\n      - \"queries/security/threat_intelligence.yaml\"\n      - \"queries/compliance/regulatory_audit.yaml\"\n      - \"queries/reports/executive_summary.yaml\"\n    parameters:\n      row_level_security_tag: \"prod-soc\"\n      environment: \"production\"\n      region: \"east-us\"\n      cost_center: \"security-ops\"\n  \n  # Development Environment\n  - resource_id: \"/subscriptions/dev-sub-id/resourcegroups/dev-rg/providers/microsoft.operationalinsights/workspaces/dev-sentinel\"\n    customer_id: \"87654321-4321-4321-4321-210987654321\"\n    queries_list:\n      - \"queries/development/test_scenarios.yaml\"\n      - \"queries/shared/baseline_metrics.yaml\"\n    parameters:\n      row_level_security_tag: \"dev\"\n      environment: \"development\"\n      region: \"west-us\"\n  \n  # Regional SOC - Europe\n  - resource_id: \"/subscriptions/eu-sub-id/resourcegroups/eu-security-rg/providers/microsoft.operationalinsights/workspaces/eu-soc-sentinel\"\n    customer_id: \"11111111-2222-3333-4444-555555555555\"\n    queries_list:\n      - \"queries/security/incident_analysis.yaml\"\n      - \"queries/compliance/gdpr_compliance.yaml\"\n      - \"queries/regional/eu_specific_threats.yaml\"\n    parameters:\n      row_level_security_tag: \"eu-soc\"\n      environment: \"production\"\n      region: \"north-europe\"\n      gdpr_compliant: true\n      data_residency: \"eu\"\n\n# Environment-specific settings\nenvironments:\n  production:\n    max_concurrent_queries: 3\n    batch_hours: 12\n    enable_monitoring: true\n  \n  development:\n    max_concurrent_queries: 2\n    batch_hours: 6\n    enable_monitoring: false\n```\n\n### Query Organization and Setup\n\nFor complex query organization, see the dedicated [Query Setup Guide](docs/query-setup.md) which covers:\n\n- **Flexible Directory Structures**: Organize queries by type, environment, or team\n- **Query File Format**: Complete YAML specification with parameters and metadata\n- **Relative Path Configuration**: Use file paths instead of hardcoded query names\n- **Parameter Substitution**: Advanced parameter handling and type validation\n- **Best Practices**: Query development, testing, and deployment guidelines\n\nQuick example of advanced query organization:\n\n```yaml\n# Advanced workspace configuration with organized queries\nworkspaces:\n  - resource_id: \"/subscriptions/.../workspaces/enterprise-sentinel\"\n    customer_id: \"your-workspace-id\"\n    queries_list:\n      # Security team queries\n      - \"queries/security-team/incident_response.yaml\"\n      - \"queries/security-team/threat_hunting.yaml\"\n      \n      # Compliance queries\n      - \"queries/compliance/sox_audit.yaml\"\n      - \"queries/compliance/pci_monitoring.yaml\"\n      \n      # Custom analytics\n      - \"custom/executive_dashboard.yaml\"\n      - \"custom/regional_analysis.yaml\"\n    parameters:\n      row_level_security_tag: \"enterprise\"\n      compliance_level: \"high\"\n```\n\n---\n\n## Programmatic Usage (Advanced)\n\nAdvanced programmatic usage for enterprise applications and custom integrations.\n\n### Azure SDK-Compliant Client\n\n```python\nimport asyncio\nfrom azure.identity.aio import DefaultAzureCredential, ClientSecretCredential\nfrom sentinel_log_aggregator import (\n    SentinelAggregatorClient,\n    SentinelAggregatorClientOptions,\n    WorkspaceConfig\n)\n\nasync def enterprise_example():\n    # Advanced client configuration with new time parameters\n    options = SentinelAggregatorClientOptions(\n        # Time range method 1: Lookback period\n        lookback_period=\"P30D\",              # 30 days back (ISO 8601)\n        batch_time_size=\"PT12H\",             # 12-hour batches (ISO 8601)\n        \n        # Alternative: Explicit time range\n        # start_time=\"2025-01-01T00:00:00Z\",\n        # end_time=\"2025-01-31T23:59:59Z\",\n        \n        # Performance settings\n        max_concurrent_queries=10,\n        query_timeout_seconds=600,\n        log_level=\"DEBUG\",\n        enable_distributed_tracing=True,\n        correlation_id_prefix=\"enterprise-analytics\"\n    )\n    \n    # Choose authentication method\n    credential = DefaultAzureCredential()\n    # Or for service principal:\n    # credential = ClientSecretCredential(tenant_id, client_id, client_secret)\n    \n    # Create client with full configuration\n    async with SentinelAggregatorClient(\n        dcr_logs_ingestion_endpoint=\"https://your-dcr.monitor.azure.com\",\n        credential=credential,\n        options=options\n    ) as client:\n        \n        # Health check with detailed diagnostics\n        service_props = await client.get_service_properties()\n        print(f\"Service version: {service_props.service_version}\")\n        print(f\"Connectivity: {service_props.connectivity_status}\")\n        print(f\"Features: {service_props.supported_features}\")\n        \n        # Complex workspace configurations\n        workspaces = [\n            WorkspaceConfig(\n                resource_id=\"/subscriptions/.../workspaces/prod-soc\",\n                customer_id=\"prod-workspace-id\",\n                queries_list=[\"queries/security/advanced_analytics.yaml\"],\n                parameters={\n                    \"row_level_security_tag\": \"prod-soc\",\n                    \"environment\": \"production\",\n                    \"compliance_level\": \"high\"\n                }\n            ),\n            WorkspaceConfig(\n                resource_id=\"/subscriptions/.../workspaces/dev-soc\",\n                customer_id=\"dev-workspace-id\", \n                queries_list=[\"queries/development/test_analytics.yaml\"],\n                parameters={\n                    \"row_level_security_tag\": \"dev-soc\",\n                    \"environment\": \"development\"\n                }\n            )\n        ]\n        \n        # Execute with advanced error handling\n        try:\n            for workspace in workspaces:\n                for query_path in workspace.queries_list:\n                    result = await client.query_workspace_with_path(\n                        workspace_id=workspace.customer_id,\n                        query_path=query_path,\n                        parameters=workspace.parameters\n                    )\n                    \n                    if result.succeeded:\n                        # Process and upload results\n                        upload_result = await client.upload_logs(\n                            data=result.data,\n                            stream_name=result.stream_name\n                        )\n                        \n                        print(f\"✅ {query_path}: {upload_result.record_count} records uploaded\")\n                    else:\n                        print(f\"❌ {query_path}: {result.error_message}\")\n                        \n        except Exception as e:\n            print(f\"Critical error: {e}\")\n\nasyncio.run(enterprise_example())\n```\n\n### Long-Running Operations (LRO)\n\n```python\nasync def batch_operations_example():\n    async with SentinelAggregatorClient(endpoint, credential, options=options) as client:\n        # Start long-running batch operation\n        poller = await client.begin_batch_operation(\n            workspaces=workspace_configs,\n            queries=[\"query_incident_summary\", \"query_user_analytics\"],\n            correlation_id=\"batch-2025-11-03\"\n        )\n        \n        print(\"🚀 Batch operation started...\")\n        \n        # Monitor progress with detailed feedback\n        while not poller.done():\n            try:\n                # Get intermediate results (non-blocking)\n                result = poller.result(timeout=30)\n                \n                print(f\"📊 Progress: {result.completed_operations}/{result.total_operations}\")\n                print(f\"✅ Successful: {result.success_count}\")\n                print(f\"❌ Failed: {result.error_count}\")\n                print(f\"⏱️ Elapsed: {result.elapsed_time}s\")\n                \n                # Optional: pause between checks\n                await asyncio.sleep(10)\n                \n            except TimeoutError:\n                print(\"⏳ Operation still in progress...\")\n        \n        # Get final comprehensive results\n        final_result = await poller.result()\n        \n        print(f\"🎯 Batch completed:\")\n        print(f\"  Total operations: {final_result.total_operations}\")\n        print(f\"  Successful: {final_result.success_count}\")\n        print(f\"  Failed: {final_result.error_count}\")\n        print(f\"  Total records: {final_result.total_records}\")\n        print(f\"  Execution time: {final_result.total_execution_time}s\")\n        \n        # Process any failures\n        for failure in final_result.failures:\n            print(f\"❌ Failed: {failure.workspace_id} - {failure.error_message}\")\n```\n\n### Custom Query Loading and Execution\n\n```python\nfrom sentinel_log_aggregator.query_registry import QueryRegistry\nfrom pathlib import Path\n\nasync def custom_query_management():\n    # Advanced query registry usage\n    registry = QueryRegistry()\n    \n    # Load queries from multiple sources\n    registry.load_queries_from_directory(Path(\"queries/production\"), recursive=True)\n    registry.load_queries_from_directory(Path(\"queries/shared\"))\n    \n    # Load specific queries dynamically\n    custom_query = registry.load_query_from_path(\n        \"custom/special_analysis.yaml\",\n        base_directory=Path(\".\")\n    )\n    \n    # Validate all queries before execution\n    validation_results = registry.validate_all_queries()\n    for query_name, issues in validation_results.items():\n        if issues:\n            print(f\"⚠️ Query {query_name} has validation issues:\")\n            for issue in issues:\n                print(f\"  - {issue}\")\n    \n    # Get query metadata\n    for query_name in registry.list_queries():\n        metadata = registry.get_metadata(query_name)\n        print(f\"📋 {query_name}: {metadata.description}\")\n        print(f\"   Tags: {metadata.tags}\")\n        print(f\"   Version: {metadata.version}\")\n    \n    # Execute with custom parameters\n    async with SentinelAggregatorClient(endpoint, credential) as client:\n        query = registry.get_query(\"advanced_threat_detection\")\n        \n        custom_params = {\n            \"row_level_security_tag\": \"enterprise\",\n            \"time_window_hours\": 72,\n            \"severity_threshold\": \"High\",\n            \"threat_types\": \"Malware,Phishing,C2\"\n        }\n        \n        result = await client.execute_query(\n            workspace_id=\"workspace-id\",\n            query=query,\n            parameters=custom_params\n        )\n```\n\n---\n\n## CLI Usage (Advanced)\n\nEnterprise-grade command-line usage for automation, monitoring, and operations.\n\n### Health Monitoring and Diagnostics\n\n```bash\n# Comprehensive health check with detailed output\nsentinel-aggregator health \\\n  --workspace-config workspaces.yaml \\\n  --verbose \\\n  --output-format json\n\n# Health check with specific workspace validation\nsentinel-aggregator health \\\n  --workspace-config workspaces.yaml \\\n  --workspace-filter \"environment=production\" \\\n  --test-queries\n\n# Export health metrics for monitoring systems\nsentinel-aggregator health \\\n  --workspace-config workspaces.yaml \\\n  --export-metrics \\\n  --metrics-format prometheus\n```\n\n### Production Execution\n\n```bash\n# Production run with lookback period (recommended)\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --config config.yaml \\\n  --lookback-period \"P30D\" \\\n  --batch-time-size \"PT12H\" \\\n  --max-concurrent 5 \\\n  --correlation-id \"prod-run-$(date +%Y%m%d-%H%M%S)\"\n\n# Run with explicit time range (historical analysis)\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --start-time \"2025-01-01T00:00:00Z\" \\\n  --end-time \"2025-01-31T23:59:59Z\" \\\n  --batch-time-size \"PT6H\"\n\n# Continue from last successful run (incremental processing)\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --use-last-successful \\\n  --health-logging-enabled\n\n# Run with workspace filtering (enterprise scenarios)\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --workspace-filter \"environment=production,region=us-east\" \\\n  --query-filter \"tags=compliance,security\"\n\n# Dry run for validation (staging/testing)\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --dry-run \\\n  --validate-only \\\n  --verbose\n```\n\n### New CLI Commands\n\n```bash\n# Check last successful runs across workspaces\nsentinel-aggregator query-status \\\n  --workspace-config workspaces.yaml \\\n  --show-details\n\n# Query status for specific queries\nsentinel-aggregator query-status \\\n  --workspace-config workspaces.yaml \\\n  --query-filter \"incident_summary,threat_hunting\" \\\n  --lookback-days 7\n\n# Show execution history and timing\nsentinel-aggregator query-status \\\n  --workspace-config workspaces.yaml \\\n  --show-execution-history \\\n  --format table\n```\n\n### Advanced Validation and Testing\n\n```bash\n# Comprehensive configuration validation\nsentinel-aggregator validate \\\n  --workspace-config workspaces.yaml \\\n  --config config.yaml \\\n  --validate-queries \\\n  --validate-permissions \\\n  --validate-connectivity\n\n# Query-specific validation\nsentinel-aggregator validate \\\n  --workspace-config workspaces.yaml \\\n  --query-path \"queries/security/advanced_analytics.yaml\" \\\n  --test-parameters \\\n  --check-syntax\n\n# Environment validation for deployment\nsentinel-aggregator validate \\\n  --workspace-config workspaces.yaml \\\n  --environment production \\\n  --check-quotas \\\n  --verify-dcr-access\n```\n\n### Monitoring and Logging\n\n```bash\n# Debug mode with comprehensive logging\nsentinel-aggregator \\\n  --log-level DEBUG \\\n  --log-format json \\\n  --correlation-id \"debug-session-123\" \\\n  run --workspace-config workspaces.yaml\n\n# Export execution metrics\nsentinel-aggregator run \\\n  --workspace-config workspaces.yaml \\\n  --export-metrics \\\n  --metrics-file execution-metrics.json \\\n  --enable-performance-tracking\n\n# Run with custom log destination\nsentinel-aggregator \\\n  --log-file /var/log/sentinel-aggregator.log \\\n  --log-level INFO \\\n  run --workspace-config workspaces.yaml\n```\n\n---\n\n## Configuration Reference\n\n### Client Options\n\n| Option | Environment Variable | Default | Description |\n|--------|---------------------|---------|-------------|\n| `dcr_logs_ingestion_endpoint` | `DCR_LOGS_INGESTION_ENDPOINT` | Required | Azure Monitor ingestion endpoint |\n| `dcr_rule_id` | `DCR_RULE_ID` | Required | Data Collection Rule ID |\n| **Time Range Options (choose one method)** | | | |\n| `lookback_period` | `LOOKBACK_PERIOD` | \"P30D\" | ISO 8601 duration for relative time range |\n| `start_time` | `START_TIME` | None | ISO 8601 datetime for explicit start time |\n| `end_time` | `END_TIME` | None | ISO 8601 datetime for explicit end time |\n| `use_last_successful` | `USE_LAST_SUCCESSFUL` | false | Continue from last successful completion |\n| **Batch Processing** | | | |\n| `batch_time_size` | `BATCH_TIME_SIZE` | \"PT24H\" | ISO 8601 duration for batch size |\n| **Performance** | | | |\n| `max_concurrent_queries` | `MAX_CONCURRENT_QUERIES` | 5 | Maximum concurrent query execution |\n| `query_timeout_seconds` | `QUERY_TIMEOUT_SECONDS` | 300 | Query timeout in seconds |\n| **Monitoring** | | | |\n| `health_logging_enabled` | `HEALTH_LOGGING_ENABLED` | false | Enable health tracking and logging |\n| `log_level` | `LOG_LEVEL` | INFO | Logging level |\n\n### ISO 8601 Time Format Reference\n\nThe new time parameters use ISO 8601 standardized formats for maximum precision and clarity.\n\n#### Duration Examples (`lookback_period`, `batch_time_size`)\n\n| Duration | ISO 8601 Format | Description |\n|----------|----------------|-------------|\n| 1 hour | `PT1H` | 1 hour |\n| 6 hours | `PT6H` | 6 hours |\n| 12 hours | `PT12H` | 12 hours |\n| 1 day | `P1D` or `PT24H` | 24 hours |\n| 3 days | `P3D` | 3 days |\n| 1 week | `P7D` | 7 days |\n| 1 month | `P30D` | 30 days |\n| 1.5 days | `P1DT12H` | 1 day + 12 hours |\n| 2 hours 30 minutes | `PT2H30M` | 2.5 hours |\n\n#### DateTime Examples (`start_time`, `end_time`)\n\n| Description | ISO 8601 Format |\n|-------------|----------------|\n| UTC midnight | `2025-01-01T00:00:00Z` |\n| UTC with time | `2025-01-15T14:30:00Z` |\n| With microseconds | `2025-01-15T14:30:00.123456Z` |\n| Timezone offset | `2025-01-15T14:30:00+05:00` |\n\n#### Time Range Configuration Examples\n\n```yaml\n# Example 1: Lookback processing (scheduled runs)\nlookback_period: \"P7D\"        # Look back 7 days\nbatch_time_size: \"PT12H\"      # Process in 12-hour chunks\n\n# Example 2: Historical analysis (specific period)\nstart_time: \"2025-01-01T00:00:00Z\"\nend_time: \"2025-01-31T23:59:59Z\"\nbatch_time_size: \"PT6H\"       # Process in 6-hour chunks\n\n# Example 3: Incremental processing (continue from last run)\nuse_last_successful: true     # Start from last completion\nbatch_time_size: \"PT24H\"      # Process in daily chunks\nhealth_logging_enabled: true  # Required for tracking\n```\n\n### Workspace Parameters\n\nWorkspace-specific parameters that can be used in query substitution:\n\n| Parameter | Type | Description | Example |\n|-----------|------|-------------|---------|\n| `row_level_security_tag` | string | Data isolation identifier | `\"production\"`, `\"dev\"` |\n| `environment` | string | Environment designation | `\"prod\"`, `\"staging\"`, `\"dev\"` |\n| `region` | string | Geographic region | `\"us-east\"`, `\"eu-west\"` |\n| `compliance_level` | string | Compliance requirements | `\"high\"`, `\"medium\"`, `\"basic\"` |\n| `cost_center` | string | Billing/cost allocation | `\"security-ops\"`, `\"it-dept\"` |\n| `data_residency` | string | Data location requirements | `\"us\"`, `\"eu\"`, `\"global\"` |\n\n### Query File Schema\n\nComplete schema for query YAML files:\n\n```yaml\n# Required fields\nname: string                    # Unique query identifier\ndestination_stream: string      # Target data stream\ndescription: string            # Human-readable description\nquery: string                  # KQL query with parameter placeholders\n\n# Optional fields\nstream_name: string            # Custom stream name (defaults to destination_stream)\nversion: string               # Query version for tracking\ntags: [string]                # Categorization tags\n\n# Parameters schema\nparameters:\n  parameter_name:\n    type: string               # \"string\", \"int\", \"double\", \"bool\", \"datetime\"\n    required: boolean          # Whether parameter is required\n    default: any              # Default value if not provided\n    description: string       # Parameter description\n```\n\n### Error Handling\n\nThe package provides comprehensive error handling with service-specific exceptions:\n\n```python\nfrom sentinel_log_aggregator import (\n    SentinelAggregatorError,\n    QueryExecutionError, \n    WorkspaceAccessError,\n    DataIngestionError,\n    ConfigurationError\n)\n\ntry:\n    result = await client.query_workspace(workspace_id, query)\nexcept QueryExecutionError as e:\n    print(f\"Query failed: {e.message}\")\n    print(f\"Workspace: {e.workspace_id}\")\n    print(f\"Query: {e.query_name}\")\nexcept WorkspaceAccessError as e:\n    print(f\"Access denied to workspace: {e.workspace_id}\")\nexcept ConfigurationError as e:\n    print(f\"Configuration error: {e.message}\")\nexcept SentinelAggregatorError as e:\n    print(f\"General service error: {e.message}\")\n```\n\n### Response Models\n\nAll operations return structured response objects:\n\n```python\n# Query result\nresult = await client.query_workspace(workspace_id, query)\nprint(f\"Success: {result.succeeded}\")\nprint(f\"Records: {result.record_count}\")\nprint(f\"Duration: {result.execution_time}s\")\nprint(f\"Status: {result.status}\")\n\n# Upload result  \nupload_result = await client.upload_logs(data, stream_name)\nprint(f\"Success: {upload_result.succeeded}\")\nprint(f\"Uploaded: {upload_result.record_count}\")\nprint(f\"Status: {upload_result.status}\")\n\n# Service properties\nservice_props = await client.get_service_properties()\nprint(f\"Version: {service_props.service_version}\")\nprint(f\"Status: {service_props.connectivity_status}\")\n```\n\n### Authentication\n\nThe package supports multiple Azure authentication methods:\n\n#### Managed Identity (Recommended for Azure-hosted)\n\n```python\nfrom azure.identity.aio import DefaultAzureCredential\n\n# Automatic in Azure-hosted environments\ncredential = DefaultAzureCredential()\n```\n\n#### Service Principal (CI/CD scenarios)\n\n```python\nfrom azure.identity.aio import ClientSecretCredential\n\ncredential = ClientSecretCredential(\n    tenant_id=\"your-tenant-id\",\n    client_id=\"your-client-id\", \n    client_secret=\"your-client-secret\"\n)\n```\n\n#### Azure CLI (Development)\n\n```bash\n# Login first\naz login\n\n# Then use in code\nfrom azure.identity.aio import AzureCLICredential\ncredential = AzureCLICredential()\n```\n\n### Required Azure Permissions\n\nYour identity needs the following permissions:\n\n- **Log Analytics Reader** on all source Sentinel workspaces\n- **Monitoring Metrics Publisher** for the DCR ingestion endpoint\n- **Data Collection Rule permissions** configured for your identity\n\n## Development\n\n### Setup Development Environment\n\n```bash\ngit clone \u003crepository-url\u003e\ncd sentinel-log-aggregator\npip install -e \".[dev]\"\npre-commit install\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npytest\n\n# Run with coverage\npytest --cov=sentinel_log_aggregator\n\n# Run specific test file\npytest tests/test_client.py\n```\n\n### Code Formatting\n\n```bash\n# Format code\nblack sentinel_log_aggregator tests\nisort sentinel_log_aggregator tests\n\n# Lint code\nflake8 sentinel_log_aggregator tests\nmypy sentinel_log_aggregator\n```\n\n## Architecture\n\n### Core Components\n\n- **SentinelAggregatorClient**: Main Azure SDK-compliant client\n- **SentinelAggregatorClientOptions**: Configuration management\n- **SentinelQueryEngine**: High-level batch processing engine\n- **WorkspaceManager**: Multi-workspace configuration and filtering\n- **Response Models**: Structured responses for all operations\n- **Exception Hierarchy**: Service-specific error handling\n\n### Data Flow\n\n1. **Configuration Loading**: Options loaded from environment/files\n2. **Authentication**: Azure Identity credential resolution\n3. **Client Creation**: Azure SDK-compliant client initialization\n4. **Workspace Discovery**: Load and validate workspace configurations\n5. **Batch Processing**: Time-based batching with concurrent execution\n6. **Query Execution**: KQL queries across multiple workspaces\n7. **Data Transformation**: Normalize and enrich data for reporting\n8. **Upload Processing**: Stream data to Azure Monitor ingestion\n9. **Progress Tracking**: Comprehensive logging and monitoring\n\n## Documentation\n\n### Available Documentation\n\n- **[Query Setup and Configuration](docs/query-setup.md)**: Comprehensive guide for organizing queries, using relative paths, and advanced query configuration\n- **[Installation Guide](docs/installation.md)**: Complete installation instructions and setup\n- **[CLI Usage Guide](docs/cli-usage.md)**: Detailed command-line interface documentation\n- **[Configuration Guide](docs/configuration.md)**: Advanced configuration options and environment setup\n- **[Health Logging Deployment](docs/health-logging-deployment.md)**: Deploy health logging infrastructure using Bicep templates\n- **[SDK Usage Guide](docs/sdk-usage.md)**: Programmatic usage examples and API reference\n- **[Development Guide](docs/development.md)**: Development setup and contributing guidelines\n- **[Security Implementation](docs/security-implementation.md)**: Security features and compliance documentation\n- **[GitHub Actions Workflows](docs/workflows.md)**: Comprehensive documentation of CI/CD and security workflows\n- **[Troubleshooting Guide](docs/troubleshooting.md)**: Common issues and solutions\n\n### Quick References\n\n- **API Documentation**: Generated from code docstrings (see built documentation)\n- **CLI Reference**: `sentinel-aggregator --help` for command-line usage\n- **Examples**: See `examples/` for practical implementation examples\n- **Changelog**: See `docs/changelog.md` for version history and changes\n\n### Workflow Documentation\n\nThe project includes comprehensive GitHub Actions workflows for:\n\n- **CI/CD Pipeline**: Automated testing, building, and deployment (Python 3.11+)\n- **Security Scanning**: Microsoft SDL-compliant security analysis with 12+ security tools\n- **Documentation Generation**: Automated Sphinx documentation with GitHub Pages deployment\n- **Package Distribution**: Automated PyPI publishing and GitHub releases\n\nSee [docs/workflows.md](docs/workflows.md) for detailed workflow documentation including job dependencies, troubleshooting, and best practices.\n\n## Contributing\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Development Workflow\n\nBefore contributing, please review:\n- [GitHub Actions Workflows Documentation](docs/workflows.md) for CI/CD pipeline details\n- Pre-commit hooks configuration for local security scanning\n- Test coverage requirements (target \u003e95%)\n- Security scanning requirements (zero high/critical vulnerabilities)\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Support\n\nFor support and questions:\n\n- Create an issue in the GitHub repository\n- Check the documentation and examples\n- Review the CLI help: `sentinel-aggregator --help`","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthealistairross%2Fsentinel-log-aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthealistairross%2Fsentinel-log-aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthealistairross%2Fsentinel-log-aggregator/lists"}