{"id":22301800,"url":"https://github.com/alexott/cyber-spark-data-connectors","last_synced_at":"2026-01-04T15:15:40.471Z","repository":{"id":264470184,"uuid":"875999184","full_name":"alexott/cyber-spark-data-connectors","owner":"alexott","description":"Cybersecurity-related custom data connectors for Spark","archived":false,"fork":false,"pushed_at":"2025-03-16T07:52:54.000Z","size":399,"stargazers_count":1,"open_issues_count":10,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-16T08:27:13.467Z","etag":null,"topics":["cybersecurity","databricks","pyspark","spark"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexott.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-21T08:27:07.000Z","updated_at":"2025-03-16T07:52:52.000Z","dependencies_parsed_at":"2025-01-05T09:24:09.511Z","dependency_job_id":"2c49b3a3-8325-439e-bb21-bef613f9a06d","html_url":"https://github.com/alexott/cyber-spark-data-connectors","commit_stats":null,"previous_names":["alexott/cyber-spark-data-connectors"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexott%2Fcyber-spark-data-connectors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexott%2Fcyber-spark-data-connectors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexott%2Fcyber-spark-data-connectors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexott%2Fcyber-spark-data-connectors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexott","download_url":"https://codeload.github.com/alexott/cyber-spark-data-connectors/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245563037,"owners_count":20635907,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cybersecurity","databricks","pyspark","spark"],"created_at":"2024-12-03T18:31:10.533Z","updated_at":"2026-01-04T15:15:40.464Z","avatar_url":"https://github.com/alexott.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Custom data sources/sinks for Cybersecurity-related work\n\nBased on [PySpark DataSource API](https://spark.apache.org/docs/preview/api/python/user_guide/sql/python_data_source.html) available with Spark 4 \u0026 [DBR 15.3+](https://docs.databricks.com/en/pyspark/datasources.html).  See [blog post](https://alexott.blogspot.com/2024/11/spark-custom-data-sources-and-sinks-for.html) for more details about implementation.\n\n- [Custom data sources/sinks for Cybersecurity-related work](#custom-data-sourcessinks-for-cybersecurity-related-work)\n  - [Available data sources](#available-data-sources)\n    - [Splunk data source](#splunk-data-source)\n      - [Writing to Splunk](#writing-to-splunk)\n      - [Reading from Splunk](#reading-from-splunk)\n        - [Batch Read](#batch-read)\n        - [Streaming Read](#streaming-read)\n    - [Microsoft Sentinel / Azure Monitor](#microsoft-sentinel--azure-monitor)\n      - [Authentication Requirements](#authentication-requirements)\n      - [Writing to Microsoft Sentinel / Azure Monitor](#writing-to-microsoft-sentinel--azure-monitor)\n      - [Reading from Microsoft Sentinel / Azure Monitor](#reading-from-microsoft-sentinel--azure-monitor)\n        - [Batch Read](#batch-read-1)\n        - [Streaming Read](#streaming-read-1)\n    - [Simple REST API](#simple-rest-api)\n  - [Installation](#installation)\n  - [Building](#building)\n  - [References](#references)\n\n## Available data sources\n\n\u003e [!NOTE]\n\u003e Most of these data sources/sinks are designed to work with relatively small amounts of data - alerts, etc.  If you need to read or write huge amounts of data, use native export/import functionality of corresponding external system.\n\n### Splunk data source\n\nThis data source supports both reading from and writing to [Splunk](https://www.splunk.com/) - both batch and streaming modes. Registered data source name is `splunk`.\n\n#### Writing to Splunk\n\nBy default, this data source will put all columns into the `event` object and send it to Splunk together with metadata (`index`, `source`, ...).  This behavior could be changed by providing `single_event_column` option to specify which string column should be used as the single value of `event`.\n\nBatch write usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(SplunkDataSource)\n\ndf = spark.range(10)\ndf.write.format(\"splunk\").mode(\"overwrite\") \\\n  .option(\"url\", \"http://localhost:8088/services/collector/event\") \\\n  .option(\"token\", \"...\").save()\n```\n\nStreaming write usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(SplunkDataSource)\n\ndir_name = \"tests/samples/json/\"\nbdf = spark.read.format(\"json\").load(dir_name)  # to infer schema - not use in the prod!\n\nsdf = spark.readStream.format(\"json\").schema(bdf.schema).load(dir_name)\n\nstream_options = {\n  \"url\": \"http://localhost:8088/services/collector/event\",\n  \"token\": \"....\",\n  \"source\": \"zeek\",\n  \"index\": \"zeek\",\n  \"host\": \"my_host\",\n  \"time_column\": \"ts\",\n  \"checkpointLocation\": \"/tmp/splunk-checkpoint/\"\n}\nstream = sdf.writeStream.format(\"splunk\") \\\n  .trigger(availableNow=True) \\\n  .options(**stream_options).start()\n```\n\nSupported write options:\n\n- `url` (string, required) - URL of the Splunk HTTP Event Collector (HEC) endpoint to send data to.  For example, `http://localhost:8088/collector/services/event`.\n- `token` (string, required) - HEC token to [authenticate to HEC endpoint](https://docs.splunk.com/Documentation/Splunk/9.3.1/Data/FormateventsforHTTPEventCollector#HTTP_authentication).\n- `index` (string, optional) - name of the Splunk index to send data to.  If omitted, the default index configured for HEC endpoint is used.\n- `source` (string, optional) - the source value to assign to the event data.\n- `host` (string, optional) - the host value to assign to the event data.\n- `sourcetype` (string, optional, default: `_json`) - the sourcetype value to assign to the event data. \n- `single_event_column` (string, optional) - specify which string column will be used as `event` payload.  Typically this is used to ingest log files content.\n- `time_column` (string, optional) - specify which column to use as event time value (the `time` value in Splunk payload).  Supported data types: `timestamp`, `float`, `int`, `long` (`float`/`int`/`long` values are treated as seconds since epoch).  If not specified, current timestamp will be used.\n- `indexed_fields` (string, optional) - comma-separated list of string columns to be [indexed in the ingestion time](http://docs.splunk.com/Documentation/Splunk/9.3.1/Data/IFXandHEC).\n- `remove_indexed_fields` (boolean, optional, default: `false`) - if indexed fields should be removed from the `event` object.\n- `batch_size` (int. optional, default: 50) - the size of the buffer to collect payload before sending to Splunk.\n\n#### Reading from Splunk\n\nThe data source supports both batch and streaming reads from Splunk using the [Splunk export API](https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs.2Fexport). You can execute SPL (Search Processing Language) queries and bring results into Spark for further analysis.\n\n\u003e [!NOTE]\n\u003e This is not real-time event streaming. The export API returns results for bounded time windows; streaming mode implements micro-batch polling by repeatedly querying with advancing time ranges.\n\n**Authentication Requirements:**\n\nReading from Splunk requires authentication to the Splunk management API (splunkd). You need:\n\n- **Splunk management API URL** (`splunkd_url`): The base URL for Splunk's REST API, typically `https://\u003csplunk-host\u003e:8089`\n- **Authentication token** (`splunkd_token`): A Splunk authentication token for API access (NOT the same as HEC token used for writing)\n  - Can also be provided via `SPLUNK_AUTH_TOKEN` environment variable\n\n##### Batch Read\n\nBatch read usage with full SPL query:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(SplunkDataSource)\n\n# Full SPL query (recommended for complex queries)\nread_options = {\n    \"splunkd_url\": \"https://splunk.example.com:8089\",\n    \"splunkd_token\": \"your-splunkd-token\",\n    \"query\": \"search index=main error | stats count by host\",\n    \"timespan\": \"P1D\",  # Last 1 day\n}\n\ndf = spark.read.format(\"splunk\") \\\n    .options(**read_options) \\\n    .load()\n\ndf.show()\n```\n\nBatch read with simple parameters (for basic searches):\n\n```python\n# Simple parameters mode (for basic searches only)\nread_options = {\n    \"splunkd_url\": \"https://splunk.example.com:8089\",\n    \"splunkd_token\": \"your-splunkd-token\",\n    \"index\": \"security\",\n    \"sourcetype\": \"firewall\",\n    \"search_filter\": \"status=200\",\n    \"start_time\": \"2024-01-01T00:00:00Z\",\n    \"end_time\": \"2024-01-02T00:00:00Z\",\n    \"num_partitions\": \"4\",  # Parallel reading\n}\n\ndf = spark.read.format(\"splunk\") \\\n    .options(**read_options) \\\n    .load()\n```\n\nSupported batch read options:\n\n**Required:**\n- `splunkd_url` (string, required) - Splunk management API URL (e.g., `https://splunk.example.com:8089`)\n- `splunkd_token` (string, required) - Splunk authentication token (or set `SPLUNK_AUTH_TOKEN` env var)\n\n**Query options (choose one approach):**\n- **Full SPL query** (preferred for complex queries):\n  - `query` (string) - Complete SPL query (e.g., `\"search index=main error | stats count by status\"`)\n- **Simple parameters** (for basic searches only - cannot use pipes or transforming commands):\n  - `index` (string, required if no `query`) - Splunk index name (alphanumeric, underscore, hyphen only)\n  - `sourcetype` (string, optional) - Source type filter (alphanumeric, colon, underscore, hyphen)\n  - `search_filter` (string, optional) - Additional filters (cannot contain pipes `|` or commands)\n\n**Time range options (choose one approach):**\n- `timespan` (string) - ISO 8601 duration (e.g., `\"P1D\"` = 1 day, `\"PT6H\"` = 6 hours, `\"P7D\"` = 7 days)\n- OR `start_time` (string) - Start time in ISO 8601 format (e.g., `\"2024-01-01T00:00:00Z\"`)\n  - `end_time` (string, optional) - End time in ISO 8601 format (defaults to current time if not specified)\n\n**Partitioning options:**\n- `num_partitions` (int, optional, default: 1) - Number of parallel partitions for reading\n- `partition_duration` (int, optional) - Partition duration in seconds (takes precedence over `num_partitions`)\n\n**Schema options:**\n- `inferSchema` (bool, optional, default: false) - Infer schema from sample data (queries with `| head 10`)\n- `mode` (string, optional, default: `\"FAILFAST\"`) - Error handling mode:\n  - `\"FAILFAST\"` - Raise exception on first type conversion error\n  - `\"PERMISSIVE\"` - Set field to null on conversion error, continue processing\n\n**Connection options:**\n- `auth_scheme` (string, optional, default: `\"Splunk\"`) - Authorization scheme (`\"Splunk\"` or `\"Bearer\"`)\n- `verify_ssl` (string/bool, optional, default: `\"true\"`) - TLS certificate verification:\n  - `\"true\"` - Verify using system CA bundle\n  - `\"false\"` - Disable verification (insecure, not recommended)\n  - Path string (e.g., `\"/path/to/ca-bundle.crt\"`) - Use custom CA bundle\n- `connect_timeout` (int, optional, default: 10) - Connect timeout in seconds\n- `read_timeout` (int, optional, default: 300) - Read timeout in seconds\n- `max_retries` (int, optional, default: 3) - Maximum retry attempts\n- `initial_backoff` (float, optional, default: 1.0) - Initial backoff for retries in seconds\n- `output_mode` (string, optional, default: `\"json\"`) - Splunk response format (`\"json\"` or `\"json_rows\"`)\n\n**SPL Query Examples:**\n\n```python\n# Get recent firewall logs\nquery = \"search index=security sourcetype=firewall | fields _time, src_ip, dest_ip, action\"\n\n# Search for errors with stats\nquery = \"search index=main error | stats count by host, source\"\n\n# Complex query with multiple commands\nquery = \"search index=web status\u003e=400 | eval hour=strftime(_time, \\\"%H\\\") | stats count by hour\"\n```\n\n**Default Schema:**\n\nIf you don't specify a schema, Splunk reader returns a default schema with common Splunk fields:\n- `_time` (TimestampType) - Event timestamp\n- `_indextime` (TimestampType) - Index timestamp\n- `_raw` (StringType) - Raw event data\n- `host` (StringType) - Host field\n- `source` (StringType) - Source field\n- `sourcetype` (StringType) - Sourcetype field\n- `index` (StringType) - Index name\n\nFor custom fields or typed fields, provide an explicit schema or use `inferSchema=true`.\n\n**Security Notes:**\n\n- Simple parameter mode validates inputs to prevent injection attacks\n- For queries with pipes, stats, eval, etc., use the `query` option (simple mode rejects these)\n- Tokens are never logged and can be read from environment variables\n\n##### Streaming Read\n\nThe data source supports streaming reads (micro-batch polling) from Splunk. The streaming reader uses `_indextime`-based offsets to track progress.\n\n\u003e [!IMPORTANT]\n\u003e Streaming mode only works with **event-returning queries** (non-transforming SPL). Transforming commands like `stats`, `chart`, `timechart`, etc. produce aggregated results without `_indextime` and will cause errors.\n\nStreaming read usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(SplunkDataSource)\n\nstream_options = {\n    \"splunkd_url\": \"https://splunk.example.com:8089\",\n    \"splunkd_token\": \"your-splunkd-token\",\n    \"query\": \"search index=security sourcetype=firewall\",  # Event-returning query only\n    \"start_time\": \"latest\",  # Start from current time\n    \"partition_duration\": \"3600\",  # 1 hour partitions\n    \"safety_lag_seconds\": \"60\",  # 60 second lag for late events\n    \"checkpointLocation\": \"/tmp/splunk-stream-checkpoint/\",\n}\n\nstream_df = spark.readStream.format(\"splunk\") \\\n    .options(**stream_options) \\\n    .load()\n\n# Write to console\nquery = stream_df.writeStream \\\n    .format(\"console\") \\\n    .trigger(availableNow=True) \\\n    .option(\"checkpointLocation\", \"/tmp/splunk-stream-checkpoint/\") \\\n    .start()\n\nquery.awaitTermination()\n```\n\nSupported streaming read options:\n\n**Required:**\n- `splunkd_url` (string, required) - Splunk management API URL\n- `splunkd_token` (string, required) - Splunk authentication token\n- `query` or `index` (string, required) - SPL query or index name (must be event-returning, no transforming commands)\n\n**Streaming-specific options:**\n- `start_time` (string, optional, default: `\"latest\"`) - Start time:\n  - `\"latest\"` - Start from current time\n  - ISO 8601 timestamp (e.g., `\"2024-01-01T00:00:00Z\"`) - Start from specific time\n- `partition_duration` (int, optional, default: 3600) - Partition duration in seconds (controls parallelism)\n- `safety_lag_seconds` (int, optional, default: 60) - Lag behind current time for late-arriving events\n- `allow_transforming_queries` (bool, optional, default: false) - Allow transforming queries (NOT recommended)\n\n**Important notes for streaming:**\n\n- The reader tracks `_indextime` (when Splunk indexed the event) as offset for reliable progression\n- Queries must return raw events with `_indextime` field - transforming commands (stats, chart, etc.) are rejected\n- Use `start_time: \"latest\"` to begin from current time (useful for monitoring new events)\n- The `safety_lag_seconds` helps handle late-arriving events that may be indexed after their event time\n\n**Query Validation:**\n\nStreaming mode automatically validates queries and rejects transforming commands:\n\n```python\n# ✅ Valid for streaming (event-returning)\nquery = \"search index=main error\"\nquery = \"search index=security | fields _time, src_ip, dest_ip\"\n\n# ❌ Invalid for streaming (transforming)\nquery = \"search index=main | stats count by host\"  # Will raise error\nquery = \"search index=main | chart count by _time\"  # Will raise error\n```\n\nTo bypass validation (not recommended), set `allow_transforming_queries: \"true\"`.\n\n### Microsoft Sentinel / Azure Monitor\n\nThis data source supports both reading from and writing to [Microsoft Sentinel](https://learn.microsoft.com/en-us/azure/sentinel/overview/) / [Azure Monitor Log Analytics](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-overview). Registered data source names are `ms-sentinel` and `azure-monitor`.\n\n#### Authentication Requirements\n\nThis connector uses Azure Service Principal Client ID/Secret for authentication.\n\nThe service principal needs the following permissions:\n\n- For reading: **Log Analytics Reader** role on the Log Analytics workspace\n- For writing: **Monitoring Metrics Publisher** role on the DCE and DCR\n\nAuthentication options:\n\n- `tenant_id` (string, required) - Azure Tenant ID\n- `client_id` (string, required) - Application ID (client ID) of Azure Service Principal\n- `client_secret` (string, required) - Client Secret of Azure Service Principal\n- `azure_cloud` (string, optional, default: \"public\") - Azure cloud environment. Valid values:\n  - `\"public\"` - Azure Public Cloud (default)\n  - `\"government\"` - Azure Government (GovCloud)\n  - `\"china\"` - Azure China (21Vianet)\n\n#### Writing to Microsoft Sentinel / Azure Monitor\n\nThe integration uses [Logs Ingestion API of Azure Monitor](https://learn.microsoft.com/en-us/azure/sentinel/create-custom-connector#connect-with-the-log-ingestion-api) for writing data.\n\nTo push data you need to create Data Collection Endpoint (DCE), Data Collection Rule (DCR), and create a custom table in Log Analytics workspace.  See [documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) for description of this process.  The structure of the data in DataFrame should match the structure of the defined custom table.\n\nYou need to grant correct permissions (`Monitoring Metrics Publisher`) to the service principal on the DCE and DCR.\n\nBatch write usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(MicrosoftSentinelDataSource)\n\nsentinel_options = {\n    \"dce\": dc_endpoint,\n    \"dcr_id\": dc_rule_id,\n    \"dcs\": dc_stream_name,\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n  }\n\ndf = spark.range(10)\ndf.write.format(\"ms-sentinel\") \\\n  .mode(\"overwrite\") \\\n  .options(**sentinel_options) \\\n  .save()\n```\n\nStreaming write usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(MicrosoftSentinelDataSource)\n\ndir_name = \"tests/samples/json/\"\nbdf = spark.read.format(\"json\").load(dir_name)  # to infer schema - not use in the prod!\n\nsdf = spark.readStream.format(\"json\").schema(bdf.schema).load(dir_name)\n\nsentinel_stream_options = {\n    \"dce\": dc_endpoint,\n    \"dcr_id\": dc_rule_id,\n    \"dcs\": dc_stream_name,\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n    \"checkpointLocation\": \"/tmp/sentinel-checkpoint/\"\n}\n\nstream = sdf.writeStream.format(\"ms-sentinel\") \\\n  .trigger(availableNow=True) \\\n  .options(**sentinel_stream_options).start()\n```\n\nSupported write options:\n\n- `dce` (string, required) - URL of the Data Collection Endpoint.\n- `dcr_id` (string, required) - ID of Data Collection Rule.\n- `dcs` (string, required) - name of custom table created in the Log Analytics Workspace.\n- `batch_size` (int. optional, default: 50) - the size of the buffer to collect payload before sending to MS Sentinel.\n\n#### Reading from Microsoft Sentinel / Azure Monitor\n\nThe data source supports both batch and streaming reads from Azure Monitor / Log Analytics workspaces using KQL (Kusto Query Language) queries.  If schema isn't specified with `.schema`, it will be inferred automatically.\n\n\u003e [!NOTE]\n\u003e For streaming reads of big amounts of data, it's recommended to export necessary tables to EventHubs, and consume from there.\n\n##### Batch Read\n\nBatch read usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(AzureMonitorDataSource)\n\n# Option 1: Using timespan (ISO 8601 duration)\nread_options = {\n    \"workspace_id\": \"your-workspace-id\",\n    \"query\": \"AzureActivity | where TimeGenerated \u003e ago(1d) | take 100\",\n    \"timespan\": \"P1D\",  # ISO 8601 duration: 1 day\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n}\n\n# Option 2: Using start_time and end_time (ISO 8601 timestamps)\nread_options = {\n    \"workspace_id\": \"your-workspace-id\",\n    \"query\": \"AzureActivity | take 100\",\n    \"start_time\": \"2024-01-01T00:00:00Z\",\n    \"end_time\": \"2024-01-02T00:00:00Z\",\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n}\n\n# Option 3: Using only start_time (end_time defaults to current time)\nread_options = {\n    \"workspace_id\": \"your-workspace-id\",\n    \"query\": \"AzureActivity | take 100\",\n    \"start_time\": \"2024-01-01T00:00:00Z\",  # Query from start_time to now\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n}\n\ndf = spark.read.format(\"azure-monitor\") \\\n    .options(**read_options) \\\n    .load()\n\ndf.show()\n```\n\nSupported read options:\n\n- `workspace_id` (string, required) - Log Analytics workspace ID\n- `query` (string, required) - KQL query to execute (could be just a table name)\n- **Time range options (choose one approach):**\n  - `timespan` (string) - Time range in ISO 8601 duration format (e.g., \"P1D\" = 1 day, \"PT1H\" = 1 hour, \"P7D\" = 7 days)\n  - `start_time` (string) - Start time in ISO 8601 format (e.g., \"2024-01-01T00:00:00Z\"). If provided without `end_time`, queries from `start_time` to current time\n  - `end_time` (string, optional) - End time in ISO 8601 format. Only valid when `start_time` is specified\n  - **Note**: `timespan` and `start_time/end_time` are mutually exclusive - choose one approach\n- `num_partitions` (int, optional, default: 1) - Number of partitions for reading data\n- `inferSchema` (bool, optional, default: true) - if we do the schema inference by sampling result.\n- `max_retries` (int, optional, default: 5) - Maximum retry attempts for HTTP 429 throttling errors\n- `initial_backoff` (float, optional, default: 1.0) - Initial backoff time in seconds for retries (uses exponential backoff)\n- `min_partition_seconds` (int, optional, default: 60) - Minimum partition duration in seconds when subdividing large result sets\n\n**KQL Query Examples:**\n\n```python\n# Get recent Azure Activity logs\nquery = \"AzureActivity | where TimeGenerated \u003e ago(24h) | project TimeGenerated, OperationName, ResourceGroup\"\n\n# Get security alerts\nquery = \"SecurityAlert | where TimeGenerated \u003e ago(7d) | project TimeGenerated, AlertName, Severity\"\n\n# Custom table query\nquery = \"MyCustomTable_CL | where TimeGenerated \u003e ago(1h)\"\n```\n\n**Azure Sovereign Clouds:**\n\nFor Azure Government or Azure China environments, use the `azure_cloud` option:\n\n```python\n# Azure Government (GovCloud)\nread_options = {\n    \"workspace_id\": \"your-workspace-id\",\n    \"query\": \"SecurityEvent | take 100\",\n    \"timespan\": \"P1D\",\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n    \"azure_cloud\": \"government\",  # Uses login.microsoftonline.us and api.loganalytics.us\n}\n\n# Azure China (21Vianet)\nread_options = {\n    # ... other options ...\n    \"azure_cloud\": \"china\",  # Uses login.chinacloudapi.cn and api.loganalytics.azure.cn\n}\n```\n\n**Automatic Throttling and Large Result Set Handling:**\n\nThe connector automatically handles common Azure Monitor query issues:\n\n- **Throttling (HTTP 429)**: When Azure Monitor returns rate limit errors, the connector automatically retries with exponential backoff. Configure with `max_retries` and `initial_backoff` options.\n\n- **Large Result Sets**: When a query exceeds Azure's [result size limits](https://learn.microsoft.com/en-us/azure/azure-monitor/service-limits#query-api) (500,000 records or ~64MB), the connector automatically subdivides the time range into smaller chunks and queries each separately. Configure the minimum subdivision size with `min_partition_seconds`.\n\n##### Streaming Read\n\nThe data source supports streaming reads from Azure Monitor / Log Analytics. The streaming reader uses time-based offsets to track progress and splits time ranges into partitions for parallel processing.\n\nStreaming read usage:\n\n```python\nfrom cyber_connectors import *\nspark.dataSource.register(AzureMonitorDataSource)\n\n# Stream from a specific timestamp\nstream_options = {\n    \"workspace_id\": \"your-workspace-id\",\n    \"query\": \"AzureActivity | project TimeGenerated, OperationName, ResourceGroup\",\n    \"start_time\": \"2024-01-01T00:00:00Z\",  # Start streaming from this timestamp\n    \"tenant_id\": tenant_id,\n    \"client_id\": client_id,\n    \"client_secret\": client_secret,\n    \"partition_duration\": \"3600\",  # Optional: partition size in seconds (default 1 hour)\n}\n\n# Read stream\nstream_df = spark.readStream.format(\"azure-monitor\") \\\n    .options(**stream_options) \\\n    .load()\n\n# Write to console or another sink\nquery = stream_df.writeStream \\\n    .format(\"console\") \\\n    .trigger(availableNow=True) \\\n    .option(\"checkpointLocation\", \"/tmp/azure-monitor-checkpoint/\") \\\n    .start()\n\nquery.awaitTermination()\n```\n\nSupported streaming read options:\n\n- `workspace_id` (string, required) - Log Analytics workspace ID\n- `query` (string, required) - KQL query to execute (could be just a table name). Note: *it should not include time filters - these are added automatically!*\n- `start_time` (string, optional, default: \"latest\") - Start time in ISO 8601 format (e.g., \"2024-01-01T00:00:00Z\"). Use \"latest\" to start from the current time\n- `partition_duration` (int, optional, default: 3600) - Duration in seconds for each partition (controls parallelism)\n\n**Important notes for streaming:**\n\n- The reader automatically tracks the timestamp of the last processed data in checkpoints\n- Time ranges are split into partitions based on `partition_duration` for parallel processing\n- The query should NOT include time filters (e.g., `where TimeGenerated \u003e ago(1d)`) - the reader adds these automatically based on offsets\n- Use `start_time: \"latest\"` to begin streaming from the current time (useful for monitoring real-time data)\n\n### Simple REST API\n\nRight now only implements writing to arbitrary REST API - both batch \u0026 streaming.  Registered data source name is `rest`.\n\nBasic usage:\n\n```python\nfrom cyber_connectors import *\n\nspark.dataSource.register(RestApiDataSource)\n\ndf = spark.range(10)\ndf.write.format(\"rest\").mode(\"overwrite\") \\\n  .option(\"url\", \"http://localhost:8001/\") \\\n  .save()\n```\n\nUsage with authentication and custom headers:\n\n```python\ndf.write.format(\"rest\").mode(\"overwrite\") \\\n  .option(\"url\", \"http://api.example.com/data\") \\\n  .option(\"http_header_Authorization\", \"Bearer token123\") \\\n  .option(\"http_header_X-API-Key\", \"secret\") \\\n  .save()\n```\n\nUsage with form data:\n\n```python\ndf.write.format(\"rest\").mode(\"overwrite\") \\\n  .option(\"url\", \"http://api.example.com/form\") \\\n  .option(\"http_format\", \"form-data\") \\\n  .option(\"http_method\", \"post\") \\\n  .save()\n```\n\nSupported options:\n\n- `url` (string, required) - URL of the REST API endpoint to send data to.\n- `http_format` (string, optional, default: `json`) - Payload format to use. Supported values:\n  - `json` - Send data as JSON (sets `Content-Type: application/json`)\n  - `form-data` - Send data as form-encoded data (all values converted to strings)\n- `http_method` (string, optional, default: `post`) - HTTP method to use (`post` or `put`).\n- `http_header_*` (string, optional) - Custom HTTP headers. Use prefix `http_header_` followed by the header name.\n  - Example: `http_header_Authorization`, `http_header_X-API-Key`, `http_header_Content-Type`\n  - Custom headers take precedence over default headers (e.g., you can override `Content-Type` for special API requirements)\n\n**Using with Tines webhook:**\n\nThis data source can be easily used to write to Tines webhook. Just specify [Tines webhook URL](https://www.tines.com/docs/actions/types/webhook/#secrets-in-url) as `url` option:\n\n```python\ndf.write.format(\"rest\").mode(\"overwrite\") \\\n  .option(\"url\", \"https://tenant.tines.com/webhook/\u003cpath\u003e/\u003csecret\u003e\") \\\n  .save()\n```\n\n**Custom Content-Type example:**\n\nSome APIs require specific Content-Type headers:\n\n```python\ndf.write.format(\"rest\").mode(\"overwrite\") \\\n  .option(\"url\", \"http://api.example.com/jsonapi\") \\\n  .option(\"http_format\", \"json\") \\\n  .option(\"http_header_Content-Type\", \"application/vnd.api+json\") \\\n  .save()\n```\n\n## Installation\n\nJust install the package from PyPI:\n\n```shell\npip install cyber-spark-data-connectors\n```\n\n## Building\n\nThis project uses [Poetry](https://python-poetry.org/) to manage dependencies and building the package. \n\nInitial setup \u0026 build:\n\n- Install Poetry\n- Set the Poetry environment with `poetry env use 3.10` (or higher Python version)\n- Activate Poetry environment with `. $(poetry env info -p)/bin/activate`\n- Build the wheel file with `poetry build`. Generated file will be stored in the `dist` directory.\n\n\u003e [!CAUTION]\n\u003e Right now, some dependencies aren't included into manifest, so if you will try it with OSS Spark, you will need to make sure that you have following dependencies set: `pyspark[sql]` (version `4.0.0.dev2` or higher), `grpcio` (`\u003e=1.48,\u003c1.57`), `grpcio-status` (`\u003e=1.48,\u003c1.57`), `googleapis-common-protos` (`1.56.4`).\n\n## References\n\n- Splunk: [Format events for HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/9.3.1/Data/FormateventsforHTTPEventCollector)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexott%2Fcyber-spark-data-connectors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexott%2Fcyber-spark-data-connectors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexott%2Fcyber-spark-data-connectors/lists"}