{"id":50681170,"url":"https://github.com/mostafa/rsigma-dynamic-pipelines-article","last_synced_at":"2026-06-08T19:03:48.180Z","repository":{"id":357326381,"uuid":"1234913093","full_name":"mostafa/rsigma-dynamic-pipelines-article","owner":"mostafa","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-12T08:47:44.000Z","size":11,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-12T10:38:36.423Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://mostafa.dev/wiring-live-threat-intel-into-sigma-detection-with-dynamic-pipelines-4de29b4af7ca","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mostafa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-10T19:57:08.000Z","updated_at":"2026-05-12T08:47:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/mostafa/rsigma-dynamic-pipelines-article","commit_stats":null,"previous_names":["mostafa/rsigma-dynamic-pipelines-article"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/mostafa/rsigma-dynamic-pipelines-article","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa%2Frsigma-dynamic-pipelines-article","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa%2Frsigma-dynamic-pipelines-article/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa%2Frsigma-dynamic-pipelines-article/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa%2Frsigma-dynamic-pipelines-article/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mostafa","download_url":"https://codeload.github.com/mostafa/rsigma-dynamic-pipelines-article/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mostafa%2Frsigma-dynamic-pipelines-article/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34076003,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-08T19:03:45.831Z","updated_at":"2026-06-08T19:03:48.167Z","avatar_url":"https://github.com/mostafa.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RSigma Dynamic Pipelines: Live Threat Intel Demo\n\nCompanion repository for the article [\"Wiring Live Threat Intel into Sigma Detection with Dynamic Pipelines\"](https://mostafa.dev/wiring-live-threat-intel-into-sigma-detection-with-dynamic-pipelines-4de29b4af7ca).\n\nThis repo demonstrates RSigma [v0.10.0](https://github.com/timescale/rsigma/releases/tag/v0.10.0)'s dynamic pipeline feature by wiring two public threat intelligence sources into Sigma detection rules at runtime, without modifying the rules themselves.\n\n## What is in this repo\n\n```\npipelines/threat_intel.yml    # Dynamic pipeline with HTTP + command sources\nrules/\n  botnet_c2_connection.yml    # Sigma rule: firewall C2 IP detection\n  lummac2_dns_query.yml       # Sigma rule: DNS C2 domain detection\nscripts/extract_iocs.py       # ioc-finder wrapper for command source\nadvisories/aa25-141b.txt      # CISA AA25-141B LummaC2 advisory text\nevents/\n  firewall.jsonl              # Sample firewall events (3 match, 3 benign)\n  dns.jsonl                   # Sample DNS events (3 match, 3 benign)\n```\n\n## Data sources\n\n| Source | Type | What it provides | Refresh |\n|--------|------|-----------------|---------|\n| [Feodo Tracker](https://feodotracker.abuse.ch/blocklist/) | HTTP | Botnet C2 IPs (Emotet, Dridex, TrickBot, QakBot) | Every 5 min |\n| [CISA AA25-141B](https://www.cisa.gov/news-events/cybersecurity-advisories/aa25-141b) via [ioc-finder](https://github.com/fhightower/ioc-finder) | Command | LummaC2 C2 domains (~114 domains) | Once |\n\nBoth sources are free and require no authentication.\n\n## Prerequisites\n\n- [RSigma](https://github.com/timescale/rsigma) v0.10.0 or later\n- Python 3.9+ with `ioc-finder` installed\n\n```bash\npip install ioc-finder\n```\n\n## Quick start\n\n### 1. Inspect the resolved sources\n\nSee what the pipeline fetches from Feodo Tracker and extracts from the CISA advisory:\n\n```bash\nrsigma resolve -p pipelines/threat_intel.yml --pretty\n```\n\nInspect a single source:\n\n```bash\nrsigma resolve -p pipelines/threat_intel.yml -s c2_ips --pretty\nrsigma resolve -p pipelines/threat_intel.yml -s advisory_domains --pretty\n```\n\n### 2. Run with the daemon\n\nDynamic source resolution requires `rsigma daemon`, which uses the `rsigma-runtime` crate\nto fetch HTTP endpoints, execute commands, and manage refresh intervals. The `rsigma eval`\nCLI command operates on static pipelines only, because `rsigma-eval` is a pure synchronous\nevaluation library with no I/O dependencies.\n\nStart the daemon with the dynamic pipeline and rules:\n\n```bash\nrsigma daemon \\\n  --rules rules/ \\\n  --pipeline pipelines/threat_intel.yml \\\n  --input http \\\n  --api-addr 127.0.0.1:8080\n```\n\nThen send events to the daemon:\n\n```bash\ncurl -X POST http://127.0.0.1:8080/api/v1/events \\\n  -H \"Content-Type: application/json\" \\\n  -d @events/firewall.jsonl\n```\n\n## How it works\n\n1. The pipeline YAML declares two `sources`: an HTTP source that fetches C2 IPs from Feodo Tracker, and a command source that runs `ioc-finder` against the CISA advisory text.\n\n2. At startup, the daemon resolves both sources, extracting IP addresses and domains respectively.\n\n3. The `vars` section maps resolved data to template variables (`${source.c2_ips}`, `${source.advisory_domains}`).\n\n4. The `value_placeholders` transformation replaces `%c2_ips%` and `%advisory_domains%` in the Sigma rules with the resolved values.\n\n5. The Sigma rules never change. Detection adapts as the feeds update.\n\n## Related articles\n\n1. [Pattern Detection and Correlation in JSON Logs](https://mostafa.dev) (Feb 2026)\n2. [Streaming Logs to RSigma for Real-Time Detection](https://mostafa.dev) (Apr 2026)\n3. [Building a Detection Layer on PostgreSQL with Sigma Rules](https://mostafa.dev) (Apr 2026)\n4. [Security Observability with RSigma and the LGTM Stack](https://mostafa.dev) (May 2026)\n5. [Wiring Live Threat Intel into Sigma Detection with Dynamic Pipelines](https://mostafa.dev/wiring-live-threat-intel-into-sigma-detection-with-dynamic-pipelines-4de29b4af7ca) (this article)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafa%2Frsigma-dynamic-pipelines-article","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmostafa%2Frsigma-dynamic-pipelines-article","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmostafa%2Frsigma-dynamic-pipelines-article/lists"}