{"id":13779297,"url":"https://github.com/brexhq/substation","last_synced_at":"2025-05-16T08:04:33.189Z","repository":{"id":37083031,"uuid":"481983343","full_name":"brexhq/substation","owner":"brexhq","description":"Substation is a toolkit for routing, normalizing, and enriching security event and audit logs.","archived":false,"fork":false,"pushed_at":"2024-10-29T22:28:09.000Z","size":6335,"stargazers_count":326,"open_issues_count":3,"forks_count":19,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-10-29T23:48:48.291Z","etag":null,"topics":["automation","aws","logging","monitoring","observability","security"],"latest_commit_sha":null,"homepage":"https://substation.readme.io","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brexhq.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-15T14:23:32.000Z","updated_at":"2024-10-29T22:03:39.000Z","dependencies_parsed_at":"2023-10-01T19:09:57.170Z","dependency_job_id":"cb1f52d7-9dd8-4a2a-b86b-ecc7dcbdf851","html_url":"https://github.com/brexhq/substation","commit_stats":null,"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brexhq%2Fsubstation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brexhq%2Fsubstation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brexhq%2Fsubstation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brexhq%2Fsubstation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brexhq","download_url":"https://codeload.github.com/brexhq/substation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254493378,"owners_count":22080126,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","aws","logging","monitoring","observability","security"],"created_at":"2024-08-03T18:01:03.603Z","updated_at":"2025-05-16T08:04:28.180Z","avatar_url":"https://github.com/brexhq.png","language":"Go","funding_links":[],"categories":["Network","All in one solutions","Table of Contents","Recently Updated","Applications","Open Source Repos","Batch Processing"],"sub_categories":["Monitoring / Logging","Streaming Library","[Mar 15, 2025](/content/2025/03/15/README.md)","Miscellaneous Repos"],"readme":"# Substation\n\n![Substation Banner](.github/media/substation_banner.png)\n\n\u003cp align=\"center\"\u003e\u003cb\u003eSubstation is a toolkit for routing, normalizing, and enriching security event and audit logs.\u003c/b\u003e\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n[Releases][releases]\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[Documentation][docs]\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[Adopters][adopters]\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[Announcement (2022)][announcement]\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[v1.0 Release (2024)][v1_release] \n\n[![OSSF-Scorecard Score](https://img.shields.io/ossf-scorecard/github.com/brexhq/substation?style=for-the-badge)](https://scorecard.dev/viewer/?uri=github.com/brexhq/substation)\n![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/brexhq/substation/code.yml?style=for-the-badge)\n[![GitHub Release](https://img.shields.io/github/v/release/brexhq/substation?sort=semver\u0026style=for-the-badge\u0026link=https%3A%2F%2Fgithub.com%2Fbrexhq%2Fsubstation%2Freleases%2Flatest)](https://github.com/brexhq/substation/releases)\n![GitHub Created At](https://img.shields.io/github/created-at/brexhq/substation?style=for-the-badge\u0026label=created)\n[![GitHub License](https://img.shields.io/github/license/brexhq/substation?style=for-the-badge)](https://github.com/brexhq/substation/blob/main/LICENSE)\n\n\u003c/div\u003e\n\n## Quickstart\n\nWant to see a demo before diving into the documentation? Run this command:\n\n```sh\nexport PATH=$PATH:$(go env GOPATH)/bin \u0026\u0026 \\\ngo install github.com/brexhq/substation/v2/cmd/substation@latest \u0026\u0026 \\\nsubstation demo\n```\n\n## At a Glance\n\nSubstation is inspired by data pipeline systems like Logstash and Fluentd, but is built for modern security teams:\n\n- **Extensible Data Processing**: Build data processing pipeline systems and microservices using out-of-the-box applications and 100+ data transformation functions, or create your own written in Go.\n- **Route Data Across the Cloud**: Conditionally route data to, from, and between AWS cloud services, including S3, Kinesis, SQS, and Lambda, or to any HTTP endpoint.\n- **Bring Your Own Schema**: Format, normalize, and enrich event logs to comply with the Elastic Common Schema (ECS), Open Cybersecurity Schema Framework (OCSF), or any other schema.\n- **Unlimited Data Enrichment**: Use external APIs to enrich event logs affordably and at scale with enterprise and threat intelligence, or build a microservice that reduces spend in expensive security APIs.\n- **No Servers, No Maintenance**: Deploys as a serverless application in your AWS account, launches in minutes using Terraform, and requires no maintenance after deployment.\n- **Runs Almost Anywhere**: Create applications that run on most platforms supported by Go and transform data consistently across laptops, servers, containers, and serverless functions.\n- **High Performance, Low Cost**: Transform 100,000+ events per second while keeping cloud costs as low as a few cents per GB. Vendor solutions, like [Cribl](https://cribl.io/cribl-pricing/) and [Datadog](https://www.datadoghq.com/pricing/?product=observability-pipelines#products), can cost up to 10x more.\n\nAll of these data pipeline and microservice systems, and many more, can be built with Substation:\n\n![Example Substation architectures](.github/media/substation_architecture.png)\n\n## Transforming Event Logs\n\nSubstation excels at formatting, normalizing, and enriching event logs. For example, Zeek connection logs can be transformed to comply with the Elastic Common Schema:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eOriginal Event\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003eTransformed Event\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```json\n{\n  \"ts\": 1591367999.430166,\n  \"uid\": \"C5bLoe2Mvxqhawzqqd\",\n  \"id.orig_h\": \"192.168.4.76\",\n  \"id.orig_p\": 46378,\n  \"id.resp_h\": \"31.3.245.133\",\n  \"id.resp_p\": 80,\n  \"proto\": \"tcp\",\n  \"service\": \"http\",\n  \"duration\": 0.25411510467529297,\n  \"orig_bytes\": 77,\n  \"resp_bytes\": 295,\n  \"conn_state\": \"SF\",\n  \"missed_bytes\": 0,\n  \"history\": \"ShADadFf\",\n  \"orig_pkts\": 6,\n  \"orig_ip_bytes\": 397,\n  \"resp_pkts\": 4,\n  \"resp_ip_bytes\": 511\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```json\n{\n  \"event\": {\n    \"original\": {\n      \"ts\": 1591367999.430166,\n      \"uid\": \"C5bLoe2Mvxqhawzqqd\",\n      \"id.orig_h\": \"192.168.4.76\",\n      \"id.orig_p\": 46378,\n      \"id.resp_h\": \"31.3.245.133\",\n      \"id.resp_p\": 80,\n      \"proto\": \"tcp\",\n      \"service\": \"http\",\n      \"duration\": 0.25411510467529297,\n      \"orig_bytes\": 77,\n      \"resp_bytes\": 295,\n      \"conn_state\": \"SF\",\n      \"missed_bytes\": 0,\n      \"history\": \"ShADadFf\",\n      \"orig_pkts\": 6,\n      \"orig_ip_bytes\": 397,\n      \"resp_pkts\": 4,\n      \"resp_ip_bytes\": 511\n    },\n    \"hash\": \"af70ea0b38e1fb529e230d3eca6badd54cd6a080d7fcb909cac4ee0191bb788f\",\n    \"created\": \"2022-12-30T17:20:41.027505Z\",\n    \"id\": \"C5bLoe2Mvxqhawzqqd\",\n    \"kind\": \"event\",\n    \"category\": [\n      \"network\"\n    ],\n    \"action\": \"network-connection\",\n    \"outcome\": \"success\",\n    \"duration\": 254115104.675293\n  },\n  \"@timestamp\": \"2020-06-05T14:39:59.430166Z\",\n  \"client\": {\n    \"address\": \"192.168.4.76\",\n    \"ip\": \"192.168.4.76\",\n    \"port\": 46378,\n    \"packets\": 6,\n    \"bytes\": 77\n  },\n  \"server\": {\n    \"address\": \"31.3.245.133\",\n    \"ip\": \"31.3.245.133\",\n    \"port\": 80,\n    \"packets\": 4,\n    \"bytes\": 295,\n    \"domain\": \"h31-3-245-133.host.redstation.co.uk\",\n    \"top_level_domain\": \"co.uk\",\n    \"subdomain\": \"h31-3-245-133.host\",\n    \"registered_domain\": \"redstation.co.uk\",\n    \"as\": {\n      \"number\": 20860,\n      \"organization\": {\n        \"name\": \"Iomart Cloud Services Limited\"\n      }\n    },\n    \"geo\": {\n      \"continent_name\": \"Europe\",\n      \"country_name\": \"United Kingdom\",\n      \"city_name\": \"Manchester\",\n      \"location\": {\n        \"latitude\": 53.5039,\n        \"longitude\": -2.1959\n      },\n      \"accuracy\": 1000\n    }\n  },\n  \"network\": {\n    \"protocol\": \"tcp\",\n    \"bytes\": 372,\n    \"packets\": 10,\n    \"direction\": \"outbound\"\n  }\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Routing Data\n\nSubstation can route data to several destinations from a single process and, unlike most other data pipeline systems,\ndata transformation and routing are functionally equivalent -- this means that data can be transformed or routed in any order.\n\nIn this configuration, data is:\n\n- Written to AWS S3\n- Printed to stdout\n- Conditionally dropped (filtered, removed)\n- Sent to an HTTPS endpoint\n\n```jsonnet\n// The input is a JSON array of objects, such as:\n// [\n//   { \"field1\": \"a\", \"field2\": 1, \"field3\": true },\n//   { \"field1\": \"b\", \"field2\": 2, \"field3\": false },\n//   ...\n// ]\nlocal sub = import 'substation.libsonnet';\n\n// This filters events based on the value of field3.\nlocal is_false = sub.cnd.str.eq({ object: { source_key: 'field3' }, value: 'false' });\n\n{\n  transforms: [\n    // Pre-transformed data is written to an object in AWS S3 for long-term storage.\n    sub.tf.send.aws.s3({ aws: { arn: 'arn:aws:s3:::example-bucket-name' } }),\n    // The JSON array is split into individual events that go through\n    // the remaining transforms. Each event is printed to stdout.\n    sub.tf.agg.from.array(),\n    sub.tf.send.stdout(),\n    // Events where field3 is false are removed from the pipeline.\n    sub.pattern.tf.conditional(condition=is_false, transform=sub.tf.util.drop()),\n    // The remaining events are sent to an HTTPS endpoint.\n    sub.tf.send.http.post({ url: 'https://example-http-endpoint.com' }),\n  ],\n}\n```\n\nAlternatively, the data can be conditionally routed to different destinations:\n\n```jsonnet\nlocal sub = import 'substation.libsonnet';\n\n{\n  transforms: [\n    // If field3 is false, then the event is sent to an HTTPS endpoint; otherwise,\n    // the event is written to an object in AWS S3.\n    sub.tf.meta.switch({ cases: [\n      {\n        condition: sub.cnd.str.eq({ object: { source_key: 'field3' }, value: 'false' }),\n        transforms: [\n          sub.tf.send.http.post({ url: 'https://example-http-endpoint.com' }),\n        ],\n      },\n      {\n        transforms: [\n          sub.tf.send.aws.s3({ aws: { arn: 'arn:aws:s3:::example-bucket-name' } }),\n        ],\n      },\n    ] }),\n    // The event is always available to any remaining transforms.\n    sub.tf.send.stdout(),\n  ],\n}\n```\n\n## Configuring Applications\n\nSubstation applications run almost anywhere (laptops, servers, containers, serverless functions) and all transform functions behave identically regardless of where they are run. This makes it easy to develop configuration changes locally, validate them in a build (CI/CD) pipeline, and run integration tests in a staging environment before deploying to production.\n\nConfigurations are written in Jsonnet and can be expressed as functional code, simplifying version control and making it easy to build custom data processing libraries. For power users, configurations also have abbreviations that make them easier to write. Compare the configuration below to similar configurations for Logstash and Fluentd:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eSubstation\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003eLogstash\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003eFluentd\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```jsonnet\nlocal sub = import 'substation.libsonnet';\n\n{\n  transforms: [\n    sub.tf.obj.cp({ object: { source_key: 'src_field_1', target_key: 'dest_field_1' } }),\n    sub.tf.obj.cp({ obj: { src: 'src_field_2', trg: 'dest_field_2' } }),\n    sub.tf.send.stdout(),\n    sub.tf.send.http.post({ url: 'https://example-http-endpoint.com' }),\n  ],\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```ruby\ninput {\n  file {\n    path =\u003e \"/path/to/your/file.log\"\n    start_position =\u003e \"beginning\"\n    sincedb_path =\u003e \"/dev/null\"\n    codec =\u003e \"json\"\n  }\n}\n\nfilter {\n  json {\n    source =\u003e \"message\"\n  }\n\n  mutate {\n    copy =\u003e { \"src_field_1\" =\u003e \"dest_field_1\" }\n    copy =\u003e { \"src_field_2\" =\u003e \"dest_field_2\" }\n  }\n}\n\noutput {\n  stdout {\n    codec =\u003e rubydebug\n  }\n\n  http {\n    url =\u003e \"https://example-http-endpoint.com\"\n    http_method =\u003e \"post\"\n    format =\u003e \"json\"\n  }\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```xml\n\u003csource\u003e\n  @type tail\n  path /path/to/your/file.log\n  pos_file /dev/null\n  tag file.log\n  format json\n\u003c/source\u003e\n\n\u003cfilter file.log\u003e\n  @type record_transformer\n  enable_ruby\n  \u003crecord\u003e\n    dest_field_1 ${record['src_field_1']}\n    dest_field_2 ${record['src_field_2']}\n  \u003c/record\u003e\n\u003c/filter\u003e\n\n\u003cmatch file.log\u003e\n  @type copy\n  \u003cstore\u003e\n    @type stdout\n  \u003c/store\u003e\n  \u003cstore\u003e\n    @type http\n    url https://example-http-endpoint.com\n    http_method post\n    \u003cformat\u003e\n      @type json\n    \u003c/format\u003e\n  \u003c/store\u003e\n\u003c/match\u003e\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Deploying to AWS\n\nSubstation includes Terraform modules for securely deploying data pipelines and microservices in AWS. These modules are designed for ease of use, but are also flexible enough to support managing complex systems. This configuration deploys a data pipeline that is capable of receiving data from API Gateway and storing it in an S3 bucket:\n\n\u003ctable\u003e\n\u003ctr\u003e\n\u003cth\u003e\u003ccode\u003eresources.tf\u003c/code\u003e\u003c/th\u003e\n\u003cth\u003e\u003ccode\u003enode.tf\u003c/code\u003e\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd\u003e\n\n```tcl\n# These resources are deployed once and are used by all Substation infrastructure.\n\n# Substation resources can be encrypted using a customer-managed KMS key.\nmodule \"kms\" {\n  source = \"build/terraform/aws/kms\"\n\n  config = {\n    name   = \"alias/substation\"\n  }\n}\n\n# Substation typically uses AppConfig to manage configuration files, but\n# configurations can also be loaded from an S3 URI or an HTTP endpoint.\nmodule \"appconfig\" {\n  source = \"build/terraform/aws/appconfig\"\n\n  config = {\n    name = \"substation\"\n    environments = [{ name = \"example\" }]\n  }\n}\n\nmodule \"ecr\" {\n  source = \"build/terraform/aws/ecr\"\n  kms    = module.kms\n\n  config = {\n    name         = \"substation\"\n    force_delete = true\n  }\n}\n\nresource \"random_uuid\" \"s3\" {}\n\nmodule \"s3\" {\n  source = \"build/terraform/aws/s3\"\n  kms    = module.kms\n\n  config = {\n    # Bucket name is randomized to avoid collisions.\n    name = \"${random_uuid.s3.result}-substation\"\n  }\n\n  # Access is granted by providing the role name of a\n  # resource. This access applies least privilege and\n  # grants access to dependent resources, such as KMS.\n  access = [\n    # Lambda functions create unique roles that are\n    # used to access resources.\n    module.node.role.name,\n  ]\n}\n```\n\u003c/td\u003e\n\u003ctd\u003e\n\n```tcl\n# Deploys an unauthenticated API Gateway that forwards data to the node.\nmodule \"node_gateway\" {\n  source = \"build/terraform/aws/api_gateway/lambda\"\n  lambda = module.node\n\n  config = {\n    name = \"node_gateway\"\n  }\n\n  depends_on = [\n    module.node\n  ]\n}\n\nmodule \"node\" {\n  source = \"build/terraform/aws/lambda\"\n  kms       = module.kms  # Optional\n  appconfig = module.appconfig  # Optional\n\n  config = {\n    name        = \"node\"\n    description = \"Substation node that writes data to S3.\"\n    image_uri   = \"${module.ecr.url}:latest\"\n    image_arm   = true\n\n    env = {\n      \"SUBSTATION_CONFIG\" : \"https://localhost:2772/applications/substation/environments/example/configurations/node\"\n      \"SUBSTATION_DEBUG\" : true\n      # This Substation node will ingest data from API Gateway. More nodes can be\n      # deployed to ingest data from other sources, such as Kinesis or SQS.\n      \"SUBSTATION_LAMBDA_HANDLER\" : \"AWS_API_GATEWAY\"\n    }\n  }\n\n  depends_on = [\n    module.appconfig.name,\n    module.ecr.url,\n  ]\n}\n```\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## Getting Started\n\nYou can run Substation on:\n\n- [Docker](https://substation.readme.io/docs/try-substation-on-docker)\n- [macOS / Linux](https://substation.readme.io/docs/try-substation-on-macos-linux)\n- [AWS](https://substation.readme.io/docs/try-substation-on-aws)\n\n### Testing\n\nUse the Substation CLI tool to run through [examples](examples/) and unit test configurations:\n\n```sh\nsubstation test -h\n```\n\nExamples can be tested by running this command from the root of the project. For example:\n\n```sh\n% substation test -R examples/transform/time/str_conversion \n{\"time\":\"2024-01-01T01:02:03.123Z\"}\n{\"time\":\"2024-01-01T01:02:03\"}\nok\texamples/transform/time/str_conversion/config.jsonnet\t133µs\n```\n\n### Development\n\n[VS Code](https://code.visualstudio.com/docs/devcontainers/containers) is the recommended development environment for Substation. The project includes a [development container](.devcontainer/Dockerfile) that should be used to develop and test the system. Refer to the [development guide](CONTRIBUTING.md) for more information.\n\nIf you don't use VS Code, then you should run the development container from the command line:\n\n```sh\ngit clone https://github.com/brexhq/substation.git \u0026\u0026 cd substation \u0026\u0026 \\\ndocker build -t substation-dev .devcontainer/ \u0026\u0026 \\\ndocker run -v $(pwd):/workspaces/substation/  -w /workspaces/substation -v /var/run/docker.sock:/var/run/docker.sock -it substation-dev\n```\n\n### Deployment\n\nThe [Terraform documentation](build/terraform/aws/) includes guidance for deploying Substation to AWS.\n\n## Licensing\n\nSubstation and its associated code is released under the terms of the [MIT License](LICENSE).\n\n\u003c!--Links--\u003e\n[releases]:https://github.com/brexhq/substation/releases \"Substation Releases\"\n[docs]:https://substation.readme.io/docs \"Substation Documentation\"\n[adopters]:https://github.com/brexhq/substation/blob/main/ADOPTERS.md \"Substation Adopters\"\n[announcement]:https://medium.com/brexeng/announcing-substation-188d049d979b \"Substation Announcement Post\"\n[v1_release]:https://medium.com/brexeng/releasing-substation-v1-0-4d0314cbc45b \"Substation v1.0 Release Post\"\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrexhq%2Fsubstation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrexhq%2Fsubstation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrexhq%2Fsubstation/lists"}