{"id":46211889,"url":"https://github.com/buoyant-data/hotdog","last_synced_at":"2026-03-03T09:17:03.615Z","repository":{"id":38330464,"uuid":"256028474","full_name":"buoyant-data/hotdog","owner":"buoyant-data","description":"Hotdog is a syslog-to-Kafka forwarder which aims to get log entries into Apache Kafka as quickly as possible.","archived":false,"fork":false,"pushed_at":"2025-05-02T17:15:01.000Z","size":523,"stargazers_count":47,"open_issues_count":5,"forks_count":8,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-10-14T11:17:26.778Z","etag":null,"topics":["async","kafka","rust","syslog"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/buoyant-data.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-15T20:25:25.000Z","updated_at":"2025-10-11T23:52:25.000Z","dependencies_parsed_at":"2025-04-29T19:36:58.108Z","dependency_job_id":"061545fc-d080-4399-993f-d820cc35c99d","html_url":"https://github.com/buoyant-data/hotdog","commit_stats":null,"previous_names":["buoyant-data/hotdog"],"tags_count":23,"template":false,"template_full_name":null,"purl":"pkg:github/buoyant-data/hotdog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buoyant-data%2Fhotdog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buoyant-data%2Fhotdog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buoyant-data%2Fhotdog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buoyant-data%2Fhotdog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/buoyant-data","download_url":"https://codeload.github.com/buoyant-data/hotdog/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/buoyant-data%2Fhotdog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30038872,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-03T06:58:30.252Z","status":"ssl_error","status_checked_at":"2026-03-03T06:58:15.329Z","response_time":61,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","kafka","rust","syslog"],"created_at":"2026-03-03T09:16:59.855Z","updated_at":"2026-03-03T09:17:03.604Z","avatar_url":"https://github.com/buoyant-data.png","language":"Rust","readme":"ifdef::env-github[]\n:tip-caption: :bulb:\n:note-caption: :information_source:\n:important-caption: :heavy_exclamation_mark:\n:caution-caption: :fire:\n:warning-caption: :warning:\nendif::[]\n:toc: macro\n\n= 🌭 Hotdog!\n\nHotdog is a syslog-to-Kafka forwarder which aims to get log entries into\nlink:https://kafka.apache.org[Apache Kafka]\nas quickly as possible.\n\nIt listens for syslog messages over plaintext or TLS-encrypted TCP connection\nand depending on the defined \u003c\u003crules\u003e\u003e it will route and even modify messages\non their way into a \u003c\u003cyml-kafka, configured Kafka broker\u003e\u003e.\n\ntoc::[]\n\n== Features\n\n* syslog over plaintext or TLS-encrypted TCP connections.\n* \u003c\u003crules\u003e\u003e and \u003c\u003cactions\u003e\u003e for matching, modifying, and routing syslog\n  messages based on the message content.\n* Rich integration with Kafka with \u003c\u003cyml-kafka-conf, full configuration passthrough\u003e\u003e for\n  link:https://github.com/edenhill/librdkafka[librdkafka]\n* Built-in \u003c\u003cyml-metrics, metrics integration\u003e\u003e for daemon health reporting.\n\n[source,bash]\n----\nHotdog 1.0.0\nR Tyler Croy \u003crtyler@buoyantdata.com\u003e\nForward syslog with ease\n\nUSAGE:\n    hotdog [OPTIONS]\n\nFLAGS:\n    -h, --help       Prints help information\n    -V, --version    Prints version information\n\nOPTIONS:\n    -c, --config \u003cFILE\u003e       Sets a custom config file [default: hotdog.yml]\n    -t, --test \u003cTEST_FILE\u003e    Test a log file against the configured rules\n----\n\n[[install]]\n== Installation\n\nHotdog can be installed by grabbing a\nlink:https://github.com/reiseburo/hotdog/releases[released binary].\nThe system which will run `hotdog` *must* have `libsasl2` installed, e.g.:\n\n.Ubuntu\n[source,bash]\n----\nsudo apt-get install libsasl2-2\n----\n\n.openSUSE\n[source,bash]\n----\nsudo zypper install cyrus-sasl-devel\n----\n\n[[performance]]\n=== Performance\n\nBy default `hotdog` will run with a single background thread for processing\nincoming messages. It is recommended to set `SMOL_THREADS` to the number of\nCPUs which should be utilized on the machine.\n\n[[configuration]]\n== Configuration\n\nHotdog is configured by the `hotdog.yml` file, which has a very fluid syntax at\nthe moment. The two main sections are the `global` and `rules` blocks.\n\nRules defined in the configuration can be tested against an example log file in\norder to verify that the right rules are matching the expected log inputs, for\nexample:\n\n[source,bash]\n----\n❯ RUST_LOG=info ./target/debug/hotdog -t example.log\nLine 1 matches on:\n         - Regex: ^hello\\s+(?P\u003cname\u003e\\w+)?\n         - Regex: .*\nLine 2 matches on:\n         - Regex: .*\nLine 3 matches on:\n         - Regex: .*\nLine 4 matches on:\n         - JMESPath: meta.topic\n         - Regex: .*\n----\n\n[[global]]\n=== Global\n\nThe `global` configuration configures `hotdog` itself. The \u003c\u003cyml-listen,\n`listen`\u003e\u003e, \u003c\u003cyml-kafka, `kafka`\u003e\u003e, and \u003c\u003cyml-metrics, `metrics`\u003e\u003e keys are all\nrequired by default in order for `hotdog` to start properly.\n\n[[yml-listen]]\n==== Listen\n\nThe `global.listen` configuration is required and will determine on which\naddress and port `hotdog` will listen. The \u003c\u003cyml-listen-tls, `tls`\u003e\u003e\nconfiguration key is required to function as well. When `tls` is left blank,\n`hotdog` will listen for syslog messages in plaintext on the specified `port`.\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  listen:\n    address: '127.0.0.1'\n    port: 1514\n    tls:\n----\n\n[[yml-listen-tls]]\n===== TLS\n\nThe `global.listen.tls` configuration section can be used to enable\nsyslog-over-TLS support from `hotdog`. Currently the only two valid keys for\nthis section are `cert` and `key`, both of which should be absolute or relative\npaths to PEM-encoded files on disk.\n\nCertificate and Key files can be created with `certtool --generate-privkey\n--outfile ca-key.pem`\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  listen:\n    tls:\n      cert: './a/path.crt'\n      key: './a/path.key'\n      # ca is optional and when provided will ensure certificate validation\n      # happens\n      ca: './a/ca.crt'\n----\n\n\n[[yml-status]]\n==== Status\n\nThe `global.status` is an optional configuration entry which will enable the\nlaunching of an HTTP status server on the specified `addresss` and `port`.\n\nJSON formatted statistics can be retrieved on `/stats`.\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  status:\n    address: '127.0.0.1'\n    port: 8585\n----\n\n[[yml-kafka]]\n==== Kafka\n\nA `global.kafka` configuration is required in order for `hotdog` to function\nproperly. The two main configuration values are \u003c\u003cyml-kafka-conf, `conf`\u003e\u003e and \u003c\u003cyml-kafka-topic, `topic`\u003e\u003e.\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  kafka:\n    conf:\n      bootstrap.servers: 'localhost:9092'\n      client.id: 'hotdog'\n    topic: 'logs'\n----\n\n[[yml-kafka-buffer]]\n===== Buffer\n\n**Default:** `1024`\n\n`global.kafka.buffer` may contain a number indicating the size of the internal\nqueue for sending messages to Kafka. This queue represents the number of\ninternal messages `hotdog` will buffer during Kafka availability issues.\n\nThis value is *not* the same as the librdkafka `queue.buffering.max.messages`\nconfiguration, which governs the number of in-flight messages which can be sent\nat any given time to the Kafka broker(s). To set that variable, include it in\nthe \u003c\u003cyml-kafka-conf\u003e\u003e section documented below.\n\n[CAUTION]\n====\nIf the internal Kafka queue has been filled up, new log lines received by\n`hotdog` will be discarded.\n====\n\n[[yml-kafka-conf]]\n===== Conf\n\n`global.kafka.conf` should contain a map of\nlink:https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md[librdkafka configuration values].\n`hotdog` will expect every key _and_ value to be a String. These configuration\nvalues are passed right on to the underlying librdkafka client connection, so\nwhatever librdkafka supports, `hotdog` supports!\n\n[[yml-kafka-timeout_ms]]\n===== timeout_ms\n\n**Default:** `30_000`\n\n`global.kafka.timeout_ms` is an optional configuration which defines the\ntimeout in milliseconds for `hotdog` to make an initial connection to the\nconfigured Kafka brokers.\n\n[[yml-kafka-topic]]\n===== Topic\n\n`global.kafka.topic` may contain a string value which is to be considered the\n\"default topic\" for the \u003c\u003caction-forward, Forward action\u003e\u003e.\n\n\n[[yml-parquet]]\n==== Parquet\n\nThe link:https://parquet.apache.org[Apache Parquet] sink allows for directly\nwriting to an\nlink:https://docs.rs/object_store/latest/object_store/index.html[object_store]\nsupported `url`\n\n[source,yaml]\n----\nglobal:\n  parquet:\n    url: 's3://hotdog/streams/'\n    # Bytes to buffer\n    buffer: 1024000\n    flush_ms: 60000\n----\n\n[TIP]\n====\nThe `url` can be omitted from the configuration and specified in the environment via `S3_OUTPUT_URL`\n====\n\n[[yml-metrics]]\n==== Metrics\n\nThe `global.metrics` configuration tells `hotdog` where to send its own\ninternal metrics  The only _currently_ supported metrics format is\nlink:https://github.com/statsd/statsd[statsd].\n\nIf your environment doesn't use statsd or you do not wish to report metrics,\nset the `statsd` value to an invalid host and port.\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  metrics:\n    statsd: 'localhost:8125'\n----\n\n\n[[yml-status]]\n==== Status\n\nThe `global.status` configuration is fully _optional_ but when it is enabled `hotdog`\nwill spin up an HTTP server on the configured `address` and `port` in order to provide\nreal-time status information about the daemon's runtime to HTTP clients.\n\n\n.hotdog.yml\n[source,yaml]\n----\nglobal:\n  status:\n    address: 'localhost'\n    port: 8585\n----\n\n[[rules]]\n=== Rules\n\nHotdog's rules define how it should handle and route the syslog messages it\nreceives. In the `hotdog.yml`, the rules must be defined as an array of maps.\n\nEach rule is expected to a \"matcher\" (either \u003c\u003crules-regex, `regex`\u003e\u003e or\n\u003c\u003crules-jmespath, `jmespath`\u003e\u003e), the `field`  upon which the matcher should\napply, and the \u003c\u003cactions, `actions`\u003e\u003e defining how the message should be\nhandled.\n\n.hotdog.yml\n[source,yaml]\n----\nrules:\n  - jmespath: 'meta.topic'\n    field: msg\n    actions:\n      - type: forward\n        topic: '{{value}}'\n\n  # Catch-all, send everything else to a \"logs-unknown\" topic\n  - regex: '.*'\n    field: msg\n    actions:\n      - type: forward\n        topic: 'logs-unknown'\n----\n\n.Supported Fields\n|===\n| Name | Notes\n\n| `msg`\n| The actual message sent along from the syslog server\n\n| `hostname`\n| The sender's hostname, if available.\n\n| `appname`\n| The logging application, if available, which created the syslog entry\n\n| `facility`\n| The syslog logging facility, if available, which was used to create the syslog message. For example `kern`, `user`, `auth`, etc.\n\n| `severity`\n| The severity of the syslog message, if available. For example: `notice`, `err`, `crit`, etc.\n\n|===\n\n[[rules-regex]]\n==== Matching with regular expressions\n\nThe `regex` matcher instructs `hotdog` to match the `field` against the defined\nregular expression, which must follow the syntax of the\nlink:https://docs.rs/regex/1.3.7/regex/#syntax[regex crate].\n\nThe matcher supports named groups in the regular expression, which are then exposed to actions such as\n\u003c\u003caction-merge, merge\u003e\u003e and \u003c\u003caction-replace, replace\u003e\u003e.\n\n[CAUTION]\n====\nNamed groups will **override** any built-in variables at the time of\nsubstitution, so be careful you are not naming your groups anything which might\noverlap with the built-in variable names\n====\n\n[[rules-jmespath]]\n==== Matching with JMESPath\n\n`hotdog` also supports matching on JSON based messages with\nlink:https://jmespath.org/[JMESPath] via the `jmespath` matcher. In order for a\nmatch, the log message must be a valid JSON object or array. The value of the\nmatch is also then exposed as a \u003c\u003cvariables, variable\u003e\u003e named `value`, which\ncan be used in actions such as \u003c\u003caction-merge, merge\u003e\u003e or \u003c\u003caction-replace,\nreplace\u003e\u003e.\n\n\n[[variables]]\n==== Variables\n\nSome actions, such as \u003c\u003caction-replace\u003e\u003e, can perform variable substitutions on\nlog line. The variables available are a combination of the built-in variables\nlisted below, and whatever named groups exist in the `regex` field of the\n\u003c\u003crules\u003e\u003e.\n\n[[builtin-vars]]\n.Built-in Variables\n|===\n| Name | Description\n\n| `msg`\n| The original log line message sent along from the syslog sender.\n\n| `version`\n| The version of `hotdog` which is processing the message.\n\n| `iso8601`\n| The ISO-8601 timestamp of when the message was processed.\n\n|===\n\n\n[[actions]]\n==== Actions\n\nActions determine what `hotdog` should do with the given log line when it\nreceives it.\n\n\n[[action-forward]]\n===== Forward\n\nThe forward action implies the \u003c\u003caction-stop, Stop action\u003e\u003e when used, since\nthe internally tracked `output` buffer is flushed when it is sent to Kafka.\n\n\n[[action-merge]]\n===== Merge\n\nThe `merge` action will only work when the log line is a JSON **object**. JSON\narrays, or other arbitrary strings will not merge properly, and cause **all**\nsubsequent actions for the given rule to be aborted.\n\n.Parameters\n|===\n| Key | Value\n\n| `json`\n| A YAML map which will be merged with the JSON object deserialized from the matched log line.\n\n|===\n\n.hotdog.yml\n[source,yaml]\n----\n    actions:\n      - type: merge\n        json:\n          meta:\n            hotdog:\n              version: '{{version}}'\n              timestamp: '{{iso8601}}'\n----\n\n[[action-replace]]\n===== Replace\n\nThe `template` may utilize the \u003c\u003cvariables, matched and built-in variables\u003e\u003e in\norder to generate a modified message. The output is only available to\nsubsequent actions defined _after_ the `replace` action. Subsequent rules in\nthe chain **will not** utilize this generated message.\n\n.Parameters\n|===\n| Key | Value\n\n| `template`\n| A link:https://handlebarsjs.com/[Handlebars]-style template which can be used to output a modified message.\n\n|===\n\n\n.hotdog.yml\n[source,yaml]\n----\n  - regex: '^hello\\s+(?P\u003cname\u003e\\w+)?'\n    actions:\n      - type: replace\n        template: |\n          Why hello there {{name}}!\n----\n\n[[action-stop]]\n===== Stop\n\nThe `stop` action does nothing more than stop processing on the message. It is\nnot particularly useful except in cases where `hotdog` should match on a\nmessage and then effectively discard it.\n\n\n[[metrics]]\n== Metrics\n\n`hotdog` is designed to emit Statsd metrics to the statsd endpoint configured\nin the \u003c\u003cyml-metrics\u003e\u003e section. Each metric will be prefixed under `hotdog.*`.\n\n|===\n| Key | Description\n\n| `hotdog.connections`\n| Gauge tracking the number of connections\n\n| `hotdog.lines`\n| Counter tracking the number of lines received by `hotdog`\n\n\n| `hotdog.kafka.submitted`\n| Counter tracking the number of messages submitted to Kafka\n\n| `hotdog.kafka.submitted.\u003ctopicname\u003e`\n| Counter tracking the number of messages submitted to each Kafka topic\n\n| `hotdog.kafka.producer.sent`\n| Timer which tracks the amount of time it takes to actually write messages to Kafka\n\n| `hotdog.kafka.producer.error.*`\n| Counters which count the number of different errors encountered while sending messages to Kafka. The types of possible metric names depends on the link:https://docs.rs/rdkafka/0.23.1/rdkafka/error/enum.RDKafkaError.html[RDKafkaError] enumeration from the underlying library.\n\n| `hotdog.error.log_parse`\n| Number of the log lines received which could not be parsed as link:https://tools.ietf.org/html/rfc5424[RFCC 5424] syslog lines.\n\n| `hotdog.error.full_internal_queue`\n| Count tracking the number of log lines which were *dropped* due to a full internal queue, Typically indicates an issue between `hotdog` and the Kafka brokers.\n\n| `hotdog.error.internal_push_failed`\n| Number of lines dropped because the could not be sent into the internal queue.\n\n| `hotdog.error.topic_parse_failed`\n| Number of lines dropped because the configured dynamic topic could not be parsed properly (typically indicates a configuration error).\n\n| `hotdog.error.merge_of_invalid_json`\n| Count of lines which could not have a merge action applied as configured due to a configuration error\n\n\n| `hotdog.error.merge_target_not_json`\n| Count of lines received for a merge action which were not JSON, and therefore could not be merged.\n\n|===\n\n\n[[development]]\n== Development\n\nHotdog is tested against the latest Rust stable. A simple `cargo build` should\ncompile a working `hotdog` binary for your platform.\n\n\nOn Linux systems it is easy to test with:\n\n[source,bash]\n----\nlogger --server 127.0.0.1  -T -P 1514 \"hello world\"\nlogger --server 127.0.0.1  -T -P 1514 -f example.log\n----\n\nFor TLS connections, you can use the `openssl` `s_client` command:\n\n[source,bash]\n----\necho  '\u003c13\u003e1 2020-04-18T15:16:09.956153-07:00 coconut tyler - - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"505061\"] hello world' | openssl s_client -connect localhost:6514\n----\n\n\n=== Profiling\n\nProfiling `hotdog` is best done on a Linux host with the `perf` tool, e.g.\n\n[source,bash]\n----\nRUST_LOG=info perf record --call-graph dwarf -- ./target/debug/hotdog -c ./hotdog.yml\nperf report -ng --no-inline\n----\n\nBy default this may run with a single thread, to increase the parallelism of\n:hotdog: while profiling, be sure to use the `SMOL_THREADS` environment\nvariable.\n\nThe [hotspot](https://github.com/KDAB/hotspot) profiler visualizer tool works\nwell with the generated repors.\n\n== Similar Projects\n\n`hotdog` was originally motivated by challenges with\nlink:https://github.com/rsyslog/rsyslog[rsyslog], a desire for a simple\nconfiguration, and the need for built-in metrics.\n\nSome other similar projects which can be used to get logs into Kafka:\n\n* link:https://github.com/elastic/logstash[logstash]\n* link:https://github.com/syslog-ng/syslog-ng[syslog-ng]\n* link:https://github.com/timberio/vector[vector]\n* link:https://github.com/uswitch/syslogger[syslogger], which doesn't process\n  messages itself, but rather integrates with `rsyslog`.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuoyant-data%2Fhotdog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbuoyant-data%2Fhotdog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbuoyant-data%2Fhotdog/lists"}