{"id":20066283,"url":"https://github.com/ossobv/natsomatch","last_synced_at":"2026-06-04T13:31:33.592Z","repository":{"id":222386669,"uuid":"757023087","full_name":"ossobv/natsomatch","owner":"ossobv","description":"Rust NATS suscriber that reads logs messages and publishes them in separate (hardcoded) streams for further processing","archived":false,"fork":false,"pushed_at":"2026-05-06T13:24:12.000Z","size":194,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":4,"default_branch":"develop","last_synced_at":"2026-05-06T15:18:08.438Z","etag":null,"topics":["filter","logging","nats"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ossobv.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-02-13T18:40:20.000Z","updated_at":"2026-05-06T13:08:59.000Z","dependencies_parsed_at":"2024-02-24T11:27:01.864Z","dependency_job_id":"51244311-6db4-4624-bbe9-320d3c1f7ddb","html_url":"https://github.com/ossobv/natsomatch","commit_stats":null,"previous_names":["ossobv/nats2jetstream"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ossobv/natsomatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossobv%2Fnatsomatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossobv%2Fnatsomatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossobv%2Fnatsomatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossobv%2Fnatsomatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ossobv","download_url":"https://codeload.github.com/ossobv/natsomatch/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ossobv%2Fnatsomatch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33907693,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["filter","logging","nats"],"created_at":"2024-11-13T13:55:53.961Z","updated_at":"2026-06-04T13:31:33.571Z","avatar_url":"https://github.com/ossobv.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"natsomatch (NATS o' Match)\n==========================\n\nTake *Grafana/Vector* passed log messages from *NATS JetStream*, match\ncategory and create new subjects.\n\n* Setup HOWTO:\n\n  - Make sure there is a JetStream stream to read from. Creating a\n    stream with a subjects set to the Vector subject will make it\n    collect the published message automatically:\n\n    .. code-block:: sh\n\n      nats stream add --subjects=default.nats.vector \\\n        --description='All messages from vector' \\\n        --storage=file --replicas=3 \\\n        --retention=limits --discard=old --max-bytes=20GiB --max-msgs=-1 \\\n        --max-msgs-per-subject=-1 --max-age=-1 --max-msg-size=-1 \\\n        --dupe-window=60s --no-allow-rollup --no-deny-delete \\\n        --no-deny-purge --allow-direct bulk_unfiltered\n\n  - Create a consumer which we'll use to read from:\n\n    .. code-block:: sh\n\n      nats consumer add --pull --deliver=all --ack=explicit \\\n        --replay=instant --filter= --max-deliver=-1 --max-pending=5000 \\\n        --no-headers-only --backoff=none \\\n        bulk_unfiltered natsomatch_unfiltered\n\n    *max-pending defines the max amount of unacked messages that can be\n    pending at the same time. If you set dev.max_messages_per_batch\n    on the input, you'll need max-pending at or above that number times\n    the amount of natsomatch workers.*\n\n  - Create a bunch of streams to write to, one for every possible\n    matched subject. Also create a consumer for test purposes while\n    we're at it:\n\n    .. code-block:: sh\n\n      MATCHES=$(cat lib/match/src/log_matcher.rs |\n                grep -FA2 'Ok(Match' |\n                sed -e '/subject: /!d;s/.*format!(\"bulk[.]//;s/[.].*//' |\n                sort)\n\n      for match in $MATCHES; do\n\n        nats stream add --subjects=\"bulk.$match.\u003e\" \\\n          --description=\"Bulk $match\" --storage=file --replicas=3 \\\n          --retention=limits --discard=old --max-bytes=2GiB \\\n          --max-msgs=-1 --max-msgs-per-subject=-1 \\\n          --max-age=-1 --max-msg-size=-1 --dupe-window=60s --no-allow-rollup \\\n          --no-deny-delete --no-deny-purge --allow-direct \"bulk_match_${match}\"\n\n        nats consumer add --pull --deliver=all --ack=explicit \\\n          --replay=instant --filter= --max-deliver=-1 --max-pending=0 \\\n          --no-headers-only --backoff=none \\\n          \"bulk_match_${match}\" \"bulk_match_${match}_consumer\"\n\n      done\n\n* Next, we'll need a couple of these running. Could be more or less\n  depending on how fast consumption rates are. In ``doc/`` there's a\n  *Kubernetes* setup.\n\n* Configuration can be seen below. If you're using *Kubernetes*,\n  configure it in the *ConfigMap*.\n\n* The ``/healthz`` health check and stats reporting is listening on port 3000.\n  The *Kubernetes* container uses this to check liveness.\n\n\n-------------------\nConfiguration/setup\n-------------------\n\nConfiguration for the Rust version:\n\n.. code-block:: toml\n\n    [input]\n    # Input source\n    nats.server = 'nats://10.20.30.40:4222'\n    # Server certificate validation (tls.server_name is not working)\n    tls.server_name = 'nats.local'\n    tls.ca_file = './nats_ca.crt'\n    # Client certificate\n    tls.cert_file = './nats_client.crt'\n    tls.key_file = './nats_client.key'\n    # Select manually made consumer from stream\n    nats.stream = 'bulk_unfiltered'\n    nats.consumer = 'natsomatch_unfiltered'\n\n    [sink]\n    # Output target\n    nats.server = 'nats://nats.example.com:4222'\n    nats.auth = { username = 'derek', password = 's3cr3t!' }\n    # Server certificate validation (tls.server_name is not working)\n    tls.ca_file = '/etc/ssl/certs/ca-certificates.crt'\n    # Client certificate\n    #tls.cert_file = './nats_client.crt'\n    #tls.key_file = './nats_client.key'\n    # No need to set jetstream name or subjects. The subject generation is\n    # hardcoded for now, based on the message.\n\n\n----\nTODO\n----\n\n☐  Clear (greppable) log message on startup. Clear log message on shutdown.\n\n☐  Hardcoded attributes are now in lib/json/src/payload_parser.rs. Maybe make them configurable.\n\n☐  Hardcoded matching rules are now in lib/match/src/log_matcher.rs. Maybe make them configurable.\n\n☐  See if we can add filters to remove useless messages. We'll want to check some live data here.\n\n☐  Add configurable bind address for /healthz server. Use a ping/pong test on input/sink too?\n\n☐  See if we want to rely on ghcr.io/rust-cross/rust-musl-cross ( https://github.com/rust-cross/rust-musl-cross ) or want to build something from the official images.\n\n☐  See if we want to use cargo-chef for docker layer caching (speeding up release builds).\n\n☐  Stats improvements:\n\n- Count average message length.\n- Report stats on output subscriptions (streams) so we can reorder filters for more speed.\n\n☐  Monitoring improvements:\n\n- Right now we have no easy detection of streams that are not handled quickly enough. Maybe check natsomatch_unfiltered for \"unprocessed\" counts.\n\n☐  Check and fix behaviour on NATS/JetStream disconnect/error. Consider auto-creating streams. (Where are the settings?)\n\n\n-----------------------\nBinary version and SBOM\n-----------------------\n\nThe ``git describe`` version is stored and shown on bad arguments:\n\n.. code-block:: console\n\n    $ ./target/release/natsomatch -v\n    natsomatch v0.1.0\n    Usage: ./target/release/natsomatch -c \u003cconfig-file\u003e\n\nThe built binary (if built using ``cargo auditable build``) includes a\n*Software Bill of Materials* (SBOM):\n\n.. code-block:: console\n\n    $ objcopy --dump-section .dep-v0=/dev/stdout target/release/natsomatch |\n        python3 -c 'import zlib,sys;print(zlib.decompress(sys.stdin.buffer.read()).decode(\"utf-8\"))' |\n        jq .\n    {\n      \"packages\": [\n        {\n          \"name\": \"aho-corasick\",\n          \"version\": \"1.1.2\",\n          \"source\": \"crates.io\",\n          \"dependencies\": [\n            45\n          ]\n        },\n        {\n          \"name\": \"async-nats\",\n          \"version\": \"0.33.0\",\n          \"source\": \"crates.io\",\n          \"dependencies\": [\n            3,\n    ...\n\n\n-----------\nRust idioms\n-----------\n\n* ``String vs. Box\u003cstr\u003e``: don't use ``Box\u003cstr\u003e`` to make the string\n  immutable or try to save a uint. Only use it if you have many many strings.\n  (Similarly: see ``Box\u003c[T]\u003e`` vs. ``Vec\u003cT\u003e``.)\n\n* ``into/to_string/to_owned``: ``to_string`` is to get a human\n  representation of something; ``to_owned`` is for converting a\n  ``\u0026String`` (or maybe a ``\u0026str``) to a copy/clone; ``into`` is for\n  conversion (``String`` to ``PathBuf``, ``\u0026str`` to ``String``).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossobv%2Fnatsomatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fossobv%2Fnatsomatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fossobv%2Fnatsomatch/lists"}