{"id":41982924,"url":"https://github.com/intelecy/jet-capture","last_synced_at":"2026-01-26T00:00:13.621Z","repository":{"id":61626158,"uuid":"538911077","full_name":"Intelecy/jet-capture","owner":"Intelecy","description":"Capture, decode, demux, and store NATS JetStream messages","archived":false,"fork":false,"pushed_at":"2024-06-12T13:01:36.000Z","size":133,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-06-21T11:10:09.028Z","etag":null,"topics":["golang","nats","nats-jetstream"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Intelecy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-20T09:37:10.000Z","updated_at":"2024-06-12T13:00:03.000Z","dependencies_parsed_at":"2024-06-12T18:57:16.598Z","dependency_job_id":null,"html_url":"https://github.com/Intelecy/jet-capture","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/Intelecy/jet-capture","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelecy%2Fjet-capture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelecy%2Fjet-capture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelecy%2Fjet-capture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelecy%2Fjet-capture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Intelecy","download_url":"https://codeload.github.com/Intelecy/jet-capture/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Intelecy%2Fjet-capture/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28761863,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-25T23:06:19.311Z","status":"ssl_error","status_checked_at":"2026-01-25T23:03:50.555Z","response_time":113,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","nats","nats-jetstream"],"created_at":"2026-01-26T00:00:12.523Z","updated_at":"2026-01-26T00:00:13.605Z","avatar_url":"https://github.com/Intelecy.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NATS JetStream Capture\n\n\u003e Note: requires Go 1.19 or later\n\n## Overview\n\n**jetcapture** is a library for building reliable and continuous [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream)\nbackup processes.\n\n* **Decode**      -- take the incoming NATS message and deserialize it into something concrete\n* **Split**       -- optionally split the incoming message stream and group the messages by one or more user-defined\n                     attributes\n* **Serialize**   -- write decoded messages to group-specific \"blocks\" using a provided serialization method (e.g. CSV,\n                     Parquet, raw-bytes)\n* **Store**       -- copy the completed blocks to a user-provided storage endpoint (e.g. local folders, Azure, etc.)\n\nFor example, you can take a stream of JSON pizza orders, group them by `pizza_store_id`, and write them out as flattened\nCSV in 15 minute blocks with a separate location (e.g. folder, or S3 bucket) for each store.\n\n**jetcapture** uses a [pull consumer](https://docs.nats.io/nats-concepts/jetstream/consumers) which means horizontal\nscalability is built in. Just add more instances to increase throughput.\n\n\n### Internal Data Flow\n\n1. **jetcapture** begins pulling messages from a named consumer and stream\n2. For each message:\n   1. Pass the raw NATS message into the user-provided **decoder**, returning a typed struct and an optional\n      \"destination key\"\n   2. Find a corresponding \"block\" using the **destination key** and the message timestamp (truncated to the storage\n      interval)\n   3. Call the user-provided **serialize** function and write the decoded message into a block-specific temporary buffer\n   4. Cache the message ack information for later\n3. After each storage interval has passed, for each block:\n   1. Call the user-provided **store** function to persist the block to a permanent storage location\n   2. Assuming storage of the block succeeded, \"ack\" all the messages in the block\n\n## Custom Implementation Requirements\n\n\u003e Note: **jetcapture** uses Go generics to enable strongly typed callback implementations\n\n1. Define your `Payload P` and `DestKey K` types\n2. Implement a `MessageDecoder` that takes a `*nats.Msg` and returns a decoded message of type `P` and a \"destination\n   key\" of type `K`\n3. Implement a `FormattedDataWriter[P Payload]` which takes a payload `P` \"writes\" it to an underlying `io.Writer`. Or,\n   use a helper writer like `CSVWriter[P Payload]` or `NewLineDelimitedJSON[P Payload]`\n4. Implement a `BlockStore[K DestKey]` which can write out the finalized \"block\" (exposed as `io.Reader`). Or, use a\n   helper like `LocalFSStore[K DestKey]` or `AzureBlobStore[K DestKey]`\n5. Create a typed `jetcapture.Options[P, K]` instance with options set\n6. Connect to a NATS server\n7. Call `options.Build().Run(ctx, natsConn)`\n\nFor a full example see the [sample application](apps/ndjson/main.go) that takes incoming NATS messages, encodes the entire message itself as\nJSON, and writes it out using newline-delimited JSON.\n\nFor an example of a custom decoder (which most libary users will need), see the example below\n\n### Types\n\n```golang\n// Payload is a type that represents your deserialized NATS message\ntype Payload interface {\n\tany\n}\n\n// DestKey is a type that represents how you want to group (i.e. split) your messages.\ntype DestKey interface {\n\tcomparable\n}\n```\n\n## Example\n\n```golang\n// ExamplePayload is our explicit struct for the NATS messages\ntype ExamplePayload struct {\n\tFirstName string `json:\"first_name\"`\n\tLastName  string `json:\"last_name\"`\n\tRegion    string `json:\"region\"`\n}\n\n// ExampleDestKey is just a simple string alias\ntype ExampleDestKey = string\n\n// JSONToLocalFsCSV is an example configuration that will decode JSON messages that sent over NATS, write them out into\n// CSV files, and group the output into `region` specific folder on the local file system.\n//\n// Use a pointer to an ExamplePayload as the Payload type parameter and ExampleDestKey as the DestKey type parameter\nvar JSONToLocalFsCSV = \u0026jetcapture.Options[*ExamplePayload, ExampleDestKey]{\n\tCompression: jetcapture.GZip, // use gzip compression\n\tSuffix:      \"csv\",           // suffix will end up being `.csv.gz`\n\tMaxAge:      time.Hour,       // messages will be written once an hour\n\n\t// configure the decoder\n\t// the incoming NATS messages contain a JSON string which we will decode\n\t// we also need to return a `DestKey` which we've defined to by a string\n\t// this key returned is the _region_ field of the decoded message\n\tMessageDecoder: func(msg *nats.Msg) (*ExamplePayload, ExampleDestKey, error) {\n\t\tvar p ExamplePayload\n\t\tif err := json.Unmarshal(msg.Data, \u0026p); err != nil {\n\t\t\treturn nil, \"\", err\n\t\t}\n\t\treturn \u0026p, p.Region, nil\n\t},\n\n\t// use the jetcapture.NewCSVWriter helper\n\t// we need to specify the headers, and a function that will \"flatten\" the payload\n\t// into one or more CSV rows\n\tWriterFactory: func() jetcapture.FormattedDataWriter[*ExamplePayload] {\n\t\treturn jetcapture.NewCSVWriter(\n\t\t\t[]string{\"first_name\", \"last_name\", \"region\"},\n\t\t\tfunc(p *ExamplePayload) ([][]string, error) {\n\t\t\t\treturn [][]string{{\n\t\t\t\t\tp.FirstName,\n\t\t\t\t\tp.LastName,\n\t\t\t\t\tp.Region,\n\t\t\t\t}}, nil\n\t\t\t},\n\t\t)\n\t},\n\n\t// use the jetcapture.LocalFSStore helper\n\t// we need to provide a `Resolver` that returns a filesystem path using the destination key\n\t// the path will use the `region` field to group output\n\tStore: \u0026jetcapture.LocalFSStore[ExampleDestKey]{\n\t\tResolver: func(dk ExampleDestKey) (string, error) {\n\t\t\treturn filepath.Join(\"backup\", dk), nil\n\t\t},\n\t},\n}\n```\n\n## TODO\n\n- [ ] Decide on explicit `nack` strategy where possible\n- [ ] Add S3 store example\n- [ ] Stats export\n- [ ] Add `DrainTimeout` for `Capture.sweepBlocks`. Right now a canceled context (e.g. CTRL-C) triggers a final sweep.\n      However, for calls that _take_ a context during a `BlockStore.Write` call (e.g. Azure blob store), the call will\n      often be short-circuited. A separate drain/sweep context should be created with a timeout.\n- [ ] Add better logging configuration/interface\n- [ ] Add support for checking outstanding acks and warning if near or at limit\n- [ ] Investigate a Go routine pool for `BlockStore.Write` (current code blocks during the write phase)\n- [ ] Output filenames need some more thought\n\n## Credits\n\n* Jonathan Camp @intelecy","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintelecy%2Fjet-capture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintelecy%2Fjet-capture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintelecy%2Fjet-capture/lists"}