{"id":17967213,"url":"https://github.com/astromechza/automerge-ndjson-sync","last_synced_at":"2025-06-12T01:35:07.315Z","repository":{"id":259546555,"uuid":"878182626","full_name":"astromechza/automerge-ndjson-sync","owner":"astromechza","description":"A library for serving automerge changes over HTTP1.1 and HTTP2 newline-delimited json","archived":false,"fork":false,"pushed_at":"2024-11-05T21:25:25.000Z","size":71,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-09T15:55:06.837Z","etag":null,"topics":["automerge","crdt","go","golang","http"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/astromechza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-24T23:06:38.000Z","updated_at":"2024-11-05T21:23:07.000Z","dependencies_parsed_at":"2024-10-26T12:06:21.028Z","dependency_job_id":"a37fedf5-66ed-4061-a775-0a54180deae7","html_url":"https://github.com/astromechza/automerge-ndjson-sync","commit_stats":null,"previous_names":["astromechza/automerge-ndjson-sync"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astromechza%2Fautomerge-ndjson-sync","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astromechza%2Fautomerge-ndjson-sync/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astromechza%2Fautomerge-ndjson-sync/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/astromechza%2Fautomerge-ndjson-sync/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/astromechza","download_url":"https://codeload.github.com/astromechza/automerge-ndjson-sync/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230043145,"owners_count":18163966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automerge","crdt","go","golang","http"],"created_at":"2024-10-29T14:04:40.441Z","updated_at":"2024-12-17T00:13:52.552Z","avatar_url":"https://github.com/astromechza.png","language":"Go","readme":"# automerge-ndjson-sync (an `automerge-go` sync library)\n\n![GitHub License](https://img.shields.io/github/license/astromechza/automerge-ndjson-sync)\n![GitHub go.mod Go version](https://img.shields.io/github/go-mod/go-version/astromechza/automerge-ndjson-sync)\n![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/astromechza/automerge-ndjson-sync/ci.yaml)\n\nThis library is a utility library for synchronising [automerge](https://automerge.org/) documents over HTTP using a `application/x-ndjson` protocol:\n\n1. Both the request and response bodies contain newline-delimited json lines. The content-type is `application/x-ndjson; charset=utf-8`.\n2. Each line looks like `{\"event\":\"sync\", \"data\":\"\u003cbase64-encoded sync message\u003e\"}\\n`\n3. The server stays connected, continuously receiving messages and sending messages as they are ready on the document via either HTTP2 or well-behaved HTTP1.1 clients.\n4. The client decides when to terminate the connection by observing the messages it receives, either:\n    1. The response body is closed after the server detects that the request body is complete and no more messages are available.\n    2. The client sees a sync message that meets its \"termination check\", which may indicate that the server matches the local state or that the local state contains all the remote head nodes. This can be used for local tools that need to perform a \"one-shot\" synchronisation on startup.\n5. There's a broadcast capability that allows a server to serve changes from multiple clients on the same doc simultaneously or for a client to synchronise with multiple servers.\n6. The client supports HTTP redirect behavior so that servers can implement rudimentary partitioning and balancing of requests.\n\nThis library will be used to build a series of small peer-to-peer and distributed state utilities built on Automerge. The protocol above is easy to replicate in most languages, most importantly Go (in this repo) and Javascript.\n\n## FAQ: Why not use the Automerge sync-server Websocket protocols?\n\nThe Automerge sync server \u003chttps://github.com/automerge/automerge-repo-sync-server\u003e and related \u003chttps://github.com/automerge/automerge-connection\u003e libraries are written in and generally use websockets to communicate. I wanted to try and utilise HTTP2 and concurrent HTTP1.1 to achieve a similar thing with Go.\n\nThis gives me a pure-go option with very few dependencies that I can trust to be stable and maintainable for a long time.\n\n## FAQ: Can you give me an example over the wire?\n\nBy executing `go run ./examples/server/` in one terminal, and `DOC_ID=example go run ./examples/http2writer` in another terminal, I can then execute a raw `curl` request in a 3rd terminal to follow the stream over HTTPS. I send an empty sync message to start and observe the following:\n\n```\n$ curl -k -v -X PUT https://localhost:8080/example -d '{\"event\":\"sync\",\"data\":\"QgAAAQAAAA==\"}' -H 'Content-Type: application/x-ndjson'\n...\n\u003e PUT /example HTTP/2\n\u003e Host: localhost:8080\n\u003e User-Agent: curl/8.5.0\n\u003e Accept: */*\n\u003e Content-Type: application/x-ndjson\n\u003e Content-Length: 38\n\n\u003c HTTP/2 200 \n\u003c content-type: application/x-ndjson; charset=utf-8\n\u003c date: Sat, 26 Oct 2024 09:56:23 GMT\n\u003c \n{\"event\":\"sync\",\"data\":\"QgFbkqa2LT\u003csnip\u003e9CqZQD6wA=\"}\n{\"event\":\"sync\",\"data\":\"QgEVJTDZHR\u003csnip\u003eF/AH+BAQ==\"}\n{\"event\":\"sync\",\"data\":\"QgG3qWH1RH\u003csnip\u003e8BfwB/ggE=\"}\n...\n```\n\nUntil I decide to hang up the connection with Ctrl-C. And this works perfectly fine with HTTP1.1 too:\n\n```\n$ curl -k -i -X PUT https://localhost:8080/example -d '{\"event\":\"sync\",\"data\":\"QgAAAQAAAA==\"}' -H 'Content-Type: application/x-ndjson' --http1.1\n\u003c HTTP/1.1 200 OK\n\u003c Content-Type: application/x-ndjson; charset=utf-8\n\u003c Date: Sat, 26 Oct 2024 11:32:52 GMT\n\u003c Transfer-Encoding: chunked\n\n{\"event\":\"sync\",\"data\":\"QgFbkqa2LT\u003csnip\u003e9CqZQD6wA=\"}\n{\"event\":\"sync\",\"data\":\"QgEVJTDZHR\u003csnip\u003eF/AH+BAQ==\"}\n{\"event\":\"sync\",\"data\":\"QgG3qWH1RH\u003csnip\u003e8BfwB/ggE=\"}\n...\n```\n\n## FAQ: Why do the examples use HTTPS, do I need to use HTTPS?\n\nThe examples include an HTTP2 client, so the server has a self-signed certificate so that it can present HTTP2 over HTTPS.\nThe client can still use HTTP1.1 as seen in the curl example above or the http1follower example.\n\n## Dependencies\n\nThis is purposefully built with only the Go standard library + `github.com/automerge/automerge-go`. This is to reduce maintenance burden for me.\n\n## Testing\n\nUnit tests, including Server and Client syncing, are executed through either `make test` or the Github Actions CI.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastromechza%2Fautomerge-ndjson-sync","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fastromechza%2Fautomerge-ndjson-sync","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fastromechza%2Fautomerge-ndjson-sync/lists"}