https://github.com/astromechza/automerge-ndjson-sync
A library for serving automerge changes over HTTP1.1 and HTTP2 newline-delimited json
https://github.com/astromechza/automerge-ndjson-sync
automerge crdt go golang http
Last synced: 10 months ago
JSON representation
A library for serving automerge changes over HTTP1.1 and HTTP2 newline-delimited json
- Host: GitHub
- URL: https://github.com/astromechza/automerge-ndjson-sync
- Owner: astromechza
- License: mit
- Created: 2024-10-24T23:06:38.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-05T21:25:25.000Z (over 1 year ago)
- Last Synced: 2024-12-09T15:55:06.837Z (over 1 year ago)
- Topics: automerge, crdt, go, golang, http
- Language: Go
- Homepage:
- Size: 69.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# automerge-ndjson-sync (an `automerge-go` sync library)
This library is a utility library for synchronising [automerge](https://automerge.org/) documents over HTTP using a `application/x-ndjson` protocol:
1. Both the request and response bodies contain newline-delimited json lines. The content-type is `application/x-ndjson; charset=utf-8`.
2. Each line looks like `{"event":"sync", "data":""}\n`
3. The server stays connected, continuously receiving messages and sending messages as they are ready on the document via either HTTP2 or well-behaved HTTP1.1 clients.
4. The client decides when to terminate the connection by observing the messages it receives, either:
1. The response body is closed after the server detects that the request body is complete and no more messages are available.
2. The client sees a sync message that meets its "termination check", which may indicate that the server matches the local state or that the local state contains all the remote head nodes. This can be used for local tools that need to perform a "one-shot" synchronisation on startup.
5. There's a broadcast capability that allows a server to serve changes from multiple clients on the same doc simultaneously or for a client to synchronise with multiple servers.
6. The client supports HTTP redirect behavior so that servers can implement rudimentary partitioning and balancing of requests.
This library will be used to build a series of small peer-to-peer and distributed state utilities built on Automerge. The protocol above is easy to replicate in most languages, most importantly Go (in this repo) and Javascript.
## FAQ: Why not use the Automerge sync-server Websocket protocols?
The Automerge sync server and related libraries are written in and generally use websockets to communicate. I wanted to try and utilise HTTP2 and concurrent HTTP1.1 to achieve a similar thing with Go.
This gives me a pure-go option with very few dependencies that I can trust to be stable and maintainable for a long time.
## FAQ: Can you give me an example over the wire?
By executing `go run ./examples/server/` in one terminal, and `DOC_ID=example go run ./examples/http2writer` in another terminal, I can then execute a raw `curl` request in a 3rd terminal to follow the stream over HTTPS. I send an empty sync message to start and observe the following:
```
$ curl -k -v -X PUT https://localhost:8080/example -d '{"event":"sync","data":"QgAAAQAAAA=="}' -H 'Content-Type: application/x-ndjson'
...
> PUT /example HTTP/2
> Host: localhost:8080
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/x-ndjson
> Content-Length: 38
< HTTP/2 200
< content-type: application/x-ndjson; charset=utf-8
< date: Sat, 26 Oct 2024 09:56:23 GMT
<
{"event":"sync","data":"QgFbkqa2LT9CqZQD6wA="}
{"event":"sync","data":"QgEVJTDZHRF/AH+BAQ=="}
{"event":"sync","data":"QgG3qWH1RH8BfwB/ggE="}
...
```
Until I decide to hang up the connection with Ctrl-C. And this works perfectly fine with HTTP1.1 too:
```
$ curl -k -i -X PUT https://localhost:8080/example -d '{"event":"sync","data":"QgAAAQAAAA=="}' -H 'Content-Type: application/x-ndjson' --http1.1
< HTTP/1.1 200 OK
< Content-Type: application/x-ndjson; charset=utf-8
< Date: Sat, 26 Oct 2024 11:32:52 GMT
< Transfer-Encoding: chunked
{"event":"sync","data":"QgFbkqa2LT9CqZQD6wA="}
{"event":"sync","data":"QgEVJTDZHRF/AH+BAQ=="}
{"event":"sync","data":"QgG3qWH1RH8BfwB/ggE="}
...
```
## FAQ: Why do the examples use HTTPS, do I need to use HTTPS?
The examples include an HTTP2 client, so the server has a self-signed certificate so that it can present HTTP2 over HTTPS.
The client can still use HTTP1.1 as seen in the curl example above or the http1follower example.
## Dependencies
This is purposefully built with only the Go standard library + `github.com/automerge/automerge-go`. This is to reduce maintenance burden for me.
## Testing
Unit tests, including Server and Client syncing, are executed through either `make test` or the Github Actions CI.