{"id":13689332,"url":"https://github.com/PowerDNS/lightningstream","last_synced_at":"2025-05-01T23:33:51.971Z","repository":{"id":146437575,"uuid":"610723628","full_name":"PowerDNS/lightningstream","owner":"PowerDNS","description":"Lightning Stream syncs LMDB databases through S3 buckets between multiple servers, including PowerDNS Authoritative server 4.8+ LMDBs","archived":false,"fork":false,"pushed_at":"2024-07-09T08:06:37.000Z","size":785,"stargazers_count":29,"open_issues_count":11,"forks_count":16,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-11-09T01:21:11.017Z","etag":null,"topics":["dns","golang","lmdb","powerdns","powerdns-authoritative-server"],"latest_commit_sha":null,"homepage":"https://doc.powerdns.com/lightningstream/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PowerDNS.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-07T10:58:26.000Z","updated_at":"2024-10-11T08:44:24.000Z","dependencies_parsed_at":"2024-02-07T06:34:37.096Z","dependency_job_id":"a695ac47-98bb-4a0a-9824-56800a054d76","html_url":"https://github.com/PowerDNS/lightningstream","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Flightningstream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Flightningstream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Flightningstream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PowerDNS%2Flightningstream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PowerDNS","download_url":"https://codeload.github.com/PowerDNS/lightningstream/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224282235,"owners_count":17285793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dns","golang","lmdb","powerdns","powerdns-authoritative-server"],"created_at":"2024-08-02T15:01:43.928Z","updated_at":"2024-11-12T13:31:28.453Z","avatar_url":"https://github.com/PowerDNS.png","language":"Go","readme":"# Lightning Stream\n\n[User documentation can be found here](https://doc.powerdns.com/lightningstream/)\n\n![Go build](https://github.com/PowerDNS/lightningstream/actions/workflows/go.yml/badge.svg)\n![Documentation build](https://github.com/PowerDNS/lightningstream/actions/workflows/documentation.yml/badge.svg)\n[![Go Reference](https://pkg.go.dev/badge/github.com/PowerDNS/lightningstream.svg)](https://pkg.go.dev/github.com/PowerDNS/lightningstream)\n\nLightning Stream is a tool to sync changes between a local LMDB (Lightning Memory-Mapped Database) and\nan S3 bucket in near real-time. If the application schema is compatible, this can be used in a multi-writer\nsetup where any instance can update any data, with a global eventually-consistent view of the data in seconds.\n\nOur main target application is the sync of LMDB databases in the\n[PowerDNS Authoritative Nameserver](https://doc.powerdns.com/authoritative/) (PDNS Auth). We are excited\nabout how Lightning Stream simplifies running multiple distributed PowerDNS Authoritative servers, with full support\nfor keeping DNSSEC keys in sync.\nCheck the [Getting Started](docs/getting-started.md) section to understand how you can use Lightning Stream together\nwith the PowerDNS Authoritative server.\n\nIts use is not limited to the PowerDNS Authoritative server, however. Lightning Stream does not make any assumptions\nabout the contents of the LMDB, and can be used to sync LMDBs for other applications, as long as the data is stored\nusing a [compatible schema](docs/schema.md).\n\n\n## Basic Operation\n\nLightning Stream is deployed next to an application that uses an LMDB for its data storage:\n\n![Overview](docs/images/lightningstream-overview.png)\n\nIts operation boils down to the following:\n\n- Whenever it detects that the LMDB has changed, it writes a snapshot of the data to an S3 bucket.\n- Whenever it sees a new snapshot written by a _different instance_ in the S3 bucket, it downloads the snapshot\n  and merges the data into the local LMDB.\n\nThe merge of a key is performed based on a per-record last-modified timestamp:\nthe most recent version of the entry wins. Deleted entries are cleared and marked as deleted, together with\ntheir deletion timestamp. This allows Lightning Stream to provide **Eventual Consistency** across all nodes.\n\nIf the application uses a [carefully designed data schema](docs/schema.md), this approach can be used to support\nmultiple simultaneously active writers. In other instances, it can often be used to sync data from one writer to\nmultiple read-only receivers. Or it can simply create a near real-time backup of a single instance.\n\n\n## Building\n\nAt the moment of writing, this project requires Go 1.19. Please check the `go.mod` file for the current\nversion.\n\nTo install the binary in a given location, simply run:\n\n    GOBIN=$HOME/bin go install ./cmd/lightningstream/\n\nOr run `./build.sh` to install it in a `bin/` subdirectory of this repo.\n\nEasy cross compiling is not supported, because the LMDB bindings require CGo.\n\n\n## Example in Docker Compose\n\nThis repo includes an example of syncing the PowerDNS Authoritative Nameserver LMDB. It runs two DNS\nservers, each with their own syncer, syncing to a bucket in a MinIO server.\n\nThe Lightning Stream config used can be found in `docker/pdns/lightningstream.yaml`. Note that the\nconfig file contents can reference environment variables.\n\nTo get it up and running:\n\n    docker-compose up -d\n\nYou may need to rerun this command once, because of a race condition creating the LMDBs.\n\nTo see the services:\n\n    docker-compose ps\n\nThis should show output like:\n\n```\n         Name                        Command               State                                    Ports\n-------------------------------------------------------------------------------------------------------------------------------------------\nlightningstream_auth1_1   /run.sh                          Up      127.0.0.1:4751-\u003e53/tcp, 127.0.0.1:4751-\u003e53/udp, 127.0.0.1:4781-\u003e8081/tcp\nlightningstream_auth2_1   /run.sh                          Up      127.0.0.1:4752-\u003e53/tcp, 127.0.0.1:4752-\u003e53/udp, 127.0.0.1:4782-\u003e8081/tcp\nlightningstream_minio_1   /usr/bin/docker-entrypoint ...   Up      127.0.0.1:4730-\u003e9000/tcp, 127.0.0.1:4731-\u003e9001/tcp\nlightningstream_sync1_1   /usr/local/bin/lightningst ...   Up      127.0.0.1:4791-\u003e8500/tcp\nlightningstream_sync2_1   /usr/local/bin/lightningst ...   Up      127.0.0.1:4792-\u003e8500/tcp\n```\n\nOpen one terminal with all the logs:\n\n    docker-compose logs\n\nThen in another terminal call these convenience scripts, with a delay between them to allow for syncing:\n\n    docker/pdns/pdnsutil -i 1 create-zone example.org\n    docker/pdns/pdnsutil -i 1 secure-zone example.org\n    docker/pdns/pdnsutil -i 1 set-meta example.org foo bar\n    docker/pdns/pdnsutil -i 2 generate-tsig-key example123 hmac-sha512\n\n    sleep 2\n\n    docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org\n    docker/pdns/curl-api -i 2 /api/v1/servers/localhost/zones/example.org/metadata\n    docker/pdns/curl-api -i 1 /api/v1/servers/localhost/tsigkeys\n\nTo view a dump of the LMDB contents:\n\n    docker/pdns/dump-lmdb -i 1\n    docker/pdns/dump-lmdb -i 2\n\nYou can browse the snapshots in MinIO at \u003chttp://localhost:4731/buckets/lightningstream/browse\u003e\n(login with minioadmin / minioadmin).\n\n\n\n## Open Source\n\nThis is the documentation for the Open Source edition of Lightning Stream.\nFor more information on how we provide support for Open Source products, please read\n[our blog post on this topic](https://blog.powerdns.com/2016/01/18/open-source-support-out-in-the-open/).\n\nPowerDNS also offers an Enterprise edition of Lightning Stream that includes professional support, advanced features, deployment\ntooling for large deployments, Kubernetes integration, and more.\n\n\n\n","funding_links":[],"categories":["golang"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPowerDNS%2Flightningstream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPowerDNS%2Flightningstream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPowerDNS%2Flightningstream/lists"}