{"id":13616206,"url":"https://github.com/zknill/sqledge","last_synced_at":"2025-04-12T14:57:03.762Z","repository":{"id":183693306,"uuid":"669193279","full_name":"zknill/sqledge","owner":"zknill","description":"Replicate postgres to SQLite on the edge","archived":false,"fork":false,"pushed_at":"2024-06-17T10:09:56.000Z","size":295,"stargazers_count":1016,"open_issues_count":11,"forks_count":20,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T13:18:59.381Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zknill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-21T15:13:02.000Z","updated_at":"2025-03-20T09:03:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"da2e7477-3cc7-437b-abfe-88910385e3c6","html_url":"https://github.com/zknill/sqledge","commit_stats":null,"previous_names":["zknill/sqledge"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zknill%2Fsqledge","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zknill%2Fsqledge/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zknill%2Fsqledge/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zknill%2Fsqledge/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zknill","download_url":"https://codeload.github.com/zknill/sqledge/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248586248,"owners_count":21128997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T20:01:25.084Z","updated_at":"2025-04-12T14:57:03.735Z","avatar_url":"https://github.com/zknill.png","language":"Go","readme":"# SQLedge\n\n[State: alpha]\n\nSQLedge uses Postgres logical replication to stream the changes in a source Postgres database to a SQLite database that can run on the edge.\nSQLedge serves reads from its local SQLite database, and forwards writes to the upstream Postgres server that it's replicating from.\n\nThis lets you run your apps on the edge, and have local, fast, and eventually consistent access to your data.\n\n![SQLedge](https://github.com/zknill/sqledge/blob/main/etc/sqledge.png?raw=true)\n\n## SQL generation\n\nThe `pkg/sqlgen` package has an SQL generator in it, which will generate the SQLite insert, update, delete statements based on the logical replication messages received.\n\n## SQL parsing\n\nWhen the database is started, we look at which tables already exist in the sqlite copy, and make sure new tables are created automatically on the fly.\n\n## Postgres wire proxy\n\nSQLedge contains a Postgres wire proxy, default on `localhost:5433`. This proxy uses the local SQlite database for reads, and forwards writes to the upstream Postgres server.\n\n### Compatibility \n\nWhen running, the SQL statements interact with two databases; Postgres (for writes) and SQLite (for reads). \n\nThe Postgres wire proxy (which forwards reads to SQLite) doesn't currently translate any of the SQL statements from the Postgres query format/functions to the SQLite format/functions. \nRead queries issued against the Postgres wire proxy need to be compatible with SQLite directly. \nThis is fine for simple `SELECT` queries, but you will have trouble with Postgres-specific query functions or syntax.\n\n## Copy on startup\n\nSQLEdge maintains a table called `postgres_pos`, this tracks the LSN (log sequence number) of the received logical replication messages so it can pick up processing where it left\noff.\n\nIf no LSN is found, SQLedge will start a postgres `COPY` of all tables in the `public` schema. Creating the appropriate SQLite tables, and inserting data.\n\nWhen the replication slot is first created, it exports a transaction snapshot. This snapshot is used for the initial copy. This means that the `COPY` command will read the data from\nthe transaction at the moment the replication slot was created. \n\n## Trying it out\n\n1. Create a database\n\n   ```\n   create database myappdatabase;\n   ```\n\n2. Create a user -- must be a super user because we create a publication on all tables\n\n   ```\n   create user sqledger with login superuser password 'secret';\n   ```\n\n\n3. Run the example\n\n   ```\n   SQLEDGE_UPSTREAM_USER=sqledger SQLEDGE_UPSTREAM_PASSWORD=secret SQLEDGE_UPSTREAM_NAME=myappdatabase go run ./cmd/sqledge/main.go\n   ```\n\n4. Connect to the postgres wire proxy\n\n   ```\n   psql -h localhost -p 5433\n \n   $ CREATE TABLE my_table (id serial not null primary key, names text);\n   $ INSERT INTO my_table (names) VALUES ('Jane'), ('John');\n\n   $ SELECT * FROM my_table;\n   ```\n   The read will be served from the local database\n\n5. Connect to the local sqlite db\n\n   ```\n   sqlite3 ./sqledge.db\n\n   .schema\n   ```\n\n## Config\n\nAll config is read from environment variables. The full list is available in the struct tags on the fields in `pkg/config/config.go`\n","funding_links":[],"categories":["Go","backup and replicate"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzknill%2Fsqledge","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzknill%2Fsqledge","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzknill%2Fsqledge/lists"}