{"id":21843946,"url":"https://github.com/fsaintjacques/recordlite","last_synced_at":"2025-04-14T12:10:50.022Z","repository":{"id":45173919,"uuid":"440223104","full_name":"fsaintjacques/recordlite","owner":"fsaintjacques","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-30T01:41:42.000Z","size":8,"stargazers_count":29,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-03-28T01:14:27.021Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fsaintjacques.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-20T15:45:13.000Z","updated_at":"2025-02-24T05:19:38.000Z","dependencies_parsed_at":"2022-08-27T10:22:38.579Z","dependency_job_id":null,"html_url":"https://github.com/fsaintjacques/recordlite","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsaintjacques%2Frecordlite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsaintjacques%2Frecordlite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsaintjacques%2Frecordlite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fsaintjacques%2Frecordlite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fsaintjacques","download_url":"https://codeload.github.com/fsaintjacques/recordlite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248877958,"owners_count":21176244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-27T22:17:45.702Z","updated_at":"2025-04-14T12:10:49.993Z","avatar_url":"https://github.com/fsaintjacques.png","language":"Go","readme":"# RecordLite\n\nRecordLite is a library (and executable) that declaratively maintains SQLite tables\nand views of semi-structured data (henceforth known as records). RecordLite is\nbased on a hidden gem in Backtrace's [sqlite_protobuf](https://github.com/backtrace-labs/sqlite_protobuf/blob/141486b492ccf342cbba6fa40e076a8941afc839/proto_table/proto_table.c)\nlibrary.\n\nTLDR: RecordLite stores semi-structured records into SQLite table where one column is\nthe raw payload (JSON or Protobuf) and define views with virtual columns from the\nraw column via extraction functions, e.g. [json_extract](https://www.sqlite.org/json1.html#jex) and\n[protobuf_extract](https://github.com/rgov/sqlite_protobuf#protobuf_extractprotobuf-type_name-path).\nTell RecordLite what you want the view and its indexes to look like, and RecordLite\nspits out idempotent DDL statements to make that happen, for any initial state and\nwith minimal churn.\n\n## Installation\n\nTo install the `recordlite` executable via `go get`.\n\n```\n# go install github.com/fsaintjacques/recordlite/cmd/recordlite@latest\n```\n\nTo install `recordlite` as a go module dependency\n\n```\n# go get github.com/fsaintjacques/recordlite/cmd/recordlite@latest\n```\n\n## About\n\n### Why\n\nSchema management is a hard practical problem. By using semi-structured data\nand storing it as a single BLOB column, one does not need to modify the \"true\"\ntable's schema, only the dynamic views.\n\n### How\n\nFor a given table of records, RecordLite generates a companion view that exposes\nvirtual columns of fields of interested defined by the user. The columns can be\noptionally indexed if they're often projected and/or used for filtering. The\ncolumn are defined with a function extracting the data from the raw column\n(either JSON or Protobuf), note that this would also work for any type of\nexpression.\n\nSince the view does not own the data, it is safe to delete/add/update columns\nwithout affecting the underlying table. In other words, it is relatively cheap\nto modify the view since it will not trigger a massive scan + write loop. OTOH,\nupdating any index will require recomputing the index.\n\n## Examples\n\n```\n$ # Define a schema\n$ cat schema.json\n{\n  \"name\": \"records\",\n  \"columns\": [\n    {\"name\": \"status\", \"expr\": \"json_extract(raw, '$.status')\", \"with_index\": true},\n    {\"name\": \"color\", \"expr\": \"json_extract(raw, '$.attrs.color')\", \"with_index\": true}\n  ]\n}\n\n$ # Create the table and views definitions\n$ recordlite schema.json | sqlite3 records.db\n\n$ # Simulate a process appending to the records\n$ cat \u003c\u003c EOF | awk '{print \"INSERT INTO records(raw) VALUES('\"'\"'\" $0 \"'\"'\"');\"}' | sqlite3 records.db\n\u003e {\"status\":\"ok\", \"attrs\": {\"color\": \"red\", \"size\": \"big\"}}\n\u003e {\"status\":\"failed\", \"attrs\": {\"color\":\"blue\", \"size\": \"small\"}}\n\u003e EOF\n\n$ sqlite3 --box records.db\nsqlite\u003e SELECT id, status, color from records;\n┌────┬────────┬───────┐\n│ id │ status │ color │\n├────┼────────┼───────┤\n│ 1  │ ok     │ red   │\n│ 2  │ failed │ blue  │\n└────┴────────┴───────┘\n\n$ # Let's modify the schema to index attrs.size\n$ cat schema.json\n{\n  \"name\": \"records\",\n  \"columns\": [\n    {\"name\": \"status\", \"expr\": \"json_extract(raw, '$.status')\", \"with_index\": true},\n    {\"name\": \"color\", \"expr\": \"json_extract(raw, '$.attrs.color')\", \"with_index\": true},\n    {\"name\": \"size\", \"expr\": \"json_extract(raw, '$.attrs.size')\", \"with_index\": true}\n  ]\n}\n\n$ # Update the views definition\n$ recordlite schema.json | sqlite3 records.db\n\n$ sqlite3 --box records.db\nsqlite\u003e SELECT id, status, color, size FROM records;\n┌────┬────────┬───────┬───────┐\n│ id │ status │ color │ size  │\n├────┼────────┼───────┼───────┤\n│ 1  │ ok     │ red   │ big   │\n│ 2  │ failed │ blue  │ small │\n└────┴────────┴───────┴───────┘\n\n```\n","funding_links":[],"categories":["Tools"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsaintjacques%2Frecordlite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffsaintjacques%2Frecordlite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffsaintjacques%2Frecordlite/lists"}