{"id":50663621,"url":"https://github.com/sqliteai/sqlite-columnar","last_synced_at":"2026-06-08T04:04:45.867Z","repository":{"id":357089933,"uuid":"1234388745","full_name":"sqliteai/sqlite-columnar","owner":"sqliteai","description":"Column-oriented analytics for SQLite, designed for fast local analytical queries with benchmarked speedups of up to 130,000x over standard SQLite scans on selected workloads.","archived":false,"fork":false,"pushed_at":"2026-05-11T08:25:03.000Z","size":2925,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-11T10:35:02.893Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://www.sqlite.ai","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sqliteai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-10T05:46:31.000Z","updated_at":"2026-05-11T08:25:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sqliteai/sqlite-columnar","commit_stats":null,"previous_names":["sqliteai/sqlite-columnar"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sqliteai/sqlite-columnar","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqliteai%2Fsqlite-columnar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqliteai%2Fsqlite-columnar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqliteai%2Fsqlite-columnar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqliteai%2Fsqlite-columnar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sqliteai","download_url":"https://codeload.github.com/sqliteai/sqlite-columnar/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sqliteai%2Fsqlite-columnar/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34047294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-08T02:00:07.615Z","response_time":111,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-08T04:04:45.126Z","updated_at":"2026-06-08T04:04:45.862Z","avatar_url":"https://github.com/sqliteai.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://sqlite.ai\"\u003e\n    \u003cimg src=\"https://www.sqlite.ai/social/logo-ai.png\" alt=\"SQLite AI\" height=\"56\"\u003e\n  \u003c/a\u003e\n\n  \u003ch1\u003eSQLite-Columnar\u003c/h1\u003e\n  \u003cp\u003e\u003cstrong\u003eColumn-oriented analytics inside SQLite.\u003c/strong\u003e\u003cbr\u003e\n  Sub-second OLAP queries on local files — aggregations, scans, and time-series workloads at warehouse speed, without a warehouse.\u003c/p\u003e\n\n  \u003cp\u003e\n    \u003ca href=\"https://dashboard.sqlitecloud.io/auth/sign-in\"\u003e\u003cstrong\u003eFree managed instance →\u003c/strong\u003e\u003c/a\u003e ·\n    \u003ca href=\"https://docs.sqlitecloud.io/docs/ai-overview\"\u003eDocs\u003c/a\u003e ·\n    \u003ca href=\"https://sqlite.ai\"\u003eWebsite\u003c/a\u003e ·\n    \u003ca href=\"https://blog.sqlite.ai\"\u003eBlog\u003c/a\u003e\n  \u003c/p\u003e\n\n  \u003cp\u003e\n    \u003csub\u003e\u003cstrong\u003eData:\u003c/strong\u003e\n    \u003ca href=\"https://github.com/sqliteai/sqlite-vector\"\u003eVector\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-sync\"\u003eSync\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-columnar\"\u003eColumnar\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-js\"\u003eJS\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cstrong\u003eAI:\u003c/strong\u003e\n    \u003ca href=\"https://github.com/sqliteai/sqlite-ai\"\u003eAI\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-agent\"\u003eAgent\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-memory\"\u003eMemory\u003c/a\u003e ·\n    \u003ca href=\"https://github.com/sqliteai/sqlite-mcp\"\u003eMCP\u003c/a\u003e\n    \u003c/sub\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n\u003cbr\u003e\n\n\u003e **Need analytics at cloud scale?** SQLite-Columnar runs OLAP locally; pair it with **[SQLite Cloud](https://dashboard.sqlitecloud.io/auth/sign-in)** to persist large datasets, run queries across regions, and share dashboards. Free tier available.\n\n---\n\n# sqlite-columnar\n\n`sqlite-columnar` brings column-oriented analytics to SQLite as a self-contained\nloadable extension. It lets applications keep the operational simplicity of\nSQLite while adding a storage and execution path built for analytical scans,\naggregations, and grouped summaries over wide datasets.\n\nIt does not patch SQLite's pager, btree, parser, VDBE, or shell. Build it as a\nnormal extension, load it into SQLite, and create columnar virtual tables for\nthe parts of your application that behave more like analytics than OLTP.\n\n## Why Columnar SQLite?\n\nTraditional SQLite tables are row-oriented, which is excellent for point\nlookups, small updates, and transactional application state. Analytical\nworkloads are different: they often read a few columns across many rows, compute\naggregates, group by dimensions, and filter by ranges. In those cases, reading\nentire rows means paying I/O and CPU cost for data the query never uses.\n\n`sqlite-columnar` stores each column separately, tracks chunk-level metadata,\nand provides specialized aggregate helpers that avoid generic row\nmaterialization for common analytical queries.\n\n## Performance Highlights\n\nOn the included 10 million row variance benchmark, `sqlite-columnar` shows\nlarge median speedups over standard row-oriented SQLite for operations that\nbenefit from columnar layout and precomputed metadata:\n\n- `sum(v1)` with `columnar_sum`: **130,583x faster**\n- `avg(v3)` with `columnar_avg`: **129,317x faster**\n- `count(v1)` with `columnar_count`: **125,194x faster**\n- grouped `sum` by dimension: **6.04x faster**\n- grouped `count` by dimension: **14.13x faster**\n- grouped `sum/avg/count` by dimension: **6.42x faster**\n- clustered range filter on `ts`: **248.89x faster**\n- clustered range filter plus grouped `sum/avg/count`: **273.20x faster**\n\nThese numbers are workload-specific. They are strongest when queries scan a\nsmall subset of columns, use aggregate metadata, group over low-cardinality\ndimensions, or filter on clustered/range-friendly columns. See\n[BENCHMARK.md](BENCHMARK.md) for the full dataset, commands, timings, and\ninterpretation.\n\n## Common Use Cases\n\n`sqlite-columnar` is useful when an embedded application needs analytical\nqueries without moving data into a separate database server.\n\nGood fits include:\n\n- embedded dashboards over local event, telemetry, or product analytics data\n- time-series rollups where queries filter by timestamp ranges\n- IoT and edge analytics over wide sensor records\n- desktop or mobile apps with local reporting and summary views\n- feature stores or ML preprocessing jobs that scan a few feature columns at a\n  time\n- audit logs and observability data where users aggregate by service, region,\n  status, or time bucket\n- SaaS tenant-local analytics where a single-file SQLite database is still the\n  preferred deployment model\n- ETL validation workloads that repeatedly compute counts, sums, min/max, and\n  grouped quality checks\n\nRow-oriented SQLite remains the better default for highly transactional\nworkloads, point lookups, and frequent single-row updates. `sqlite-columnar` is\nintended for the analytical tables in the same application.\n\n## How It Works\n\nEach columnar virtual table owns shadow tables for rowids, column values,\nglobal stats, chunk zone maps, dirty chunks, and table-level metadata.\n\n`columnar_analyze()` builds the metadata used by specialized analytical\nfunctions. After the initial bootstrap, analyze is incremental: inserts,\nupdates, and deletes mark touched chunks dirty, and later analyze calls rebuild\nonly those chunks. If metadata says stats are valid and there are no dirty\nchunks, analyze returns immediately.\n\nRange-filtered helpers use chunk min/max summaries to skip rowid ranges that\ncannot match a filter. Grouped helpers perform hash aggregation in C over only\nthe required column shadow tables.\n\n## Build\n\nFrom this directory:\n\n```sh\nmake\n```\n\nBy default the build uses the bundled `sqlite/` directory for `sqlite3ext.h`.\nTo build against a different SQLite checkout or amalgamation directory, pass\n`SQLITE_SRC`:\n\n```sh\nmake SQLITE_SRC=/path/to/sqlite\n```\n\nThis produces `columnar.dylib` on macOS or `columnar.so` on Linux.\n\n## Prebuilt Binaries\n\nTagged releases build loadable extension binaries for Linux, Linux musl,\nmacOS, Windows, Android, iOS, and iOS Simulator. macOS release assets are\nDeveloper ID signed and notarized ZIP archives. Other platforms are published\nas release archives. Each asset contains the platform binary plus `README.md`,\n`API.md`, `BENCHMARK.md`, and `GIT_COMMIT`. Release assets also include\n`SHA256SUMS` for archive verification.\n\n## Quick Start\n\n```sql\n.load ./columnar\n\nCREATE VIRTUAL TABLE sales USING columnar(\n  id INTEGER,\n  region TEXT,\n  amount REAL\n);\n\nINSERT INTO sales VALUES\n  (1, 'north', 10.0),\n  (2, 'north', 20.0),\n  (3, 'south', 5.0);\n\nSELECT columnar_analyze('sales');\nSELECT columnar_sum('sales', 'amount');\n\nSELECT k, \"sum\", \"avg\", \"count\"\n  FROM columnar_group_sum_avg_count('sales', 'region', 'amount')\n ORDER BY k;\n```\n\nSee [API.md](API.md) for the complete SQL API reference with examples for every\nfunction and table-valued helper.\n\n## Benchmarks\n\nThe benchmark suite is built against the bundled `sqlite/sqlite3.c` by default:\n\n```sh\nmake benchmarks\nbuild/columnar-analytics-bench ./columnar 10000000 256\n```\n\n[BENCHMARK.md](BENCHMARK.md) documents the main analytical benchmark: schema,\ndataset shape, load/analyze costs, storage size, query timings, speedups, and\nthe interpretation of where columnar storage helps most. Use it as the baseline\nfor evaluating changes and for understanding which query patterns benefit from\nthis extension.\n\nUse `make smoke-bench` for small correctness-oriented benchmark runs.\n\nUse `make variance-bench` to run the repeatable performance-variance suite. It\nbuilds multiple deterministic datasets, warms each query once, repeats each\nmeasurement, verifies row-store/columnar result hashes, and reports median and\np95 timings:\n\n```sh\nmake variance-bench VARIANCE_REPEATS=9 \\\n  VARIANCE_DATASETS=\"small:10000:64 medium:50000:128 wide:50000:512\"\n```\n\n## Test\n\n```sh\nmake test SQLITE3=/path/to/sqlite3\n```\n\nThe test target runs the SQL smoke suite plus a native robustness suite. The\nrobustness suite checks rollback and savepoint behavior, simulated process\ndeath during an uncommitted transaction, unusual table/column names and mixed\nSQLite storage classes, and automatic result equivalence between each\nspecialized columnar query helper and the matching ordinary SQLite query.\n\n## License\n\nThis project is licensed under the [Elastic License 2.0](./LICENSE.md). For production or managed service use, [contact SQLite Cloud, Inc](mailto:info@sqlitecloud.io) for a commercial license.\n\n---\n\n## ☁️ Hosted version\n\nDon't want to run analytics infrastructure yourself? **[SQLite Cloud](https://sqlite.ai)** is the managed companion to SQLite-Columnar — local-first analytics with cloud persistence, multi-region replication, and shared dashboards.\n\n[**Start free →**](https://dashboard.sqlitecloud.io/auth/sign-in)\n\n---\n\n## Part of the SQLite AI stack\n\nSQLite-Columnar is one piece of a larger ecosystem that turns SQLite into a runtime for intelligent, distributed data:\n\n**Data layer**\n- [sqlite-vector](https://github.com/sqliteai/sqlite-vector) — ANN vector search inside SQLite\n- [sqlite-sync](https://github.com/sqliteai/sqlite-sync) — Offline-first CRDT sync across devices\n- [**sqlite-columnar**](https://github.com/sqliteai/sqlite-columnar) — Column-oriented analytics for OLAP queries *(you are here)*\n- [sqlite-js](https://github.com/sqliteai/sqlite-js) — Custom SQLite functions written in JavaScript\n\n**AI layer**\n- [sqlite-ai](https://github.com/sqliteai/sqlite-ai) — On-device LLM inference and embeddings\n- [sqlite-agent](https://github.com/sqliteai/sqlite-agent) — Autonomous AI agents running inside SQLite\n- [sqlite-memory](https://github.com/sqliteai/sqlite-memory) — Persistent, searchable memory for agents\n- [sqlite-mcp](https://github.com/sqliteai/sqlite-mcp) — Call MCP tools directly from SQL queries\n\n**Managed platform**\n- [SQLite Cloud](https://sqlite.ai) — Hosted SQLite with sync, auth, edge functions, and analytics. [Free tier →](https://dashboard.sqlitecloud.io/auth/sign-in)\n\nBuilt by [SQLite AI](https://sqlite.ai). Questions? [Contact us](https://sqlite.ai/support).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqliteai%2Fsqlite-columnar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsqliteai%2Fsqlite-columnar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsqliteai%2Fsqlite-columnar/lists"}