{"id":37210967,"url":"https://github.com/everlag/naive-columstore","last_synced_at":"2026-01-15T00:01:40.192Z","repository":{"id":57658330,"uuid":"59086236","full_name":"Everlag/naive-columstore","owner":"Everlag","description":"A simple column store in golang","archived":true,"fork":false,"pushed_at":"2016-06-13T21:22:16.000Z","size":12514,"stargazers_count":37,"open_issues_count":1,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-06-21T04:29:01.380Z","etag":null,"topics":["columnar-storage","database","golang","performance-testing"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Everlag.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-18T05:56:57.000Z","updated_at":"2023-01-28T15:41:44.000Z","dependencies_parsed_at":"2022-09-12T15:13:24.093Z","dependency_job_id":null,"html_url":"https://github.com/Everlag/naive-columstore","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/Everlag/naive-columstore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Everlag%2Fnaive-columstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Everlag%2Fnaive-columstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Everlag%2Fnaive-columstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Everlag%2Fnaive-columstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Everlag","download_url":"https://codeload.github.com/Everlag/naive-columstore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Everlag%2Fnaive-columstore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28439606,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T22:37:52.437Z","status":"ssl_error","status_checked_at":"2026-01-14T22:37:31.496Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["columnar-storage","database","golang","performance-testing"],"created_at":"2026-01-15T00:01:40.126Z","updated_at":"2026-01-15T00:01:40.182Z","avatar_url":"https://github.com/Everlag.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# naive-columstore\n\nThis is a simple, toy column store implemented as a result of reading 'The Design and Implementation of Modern Column-Oriented Database Systems'[1]. To keep everything sane and typesafe, this is a bespoke implementation for a dataset collected from mtgprice.\n\nI'm starting to benchmark early and releases will be tagged for when benchmarks can be stably compared.\n\n\n## Testing Bootstrap\n\nAs all queries written here have direct equivalents to SQL, it makes sense to bootstrap all test results against a known-good implementation.\n\nAs a result, queries implemented here are tested against the same query run in postgres.\n\nThe following are the commands to run in psql as the `postgres` user to import the data file. Open psql from the repo root, `\\copy` is used relatively to import the testing data.\n\n\t/*Create a database to test under*/\n\tCREATE DATABASE pricedata_tests WITH\n\t\tOWNER     postgres\n\t\tENCODING 'UTF8';\n\n\t\\connect pricedata_tests\n\n\t/*Add a schema to work under*/\n\tCREATE SCHEMA prices;\n\n\t/*Add a table to work under*/\n\tCREATE TABLE prices.mtgprice (\n\n\t\tname TEXT NOT NULL,\n\t\tset TEXT NOT NULL,\n\t\ttime timestamp NOT NULL,\n\n\t\tprice int NOT NULL,\n\t\tCONSTRAINT uniqueMTGpriceEntryKey UNIQUE (name, set, time)\n\n\t);\n\n\t/*Import data, will take a minute or two*/\n\t\\copy prices.mtgprice from 'prices.csv' DELIMITER ',' CSV HEADER;\n\n\t/*Ensure data is present, should return true*/\n\tselect (count(*) = 1000000) as valid from prices.mtgprice;\n\n## Historical Benchmarks\n\nWhen settling on a bitmap to use, I performed some benchmarks. Here they are formatted with github.com/cespare/prettybench\n\n\twillf/bitset ~240MB\n\t[]bool ~270MB\n\tRoaringBitmap/roaring ~ 200MB\n\n\twillf/bitset\n\tBenchmarkUint32More-4                                100          16320800 ns/op         262208 B/op          4 allocs/op\n\tBenchmarkUint32Less-4                                100          15615659 ns/op         131104 B/op          2 allocs/op\n\tBenchmarkUint32Delta-4                               100          15973511 ns/op       22995204 B/op         39 allocs/op\n\tBenchmarkUint32Sum-4                                2000            743859 ns/op              0 B/op          0 allocs/op\n\tBenchmarkBoolAND-4                                 50000             31699 ns/op              0 B/op          0 allocs/op\n\tBenchmarkSelectAllMoreThanDollar-4                   100          16020921 ns/op         262208 B/op          4 allocs/op\n\tBenchmarkFiniteString32Within-4                       50          25190530 ns/op         262241 B/op          6 allocs/op\n\tBenchmarkSelectAllMoreThanDollarMaterial-4            20          74621560 ns/op       47334222 B/op         45 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTen-4                 50          30496544 ns/op         393313 B/op          6 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTenMaterial-4         20          76079955 ns/op       37299048 B/op         45 allocs/op\n\tBenchmarkSelectAfterTimeWiseMidPoint-4               100          17992860 ns/op         131104 B/op          2 allocs/op\n\tBenchmarkLatestPriceMaterial-4                        50          25104048 ns/op         266057 B/op         17 allocs/op\n\tok      github.com/Everlag/test 100.352s\n\n\t[]bool\n\tPASS\n\tBenchmarkUint32More-4                                100          12300487 ns/op       5863804 B/op          35 allocs/op\n\tBenchmarkUint32Less-4                                100          10776367 ns/op       5863804 B/op          35 allocs/op\n\tBenchmarkUint32Delta-4                               100          17705522 ns/op      22995210 B/op          39 allocs/op\n\tBenchmarkUint32Sum-4                                2000            760287 ns/op             0 B/op           0 allocs/op\n\tBenchmarkBoolAND-4                                   500           3306324 ns/op             0 B/op           0 allocs/op\n\tBenchmarkSelectAllMoreThanDollar-4                   100          12233875 ns/op       5863804 B/op          35 allocs/op\n\tBenchmarkFiniteString32Within-4                      100          23427404 ns/op      11727673 B/op          72 allocs/op\n\tBenchmarkSelectAllMoreThanDollarMaterial-4            20          77372820 ns/op      52935826 B/op          76 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTen-4                 50          27022548 ns/op      11727608 B/op          70 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTenMaterial-4         20          73027185 ns/op      48633354 B/op         111 allocs/op\n\tBenchmarkSelectAfterTimeWiseMidPoint-4               100          15291031 ns/op       5863803 B/op          35 allocs/op\n\tBenchmarkLatestPriceMaterial-4                       100          24598447 ns/op      11731457 B/op          83 allocs/op\n\tok      github.com/Everlag/test 104.711s\n\n\n\tRoaringBitmap/roaring\n\tPASS(one off errors, I skipped...)\n\tBenchmarkUint32More-4                                 50          27632560 ns/op        595473 B/op         283 allocs/op\n\tBenchmarkUint32Less-4                                 50          27661220 ns/op        595473 B/op         283 allocs/op\n\tBenchmarkUint32Delta-4                               100          18106963 ns/op      22995212 B/op          40 allocs/op\n\tBenchmarkUint32Sum-4                                2000            751310 ns/op             0 B/op           0 allocs/op\n\tBenchmarkBoolAND-4                                 20000             96504 ns/op             0 B/op           0 allocs/op\n\tBenchmarkSelectAllMoreThanDollar-4                    50          26896896 ns/op        595473 B/op         283 allocs/op\n\tBenchmarkFiniteString32Within-4                      100          13598676 ns/op          4288 B/op         141 allocs/op\n\tBenchmarkSelectAllMoreThanDollarMaterial-4            20          98438055 ns/op      47667796 B/op         341 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTen-4                 20          58682050 ns/op       1199171 B/op         568 allocs/op\n\tBenchmarkSelectAllMoreDollarLessTenMaterial-4         10         117794540 ns/op      38105236 B/op         625 allocs/op\n\tBenchmarkSelectAfterTimeWiseMidPoint-4                50          23716366 ns/op        297760 B/op         144 allocs/op\n\tBenchmarkLatestPriceMaterial-4                       100          14929403 ns/op         33240 B/op         230 allocs/op\n\tok      github.com/Everlag/test 98.223s\n\n## References\n\n[1] Abadi D, Boncz P, Harizopoulos S, Idreos S, Madden S. The design and implementation of modern column-oriented database systems. Now; 2013 Nov 25.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feverlag%2Fnaive-columstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feverlag%2Fnaive-columstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feverlag%2Fnaive-columstore/lists"}