{"id":16701883,"url":"https://github.com/gootik/hakisql","last_synced_at":"2025-04-10T04:21:13.665Z","repository":{"id":143651283,"uuid":"106699852","full_name":"gootik/hakisql","owner":"gootik","description":"Zero-copy in-memory DB for Erlang with SQL-ish query support","archived":false,"fork":false,"pushed_at":"2020-12-02T04:00:18.000Z","size":72,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T05:35:48.074Z","etag":null,"topics":["datastore","erlang","high-performance","inmemory-db","sql"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gootik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-12T13:57:45.000Z","updated_at":"2024-08-29T19:16:30.000Z","dependencies_parsed_at":null,"dependency_job_id":"9f59dbe8-ec82-4ade-9b52-f063fef03e33","html_url":"https://github.com/gootik/hakisql","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakisql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakisql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakisql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakisql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gootik","download_url":"https://codeload.github.com/gootik/hakisql/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248155433,"owners_count":21056638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datastore","erlang","high-performance","inmemory-db","sql"],"created_at":"2024-10-12T18:46:02.665Z","updated_at":"2025-04-10T04:21:13.659Z","avatar_url":"https://github.com/gootik.png","language":"Erlang","readme":"# HakiSQL [![Build Status](https://travis-ci.org/gootik/hakisql.svg?branch=master)](https://travis-ci.org/gootik/hakisql)\n\nAn in-memory datastore for Erlang that uses [hakicache][1] in the background. This\nallows for querying the data with no copy. \n\n**NOTE**: On OTP 21+, hakicache is not used and instead data is stoed in [persistent term][5] \nstorage built into Erlang.\n\nHakiSQL allows for very basic SQL-like queries.\n\nHakiSQL uses a wide bitmap to index the data, therefore this datastore is only\nuseful for datasets with low cardinality for indexed fields. This also means loading\nthe data is slow but querying is fast.\n\nThis library is by no means \"production\" ready. Even if it is, I do not recommend\nusing it. Especially if you load/insert data often.\n\n### WHY\nTo learn about Bitmap indexes and fast datastore access path problems.\n\n### Persistence\n:warning: Does not exist. Restart the VM and your data is gone :)\n\n### TODO\n1. Range query use range bitmap index\n2. Inserting rows need to recalculate index\n3. Persistence?\n4. More tests for queries\n\n### Example\n```erlang\nErlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [kernel-poll:false]\n\nEshell V9.1  (abort with ^G)\n1\u003e  hakisql:create(test, #{\n1\u003e      a =\u003e [index, atom],\n1\u003e      b =\u003e [index, number],\n1\u003e      c =\u003e [number],\n1\u003e      name =\u003e [string]\n1\u003e  }).\nok\n\n2\u003e hakisql:insert(test, [\n2\u003e     #{a =\u003e test, b =\u003e 2, c =\u003e 3.1, name =\u003e \"A\"},\n2\u003e     #{a =\u003e test2, b =\u003e 24, c =\u003e 12.1, name =\u003e \"B\"},\n2\u003e     #{a =\u003e test, b =\u003e 4, c =\u003e 12.1, name =\u003e \"C\"},\n2\u003e     #{a =\u003e test2, b =\u003e 24, c =\u003e 12.1, name =\u003e \"D\"},\n2\u003e     #{a =\u003e test, b =\u003e 8, c =\u003e 12.1, name =\u003e \"E\"},\n2\u003e     #{a =\u003e test4, b =\u003e 24, c =\u003e 12.1, name =\u003e \"F\"}\n2\u003e ]).\nok\n\n3\u003e hakisql:q(test, \"(b = 2 OR b = 8) AND a = test\").\n[#{a =\u003e test,b =\u003e 2,c =\u003e 3.1,name =\u003e \"A\"},\n #{a =\u003e test,b =\u003e 8,c =\u003e 12.1,name =\u003e \"E\"}]\n```\n\n### Query Language\nThe language is a basic version of popular SQL:\n\n### Operations\n#### And\n`expresion AND expression`\nReturn rows where both expressions are true.\n\n#### Or\n`expression OR expression`\nReturn rows where either expression is true.\n\n### =\n`field = values`\nReturn rows where field value is equal to value.\n\n### !=\n`field = values`\nReturn rows where field value is not equal to value.\n\n### \u003e / \u003c / \u003c= / =\u003e\n:warning: Not Implemented.\n\n`field (\u003e/\u003c/\u003c=/=\u003e) values`\nReturn rows where field equality is tested against value.\nComparison follows [Erlang standard](http://erlang.org/doc/reference_manual/expressions.html#term-comparisons).\n\n#### In\n`field in (values)`\nReturn rows where field value is one of values.\n\n#### Not\n`field not in (values)`\nReturn rows where expression value is not any of values.\n\n### Value Representation\n#### Atoms\n`\"field = atom\"`\n#### String\n`\"field = 'string'\"`\n#### Number\n`\"field = 23 OR field = 2.3\"`\n#### Binary\n`\"field = \u003c\u003c'binary'\u003e\u003e\"`\n#### Tuple\n:warning: Has bugs.\n\n`\"field = {a,b,23}\"`\n#### Map\n:warning: Not implemented.\n#### Record\n:warning: Not implemented.\n\n### Benchmarks\nUsing [eministat][4] to see if there is a significant\ndifference between [1000 searches in 10000 \"complex\" records][6]\nusing hakisql (persistent term storage) or ETS:\n```\nDataset: x (ETS) N=1000 CI=95.0000\nStatistic     Value     [         Bias] (Bootstrapped LB‥UB)\nMin:            783.000\n1st Qu.         826.000\nMedian:         861.000\n3rd Qu.         913.000\nMax:            3413.00\nAverage:        910.550 [    -0.127774] (      900.227 ‥       924.029)\nStd. Dev:       191.568 [     -1.55408] (      161.391 ‥       249.685)\n\nOutliers: 0/103 = 103 (μ=910.422, σ=190.014)\n        Outlier variance:      0.994651 (severe, the data set is probably unusable)\n\n------\n\nDataset: + (HakiSQL) N=1000 CI=95.0000\nStatistic     Value     [         Bias] (Bootstrapped LB‥UB)\nMin:            152.000\n1st Qu.         153.000\nMedian:         154.000\n3rd Qu.         160.000\nMax:            264.000\nAverage:        160.469 [   3.14420e-3] (      159.617 ‥       161.479)\nStd. Dev:       14.9246 [  -3.03244e-2] (      13.1941 ‥       16.9747)\n\nOutliers: 0/140 = 140 (μ=160.472, σ=14.8943)\n        Outlier variance:      0.970293 (severe, the data set is probably unusable)\n\nDifference at 95.0% confidence\n        -750.081 ± 11.9095\n        -82.3767% ± 1.30795%\n        (Student's t, pooled s = 135.870)\n------\n```\n\n\nFor small tables, ETS always wins. Bigger tables HakiSQL is much faster and more consistent in timing.\n\nLook at [this perf test][3] for more info\n\n### References\nThe Lexer and Parser code were originally copied/modified from [swirl][2].\n\n\n[1]: https://github.com/gootik/hakicache\n[2]: https://github.com/lpgauth/swirl\n[3]: https://github.com/gootik/hakisql/blob/master/test/hakisql_perf_test.erl#L13\n[4]: https://github.com/jlouis/eministat\n[5]: http://erlang.org/doc/man/persistent_term.html\n[6]: https://github.com/gootik/hakisql/blob/master/benchmark/bechmark.erl\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgootik%2Fhakisql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgootik%2Fhakisql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgootik%2Fhakisql/lists"}