{"id":16701900,"url":"https://github.com/gootik/hakicache","last_synced_at":"2025-04-10T04:21:11.077Z","repository":{"id":57504093,"uuid":"97612900","full_name":"gootik/hakicache","owner":"gootik","description":"Constant pool cache for Erlang for massive data","archived":false,"fork":false,"pushed_at":"2019-06-09T07:06:30.000Z","size":47,"stargazers_count":10,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-24T05:35:43.547Z","etag":null,"topics":["cache","erlang","ets","fast","performance"],"latest_commit_sha":null,"homepage":"","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gootik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-07-18T15:08:31.000Z","updated_at":"2024-07-16T23:06:03.000Z","dependencies_parsed_at":"2022-08-27T23:53:28.661Z","dependency_job_id":null,"html_url":"https://github.com/gootik/hakicache","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakicache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakicache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakicache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gootik%2Fhakicache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gootik","download_url":"https://codeload.github.com/gootik/hakicache/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248027407,"owners_count":21035594,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","erlang","ets","fast","performance"],"created_at":"2024-10-12T18:46:07.649Z","updated_at":"2025-04-10T04:21:11.059Z","avatar_url":"https://github.com/gootik.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"Haki Cache [![Build Status](https://travis-ci.org/gootik/hakicache.svg?branch=master)](https://travis-ci.org/gootik/hakicache) [![Hex.pm](https://img.shields.io/hexpm/v/haki.svg)](https://hex.pm/packages/haki)\n=====\n\nHaki cache is a library that compiles your data into a static module\nin Erlang for fast access. Haki was created because of a need for\nstoring _huge_ (megabytes of data) static state that is needed to be\nshared between processes. \n\nThis has obviously been done before, however most solutions do not\nwork well with megabytes of data. For reference have a look at the following:\n * [Mochiglobal](https://github.com/mochi/mochiweb/blob/master/src/mochiglobal.erl)\n * [Fling](https://github.com/basho-labs/fling) \n * [Talk by Mark Allen about Fling](http://www.erlang-factory.com/static/upload/media/1459269312665211markallenwhenetsistooslow.pdf)\n * [Exploiting Constant Pool by Yoshihiro Tanaka](https://www.youtube.com/watch?v=hcGkT3Czd7U)\n \n### Note\nSome of the hacks used in this library are very risky and unstable. Please use \nat your own risk. I don't recommend using this library unless you have \nextensive tests around your runtime system and make sure it works\nfor your use case.\n\nThere is a reason it's called Hacky Cache after all.\n\n### Usage\n```erlang\nErlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [kernel-poll:false]\n\nEshell V9.0  (abort with ^G)\n1\u003e haki:cache(some_key, \"VALUE\").\nok\n\n2\u003e haki:get(some_key).\n\"VALUE\"\n```\n\nYou can also group a set of key/values into a single module by using the bucketing feature.\nYou have to pass an atom for the bucket name and a map of `atom() =\u003e any()` for caching:\n```erlang\nErlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [kernel-poll:false]\n\nEshell V9.0  (abort with ^G)\n1\u003e haki:cache_bucket(people, #{\n1\u003e     mike =\u003e #{age =\u003e 10, height =\u003e 180, weight =\u003e 65},\n1\u003e     christina =\u003e #{age =\u003e 10, height =\u003e 160, weight =\u003e 45}\n1\u003e }).\nok\n\n2\u003e haki:get(people, mike).\n#{age =\u003e 10,height =\u003e 180,weight =\u003e 65}\n\n3\u003e haki:get(people, christina).\n#{age =\u003e 10,height =\u003e 160,weight =\u003e 45}\n\n4\u003e haki:get(people, no_name).\nbad_key\n\n```\n\n### Super experimental features\nThe `haki_beam` compiler skips all the steps that is required\nto go from a `.S` file to a binary representation of the module.\nThe specifics of the BEAM file format is available here: https://happi.github.io/theBeamBook/#CH-beam_modules.\nIn OTP20+ we can piggy back the `beam_asm` module and directly get a \nbinary BEAM format. This is how `haki_beam_compiler` works and it is only\navailable in OTP20+: \n```erlang\nErlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:0] [hipe] [kernel-poll:false]\n\nEshell V9.0  (abort with ^G)\n1\u003e haki:cache(some_haki_key, \"VALUE\", #{compiler =\u003e haki_beam_compiler}).\nok\n\n2\u003e haki:get(some_haki_key).\n\"VALUE\"\n```\n\n### Benchmarks\nLots of small/medium sized keys:\n```erlang\nfile size: 1781 MB\nnum keys: 4554\navg. key length: 340\n\nlists:foreach(\n  fun(K) -\u003e\n     haki:cache(K, maps:get(K, Map), #{compiler =\u003e haki_beam_compiler})\n  end, maps:keys(Map)).\n  \nCaching of 4554 keys took 1 minute and 55 seconds.\n```\nFew large keys:\n```erlang\nfile size: 177.319806098938Mb\nnum keys: 3\navg. key length: 27000\n\nhaki_asm_compiler: 49,646ms\nhaki_beam_compiler: 1,389ms\n```\n\n### Troubleshooting\nIf you are loading a massive amount of data, you may run into this error:\n```\nliteral_alloc: Cannot allocate ____ bytes of memory (of type \"literal\").\n```\n\nThis is because Erlang VM by default only allows for 1GB of literal data loaded in the VM (http://erlang.org/doc/man/erts_alloc.html). \nTo go around this, you can add `+MIscs \u003csize in MB\u003e` to your VM configs at runtime and everything should work (keep in mind this is only available on 64-bit machines). \n\nI suggest you using 2x of the amount of data you are expecting to load into the VM, this way your VM will not crash if you are reloading or rebuilding the cache.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgootik%2Fhakicache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgootik%2Fhakicache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgootik%2Fhakicache/lists"}