{"id":15362295,"url":"https://github.com/gyson/ane","last_synced_at":"2025-04-15T07:31:04.308Z","repository":{"id":57478859,"uuid":"163391601","full_name":"gyson/ane","owner":"gyson","description":"Ane (atomics and ets) is a library to share mutable data efficiently by utilizing atomics and ets modules.","archived":false,"fork":false,"pushed_at":"2019-04-29T02:05:35.000Z","size":29,"stargazers_count":30,"open_issues_count":1,"forks_count":0,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-03-28T18:21:20.196Z","etag":null,"topics":["atomics","elixir","erlang","ets","mutable","shared"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gyson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-28T09:20:41.000Z","updated_at":"2024-11-19T20:14:20.000Z","dependencies_parsed_at":"2022-09-17T04:21:49.722Z","dependency_job_id":null,"html_url":"https://github.com/gyson/ane","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyson%2Fane","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyson%2Fane/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyson%2Fane/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gyson%2Fane/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gyson","download_url":"https://codeload.github.com/gyson/ane/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249026715,"owners_count":21200494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["atomics","elixir","erlang","ets","mutable","shared"],"created_at":"2024-10-01T12:59:59.230Z","updated_at":"2025-04-15T07:31:04.054Z","avatar_url":"https://github.com/gyson.png","language":"Elixir","readme":"# Ane\n\nAne (atomics and ets) is a library to share mutable data efficiently by\nutilizing [atomics](http://erlang.org/doc/man/atomics.html) and\n[ets](http://erlang.org/doc/man/ets.html) modules.\n\n## How it works ?\n\n* It stores all data with versionstamp in ETS table.\n* It keeps a cached copy with versionstamp locally.\n* It uses atomics to save latest versionstamp and syncs data between ETS table and local cache.\n* Read operation would use cached data if cache hits and fallback to ETS lookup if cache expires.\n* Write operation would update ETS table and versionstamp in atomics array.\n\n## Properties\n\nSimilar to atomics standalone,\n\n* Ane's read/write operations guarantee atomicity.\n* Ane's read/write operations are mutually ordered.\n* Ane uses one-based index.\n\nCompare to atomics standalone,\n\n* Ane could save arbitrary term instead of 64 bits integer.\n\nCompare to ETS standalone,\n\n* Ane has much faster read operation when cache hit (this is common for read-heavy application).\n\n  - It needs 1 Map operation and 1 atomics operation.\n  - It does not need to copy data from ETS table.\n  - It does not need to lookup from ETS table, which could make underneath ETS table's write operation faster.\n  - Benchmarking showed that it's 2 ~ 10+ times faster.\n\n* Ane could have slightly slower read operation when cache missed or expired.\n\n  - It needs 2 Map operations, 1+ atomics operations and 1+ ETS operations.\n  - Ane could be faster for \"hot key\" case.\n\n* Ane could have slower write operation.\n\n  - It needs to do 2 ETS operations and 2+ atomics operations.\n  - Ane could be faster for \"hot key\" case.\n\n* Ane has much faster read/write operations for \"hot key\" case.\n\n  - ETS table performance degrades when a key is too hot due to internal locking.\n  - Ane avoids \"hot key\" issue by distributing read/write operations to different keys in underneath ETS table.\n\n* Ane only supports `:atomics`-like one-based index as key.\n\n  - I feel it's possible to extend it to be `:ets`-like arbitrary key with some extra complexity. But I do not have that need at the moment.\n\nCompare to [persistent_term](http://erlang.org/doc/man/persistent_term.html),\n\n* Like persistent_term, Ane's read operation with cache hit is lock-free and copying-free (no need to copy since data exists in local cache).\n\n* Unlike persistent_term, Ane's read operation with cache miss/expire would require copy data from ETS table to the heap of current process.\n\n* Unlike persistent_term, Ane's write operation is fast and won't trigger global GC.\n\n## Installation\n\n**Note**: it requires OTP 21.2 for `:atomics`, which was released on Dec 12, 2018.\n\nIt can be installed by adding `:ane` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:ane, \"~\u003e 0.1.1\"}\n  ]\nend\n```\n\nAPI reference can be found at [https://hexdocs.pm/ane/Ane.html](https://hexdocs.pm/ane/Ane.html).\n\n## Usage\n\n```elixir\niex(1)\u003e a = Ane.new(1)\n{#Reference\u003c0.376557974.4000972807.196270\u003e,\n #Reference\u003c0.376557974.4000972807.196268\u003e,\n #Reference\u003c0.376557974.4000972807.196269\u003e, %{}}\niex(2)\u003e Ane.put(a, 1, \"hello\")\n:ok\niex(3)\u003e {a, value} = Ane.get(a, 1)\n{{#Reference\u003c0.376557974.4000972807.196270\u003e,\n  #Reference\u003c0.376557974.4000972807.196268\u003e,\n  #Reference\u003c0.376557974.4000972807.196269\u003e, %{1 =\u003e {1, \"hello\"}}}, \"hello\"}\niex(4)\u003e value\n\"hello\"\niex(5)\u003e Ane.put(a, 1, \"world\")\n:ok\niex(6)\u003e {a, value} = Ane.get(a, 1)\n{{#Reference\u003c0.376557974.4000972807.196270\u003e,\n  #Reference\u003c0.376557974.4000972807.196268\u003e,\n  #Reference\u003c0.376557974.4000972807.196269\u003e, %{1 =\u003e {2, \"world\"}}}, \"world\"}\niex(7)\u003e value\n\"world\"\n```\n\n## Compare Ane and ETS Standalone\n\nGenerally, Ane is faster for read-heavy case and ETS standalone is faster for write-heavy case. This library provide a way to switch between them seamlessly.\n\nBy specify `mode: :ets` as following, it will use ETS standalone instead:\n\n```elixir\niex(1)\u003e a = Ane.new(1, mode: :ets)\n{#Reference\u003c0.2878440188.2128478212.58871\u003e, 1}\niex(2)\u003e Ane.put(a, 1, \"hello\")\n:ok\niex(3)\u003e {a, value} = Ane.get(a, 1)\n{{#Reference\u003c0.2878440188.2128478212.58871\u003e, 1}, \"hello\"}\niex(4)\u003e value\n\"hello\"\niex(5)\u003e Ane.put(a, 1, \"world\")\n:ok\niex(6)\u003e {a, value} = Ane.get(a, 1)\n{{#Reference\u003c0.2878440188.2128478212.58871\u003e, 1}, \"world\"}\niex(7)\u003e value\n\"world\"\n```\n\nThis is useful for comparing performance between Ane and ETS standalone.\n\n## Performance Tuning\n\nThe `read_concurrency` and `write_concurrency` from ETS table are important configurations for performance tuning. You can adjust it while creating Ane instance like following:\n\n```elixir\nane = Ane.new(1, read_concurrency: true, write_concurrency: true)\n```\n\nThese options would be passed to underneath ETS table. You can read more docs about `read_concurrency` and `write_concurrency` at [erlang ets docs](http://erlang.org/doc/man/ets.html#new-2).\n\n## Benchmarking\n\nBenchmarking script is available at `bench/comparison.exs`.\n\nFollowing is the benchmarking result for comparing Ane and ETS standalone with 90% read operations and 10% write operations:\n\n```\n$ mix run bench/comparison.exs\nOperating System: macOS\"\nCPU Information: Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz\nNumber of Available Cores: 8\nAvailable memory: 16 GB\nElixir 1.7.4\nErlang 21.2\n\nBenchmark suite executing with the following configuration:\nwarmup: 2 s\ntime: 10 s\nmemory time: 0 μs\nparallel: 16\ninputs: none specified\nEstimated total run time: 24 s\n\n\nBenchmarking size=16, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100...\nBenchmarking size=16, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100...\n\nName                                                                                                                   ips        average  deviation         median         99th %\nsize=16, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100         26.76       37.37 ms    ±37.32%       36.79 ms       72.50 ms\nsize=16, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100          9.66      103.55 ms    ±37.82%       98.66 ms      187.74 ms\n\nComparison:\nsize=16, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100         26.76\nsize=16, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100          9.66 - 2.77x slower\n```\n\nFollowing is the benchamrking result for comparing Ane and ETS standalone for \"hot key\" issue:\n\n```\n$ mix run bench/comparison.exs\nOperating System: macOS\"\nCPU Information: Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz\nNumber of Available Cores: 8\nAvailable memory: 16 GB\nElixir 1.7.4\nErlang 21.2\n\nBenchmark suite executing with the following configuration:\nwarmup: 2 s\ntime: 10 s\nmemory time: 0 μs\nparallel: 16\ninputs: none specified\nEstimated total run time: 24 s\n\n\nBenchmarking size=1, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100...\nBenchmarking size=1, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100...\n\nName                                                                                                                  ips        average  deviation         median         99th %\nsize=1, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100         27.03       37.00 ms    ±45.40%       36.15 ms       71.12 ms\nsize=1, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100          1.33      754.31 ms    ±25.91%      762.88 ms     1212.87 ms\n\nComparison:\nsize=1, mode=ane, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100         27.03\nsize=1, mode=ets, Ane.get=90%, Ane.put=10.0%,  read_concurrency=true, write_concurrency=true, info_size=100          1.33 - 20.39x slower\n```\n\n## Handling Garbabge Data in Underneath ETS table\n\nWrite operation (`Ane.put`) includes one `:ets.insert` operation and one `:ets.delete` operation.\nWhen the process running `Ane.put` is interrupted (e.g. by `:erlang.exit(pid, :kill)`), garbage\ndata could be generated if it finished insert operation but did not start delete operation. These\ngarbabge data could be removed by calling `Ane.clear` (periodically if it needs to handle constantly interruptions).\n\n## Development Note\n\n```sh\n# type check with dialyzer\nmix dialyzer\n\n# type check with ex_type\nmix type\n```\n\n## License\n\nMIT\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyson%2Fane","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgyson%2Fane","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgyson%2Fane/lists"}