{"id":19725740,"url":"https://github.com/esl/cets","last_synced_at":"2025-04-30T00:31:40.520Z","repository":{"id":40426903,"uuid":"490182857","full_name":"esl/cets","owner":"esl","description":"A library to synchronise records from the ETS tables between nodes","archived":false,"fork":false,"pushed_at":"2024-12-11T13:21:38.000Z","size":616,"stargazers_count":25,"open_issues_count":2,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-26T00:48:42.040Z","etag":null,"topics":["clustering","database","erlang","in-memory-database","mnesia"],"latest_commit_sha":null,"homepage":"https://hexdocs.pm/cets/cets.html","language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2022-05-09T07:42:48.000Z","updated_at":"2025-04-04T02:14:02.000Z","dependencies_parsed_at":"2023-10-23T06:26:53.281Z","dependency_job_id":"b270f278-1f74-4734-90a4-6b63d16841db","html_url":"https://github.com/esl/cets","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esl%2Fcets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esl%2Fcets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esl%2Fcets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esl%2Fcets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esl","download_url":"https://codeload.github.com/esl/cets/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250917358,"owners_count":21507561,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clustering","database","erlang","in-memory-database","mnesia"],"created_at":"2024-11-11T23:33:00.147Z","updated_at":"2025-04-30T00:31:40.057Z","avatar_url":"https://github.com/esl.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cluster ETS\n\n[![](https://github.com/esl/cets/workflows/CI/badge.svg)](https://github.com/esl/cets/actions?query=workflow%3ACI)\n[![Hex Package](http://img.shields.io/hexpm/v/cets.svg)](https://hex.pm/packages/cets)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/cets/)\n[![codecov](https://codecov.io/github/esl/cets/graph/badge.svg?token=R1zXAjO7H7)](https://codecov.io/github/esl/cets)\n\nThe project adds replication support for Erlang Term Storage (ETS).\n\nIt allows to insert or delete objects into ETS tables across several Erlang nodes.\n\nThe closest comparison is Mnesia with dirty operations.\n\nSome features are not supported:\n- there is no schema\n- there is no persistency\n- there are no indexes\n- there are no transactions (there are bulk inserts/deletes instead).\n\nDeveloped by [Erlang Solutions](https://www.erlang-solutions.com/) to be used in [MongooseIM](https://github.com/esl/MongooseIM/).\nIt could be used in other Erlang or Elixir projects.\n\n# Merging logic\n\nWhen two database partitions are joined together, records from both partitions\nare put together. So, each record would be present on each node.\n\nThe idea that each node updates only records that it owns. The node name\nshould be inside an ETS key for this to work (or a pid).\n\nWhen some node is down, we remove all records that are owned by this node.\nWhen a node reappears, the records are added back.\n\nYou can also use totally random keys. In this case all keys should be unique.\nWe also support tables with type=bag. In such case all records would remain in the table.\n\nYou can provide the `handle_conflict` function, which would be called if two records have\nthe same key when clustering (just be aware, this function would not protect you against\noverwrites by the same node).\n\nWithout `handle_conflict`, conflicting records would overwrite each other when joining.\nWhen joining nodes we insert all data from one node to another and vice versa without any extra check.\nCheck `join_works_with_existing_data_with_conflicts` test case as an example.\n\n# API\n\nThe main module is cets.\n\nIt exports functions:\n\n- `start(Tab, Opts)` - starts a new table manager.\n   There is one gen_server for each node for each table.\n- `insert(Server, Rec)` - inserts a new object into a table.\n- `insert_many(Server, Records)` - inserts several objects into a table.\n- `delete(Server, Key)` - deletes an object from the table.\n- `delete_many(Server, Keys)` - deletes several objects from the table.\n\n`cets_join` module contains the merging logic.\n\n`cets_discovery` module handles search of new nodes.\n\nIt supports behaviours for different backends.\n\nIt defines two callbacks:\n\n- `init/1` - inits the backend.\n- `get_nodes/1` - gets a list of alive erlang nodes.\n\nOnce new nodes are found, `cets_discovery` calls `cets_join` module to merge two\ncluster partitions.\n\nThe simplest `cets_discovery` backend is `cets_discovery_file`, which just reads\na file with a list of nodes on each line. This file could be populated by an\nexternal program or by an admin.\n\n# Commands\n\nBefore making a new pull request run tests:\n\n```\nrebar3 all\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesl%2Fcets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesl%2Fcets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesl%2Fcets/lists"}