{"id":13672046,"url":"https://github.com/scality/elmerfs","last_synced_at":"2025-04-22T01:31:04.403Z","repository":{"id":43917953,"uuid":"269384179","full_name":"scality/elmerfs","owner":"scality","description":null,"archived":false,"fork":false,"pushed_at":"2022-02-21T16:33:55.000Z","size":784,"stargazers_count":99,"open_issues_count":6,"forks_count":2,"subscribers_count":49,"default_branch":"master","last_synced_at":"2025-04-01T17:53:34.970Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/scality.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-04T14:38:45.000Z","updated_at":"2025-03-04T11:34:05.000Z","dependencies_parsed_at":"2022-09-02T12:52:30.992Z","dependency_job_id":null,"html_url":"https://github.com/scality/elmerfs","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Felmerfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Felmerfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Felmerfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/scality%2Felmerfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/scality","download_url":"https://codeload.github.com/scality/elmerfs/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250161968,"owners_count":21385014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T09:01:25.209Z","updated_at":"2025-04-22T01:31:03.744Z","avatar_url":"https://github.com/scality.png","language":"Rust","funding_links":[],"categories":["Rust","others"],"sub_categories":[],"readme":"elmerfs\n-------\n\nelmerfs is a filesystem that leverage **Conflict Free Replicated Data Type**\n[CRDT](https://crdt.tech/) on top of [AntidoteDB](https://www.antidotedb.eu/)\nto be eventually consistent in a active-active geo distributed scenario.\n\n### Building the project\n\nThe project require a recent version of rustc (tested with 1.42):\n\n```\ngit clone git@github.com:scality/elmerfs.git\ncd elmerfs\ngit submodule update --init --recursive\ncargo run --bin main -- [OPTIONS]\n```\n\nWhen running the main binary, you will have the following options:\n\n```\nelmerfs\n\nUSAGE:\n    main [FLAGS] [OPTIONS] --mount \u003cMOUNTPOINT\u003e\n\nFLAGS:\n    -h, --help        Prints help information\n        --no-locks\n    -V, --version     Prints version information\n\nOPTIONS:\n    -s, --antidote \u003cURL\u003e...       [default: 127.0.0.1:8101]\n        --force-view \u003cUID\u003e\n        --listing-flavor \u003clf\u003e     [default: partial]  [possible values: full, partial]\n    -m, --mount \u003cMOUNTPOINT\u003e\n```\n\nA usual launch will also include logs and backtrace env variables, for example, to\nrun the fs on a local antidote cluster:\n\n```\nRUST_BACKTRACE=1 RUST_LOG=info cargo run --release --bin main -- --mount ../elmerfsmount/ --antidote=127.0.0.1:8101 --antidote=127.0.0.1:8102 --antidote=127.0.0.1:8103 --no-locks\n```\n\nNote that is is important that all antidote IPs address are from the same\ndatacenter !\n\n### Specifics notions\n\n#### The View\n\nEach `elmerfs` process have one or multiple view attached to it.\nFor convenience the view is based on the user which perform the operations.\nIt is expected that the user cannot perform conflicting update by himself against\nhimself. This means that an user should always contact and perform operation\non the same process.\n\nEach time you create file or a directory, the user that it was created from\nwill be saved alongside the metadata.\n\n#### Naming\n\nIn relation to the view id. There is two way to refer to a file in `elmerfs`.\nYou can either refer to it by what you are using to, by its name or,\nby its fully qualified name `name:view_id`.\n\nFor example if you are the root user:\n\n```bash\necho \"Hello\" \u003e file:root\necho \" World!\" \u003e\u003e file\ncat file\nHello World!\n```\n\nNote that consequently `:` as a special meaning.\n\n#### Lookup Resolution\n\nAs you saw above, when doing a lookup, you have two ways to refer a file.\n\nUsing the short alias, `elmerfs` will use a simple algorithm to resolve the lookup.\nFirst it will check if there is only one entry in the parent dir with this name. If so,\nthe lookup is successful and the entry is returned **even if the entry as a different view id associated\nto it**.\n\nIf they are multiple entries, this mean that there was a concurrent update when updating the parent directory.\nIn this case, we try to find if an entry with the same view id as the running process exists.\n\nFor example lets say you have **elmerfs0** and **elmerfs1**:\n\n```\nelmerfs0/dir\u003e touch f\n```\n\n```\nelmerfs1/dir\u003e touch f\n```\n\nAt this point there is two file in the same directory, if we perform a listing\non both site, you will see the following:\n\n```\nelmerfs0/dir\u003e ls\nf\nf:toto\necho \"Hello from elmerfs0' \u003e\u003e f\n```\n\n```\nelmerfs1/dir\u003e ls\nf\nf:root\ncat f:root\nHello from elmerfs0\n```\n\nWhat is interesting here is that applications on both process will work\nseamlessly on their file without interruptions. To resolve the conflict you can\nsimply rename the file.\n\nThis works the same on directories.\n\n#### Locking\n\n`elmerfs` should work without any distributed locking, you can specify `no-locks` to avoid them.\nWhen locks are used, no conflicts can happen, but you lose latency and availability.\n\n#### Atomicity\n\nEvery fs operation is synchronous and done inside a unique transaction,\nmeaning that if an operation fails, nothing will be committed.\n\n### Running chton tests\n\nCthon04 test suite must be built\nfrom the 3rdparty submodule and copied into a vendor directory at the root\nof this project.\n\nThen, a simple `cargo test` should work.\n\n```\n$ destdir=$(realpath ./vendor/chton04)\n$ mkdir -p $destdir\n$ (cd 3rdparty/cthon04 \u0026\u0026 make all \u0026\u0026 make copy DESTDIR=$destdir)\n$ cargo test\n```\n\n### State of the project\n\n**elmerfs** is still in its early stage, basic fs operation are implemented\nand the aim is to provide a prototype that is valid and resilient\nbut not necessarily fast.\n\nThe project is able to pass basics and general connectathon test suites.\nFsx from the linux test project is also able to run for\nan extended period of time without issues.\n\nMore tests will be added in the future to check concurrent update handling.\n\nNote that **concurrent update on file content** is not well handled yet.\nConflicts resolution of such update is done at the register level,\nnot at the update boundaries.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Felmerfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fscality%2Felmerfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fscality%2Felmerfs/lists"}