{"id":16747126,"url":"https://github.com/selfup/exdaas","last_synced_at":"2025-04-10T13:43:38.979Z","repository":{"id":71266499,"uuid":"126121150","full_name":"selfup/exdaas","owner":"selfup","description":"RESTful Elixir Database as a Service! Fault Tolerant, Self Sharding, Self Caching, Single Table, Persistent Key Value Store :tada:","archived":false,"fork":false,"pushed_at":"2018-09-24T04:20:48.000Z","size":60,"stargazers_count":9,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-24T12:21:53.712Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/selfup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-21T04:00:20.000Z","updated_at":"2024-01-30T07:06:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"2e404b6c-d89a-4afe-af00-5427b8dbcc48","html_url":"https://github.com/selfup/exdaas","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selfup%2Fexdaas","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selfup%2Fexdaas/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selfup%2Fexdaas/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/selfup%2Fexdaas/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/selfup","download_url":"https://codeload.github.com/selfup/exdaas/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248226367,"owners_count":21068191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T02:09:11.362Z","updated_at":"2025-04-10T13:43:38.971Z","avatar_url":"https://github.com/selfup.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ExDaas\n\nElixir Database as a Service\n\nSerialized - fault tolerant - self caching - self sharding - NoSQL DB :rocket:\n\nProvides a RESTful API that can handle concurrent requests (Phoenix) but serializes all writes to disk.\n\nAll serialized writes can be split up by amount of shards.\n\nDefault shard size is 4. Any other wanted size can be set via `SHARD_LIMIT` (Any number above 0).\n\nDiskIO is delegated via [DETS](http://erlang.org/doc/man/dets.html) and all cache is handled using [ETS](https://elixir-lang.org/getting-started/mix-otp/ets.html).\n\nIf the entire app fails, the data is loaded form disk into cache, and performance is back to normal.\n\nExample supervision tree of a default shard size (4):\n\n![](https://user-images.githubusercontent.com/9837366/37997853-005b93e0-31e2-11e8-9fe7-0e33eb54f943.PNG) \n\n_Suprisingly performant_ :smile:\n\n## Development\n\n**Deps**:\n\n1. Docker\n1. Elixir\n1. Bash (masOS, Linux, WSL (Win10))\n\n**On first boot**:\n\n`./scripts/test.sh`\n\n1. Generate Secret\n1. Ensure all needed directories are made\n1. Ensure everything is installed\n1. Run tests\n1. Create an Alpine Docker Container\n1. Build release (not exposed, just testing it can build)\n\nEssentially the _bootstrapping_ script :rocket:\n\nCalled test, because it ensures that your dev enviornment is ready to roll, and it runs tests :smile:\n\n## Deploying to Heroku (for testing purposes)\n\n**WARNING: THIS SHOULD ONLY BE FOR TESTING PURPOSES**\n\n_Heroku will not keep the `dets_*` files on reboot/rebuild since it is not in git history_\n\nMake sure the container builds (heroku will rebuild it anyways but just be sure it works)!\n\n`./scripts/test.sh`\n\n### If not logged in to Heroku\n\n```bash\nheroku login\nheroku container:login\n```\n\nNow: `APP_NAME=\u003capp_name\u003e ./scripts/heroku.sh`\n\n## Deploying to Digital Ocean/Vultr/EC2\n\nMake sure you have Docker and docker-compose!\nMake sure you have your ssh key as an authorized key for your target node!\n\n### Build the release with Docker\n\n1. In one shell: `./scripts/docker.release.sh`\n1. In another shell (once release is built): `./scripts/docker.copy.release.sh`\n1. Grab tarball and scp: `scp -r ./exdaas.tar.gz user@\u003ctarget_ip\u003e:/home/user`\n1. SSH into your server: `ssh user@\u003ctarget_ip\u003e`\n1. Unpack the tarball: `tar -xzf exdaas.tar.gz`\n1. Make your DETS_ROOT directory: `mkdir -p $DETS_ROOT`\n1. Run the server:\n\n        a. As a Daemon: `DETS_ROOT=\u003cdets_root\u003e PORT=\u003cport\u003e ./bin/exdaas start`\n        b. In the foreground: `DETS_ROOT=\u003cdets_root\u003e PORT=\u003cport\u003e ./bin/exdaas foreground`\n        c. In interactive mode: `DETS_ROOT=\u003cdets_root\u003e PORT=\u003cport\u003e ./bin/exdaas console`\n\n## Backing up data\n\n1. Tarball: `./scripts/archive.tar.sh`\n2. Zip: `./scripts/achrive.zip.sh`\n\nYou may pass `DETS_ROOT`:\n\nExample:\n\n`DETS_ROOT=/home/user/my_persistance_dir ./scripts/archive.tar.sh`\n\n## Current Benchmarks\n\n~13k req/s in an Alpine Docker Container running on Ubuntu 17.10 in production mode on a 2 Core Intel i7 from 2013\n\n**CPU Info**\n\n```bash\nModel name:          Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz\nCPU(s):              4\nOn-line CPU(s) list: 0-3\nThread(s) per core:  2\nCore(s) per socket:  2\nSocket(s):           1\n```\n\n### To run benchmarks\n\nYou will need two tabs/panes/shell for this:\n\n1. Build the container and run it: `./scripts/test.sh`\n2. Wait for: `Attaching to exdaas_prod_1`\n3. Run the bench suite in a different shell/pane/tab: `./scripts/bench.sh`\n\n        a. To keep changes in git HEAD pass the `-c` flag\n        b. Ex: `./scripts/bench.sh -c`\n        c. Otherwise the `.results.log` file will be checked out\n\n### Another Alternative for Benching\n\n_Default sharding is set to 4_\n\n```bash\n./scripts/console.bench.sh\n```\n\n**If you want to increase the shard size**\n\n_You may set `SHARD_LIMIT` to any positive number over 0_\n\n```bash\nSHARD_LIMIT=16 ./scripts/console.bench.sh\n```\n\n_Remember this is synchronous and using a stream or a parallel map can be more realistic_\n\n```elixir\nalias ExDaas.Ets.Table, as: EtsTable\n\ndata = %{color: \"blue\"}\n\n# if you changed SHARD_LIMIT\n# ex: SHARD_LIMIT=24\n# change 0..3 to 0..23 (or limit - 1)\nets_tables = 0..3 |\u003e Enum.map(fn i -\u003e :\"ets_table_#{i}\" end)\n\n# this will be cold cache\n0..20_000 |\u003e Enum.each(fn i -\u003e\n  table_id = rem(i, length(ets_tables))\n  EtsTable.fetch(i, data, Enum.at(ets_tables, table_id))\nend)\n\n# this will be warm cache\n0..20_000 |\u003e Enum.each(fn i -\u003e\n  table_id = rem(i, length(ets_tables))\n  EtsTable.fetch(i, data, Enum.at(ets_tables, table_id))\nend)\n```\n\nExit the shell and `rm dets_*`\n\n## LICENSE\n\n**MIT**\n\nSee: `LICENSE` file in root of project\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselfup%2Fexdaas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fselfup%2Fexdaas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fselfup%2Fexdaas/lists"}