{"id":16666273,"url":"https://github.com/mateusfreira/nun-db","last_synced_at":"2025-04-05T16:07:50.593Z","repository":{"id":37012549,"uuid":"160528020","full_name":"mateusfreira/nun-db","owner":"mateusfreira","description":"A realtime database written in rust","archived":false,"fork":false,"pushed_at":"2025-03-25T10:47:16.000Z","size":805,"stargazers_count":122,"open_issues_count":10,"forks_count":7,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T15:07:23.764Z","etag":null,"topics":["hacktoberfest","realtime","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mateusfreira.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-12-05T14:10:46.000Z","updated_at":"2025-03-18T00:02:58.000Z","dependencies_parsed_at":"2023-09-25T02:08:06.782Z","dependency_job_id":"39932f7e-f984-415d-9e2a-22dc3f7e30de","html_url":"https://github.com/mateusfreira/nun-db","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mateusfreira%2Fnun-db","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mateusfreira%2Fnun-db/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mateusfreira%2Fnun-db/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mateusfreira%2Fnun-db/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mateusfreira","download_url":"https://codeload.github.com/mateusfreira/nun-db/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247361687,"owners_count":20926643,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","realtime","rust"],"created_at":"2024-10-12T11:10:25.174Z","updated_at":"2025-04-05T16:07:50.562Z","avatar_url":"https://github.com/mateusfreira.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"./logo-nundb.png\" width=\"50\" /\u003e  NunDB\n\n## What is NunDB\n\nNunDB is an open source real-time database, made to be fast, light and easy to use.\n\nNunDB is written to be memory-safe and horizontal(soon) scalable.\n\nWe believe we can keep NunDB simple, and at the same time, it can power lots of different apps and use cases.\n\n## Examples Use cases \n\nCheckout our examples of integration and see what we can do for your app\n\n* [Vue.js + NunDB jsfiddle](https://jsfiddle.net/2op63ctk/)\n* [React.js Realtime Todo app](https://github.com/mateusfreira/nun-db-js/tree/master/examples/react)\n\n\n# Installations\n\n## Docker \n\nRunning NunDB using Docker is likely the fastest method. The simplest steps to run NunDB with Docker involve running a container, as shown in the following example.\n\n```bash\n#Change user-name to the admin you want, user-pwd to the admin pwd you want, change /tmp to the directory where you want to store your data in the host computer... for dev /tmp works fine\ndocker run --env NUN_USER=user-name --env NUN_PWD=user-pwd --env NUN_DBS_DIR=\"/data\" --mount type=bind,source=\"/tmp\",target=/data -it --rm -p 3013:3013 -p 3012:3012 -p 3014:3014 --name nun-test mateusfreira/nun-db\n```\n\n## Docker-compose\n\n```yaml\n\nversion: '3'\nservices:\n  nun-db:\n    image: \"mateusfreira/nun-db\"\n    ports:\n      - \"3012:3012\" # Web socket\n      - \"3013:3013\" # Http\n      - \"3014:3014\" # Socket\n    environment:\n      - NUN_DBS_DIR=/nun_data \n      - NUN_USER=mateus\n      - NUN_PWD=mateus\n    volumes:\n        - /tmp/data/nun_db/:/nun_data\n```\n\nNote that the \"/tmp/data/nun_db/\" is a directory in the machine where the nun-db is running so you may need to create the directory for that use the command `mkdir /tmp/data/nun_db/`.\n\n\n### Create the sample DB:\n\n```\n\n# First connect to the running container\ndocker exec -it nun-test  /bin/sh\n\n# Then execute the command to create the database\nnun-db -u user-name -p user-pwd create-db -d sample -t sample-pwd\n\n# You should see something like\nResponse \"valid auth\\n;create-db success\\n\"\n\n```\n\nOnce done, you now have NunDB running in your docker and exposing all the ports for use: HTTP (3013), WebSocket (3012), and Socket (3014)  you are ready to use.\n\n\n\n# How-tos\n* [Real-time Medical Image Collaboration POC Made Easy with OHIF and NunDB](https://mateusfreira.github.io/@mateusfreira-real-time-medical-image-collaboration-ohif-nun-db/)\n\n* [How to create users with different permission levels in NunDB](https://mateusfreira.github.io/@mateusfreira-2023-09-17-how-to-create-users-in-nun-db/)\n\n* [How to Building a Trello-like React/Redux App with NunDB with offline and conflict resolution features](https://mateusfreira.github.io/@mateusfreira-trelo-clone-offline-first-with-nun-db-dealing-with-conflicts/)\n\n* [How-to Make Redux TodoMVC Example a real-time multiuser with NunDB in 10 steps](https://mateusfreira.github.io/@mateusfreira-2021-06-30-how-to-making-redux-todomvc-example-a-real-time-multiuser-with-nun-db/)\n\n* [How to create your simple version of google analytics real-time using NunDB](https://mateusfreira.github.io/@mateusfreira-create-a-simple-verison-of-google-analytics-realtime-using-nun-db/)\n\n* [NunDb How to backup one or all databases](https://mateusfreira.github.io/@mateusfreira-nundb-how-to-backups-all-databases-with-one-command/)\n\n* [How to snapshot NunDB databases from memory to disk](https://mateusfreira.github.io/@mateusfreira-nundb-how-to-backups-all-databases-with-one-command/)\n```bash\n# TLDR version\nnun-db --user $NUN_USER  -p $NUN_PWD --host \"https://http.nundb.org\" exec \"use-db $DB_NAME $DB_TOKEN; snapshot\"\n```\n* [Introduction to manage conflicts in NunDB](https://mateusfreira.github.io/@mateusfreira-dealing-with-conflicts-in-nun-db/)\n\n## Technical documentations\n\n[A fast-to-sync/search and space-optimized replication algorithm written in rust, The NunDB data replication model](https://mateusfreira.github.io/@mateusfreira-a-fast-to-sync-search-and-space-optimized-replication-algorithm-written-in-rust-the-Nun-db-data-replication-model/)\n\n[Leader election in rust the journey towards implementing NunDB leader election](https://mateusfreira.github.io/@mateusfreira-leader-election-rust-the-journey-towards-nun-db-leader-election-implementation/)\n\n[Integration tests in rust a multi-process test example](https://mateusfreira.github.io/@mateusfreira-integration-tests-for-rust-apps-testing-command-line-tools-in-rust/)\n\n[Writing a prometheus exporter in rust from idea to grafana chart](https://mateusfreira.github.io/@mateusfreira-writing-a-prometheus-exporter-in-rust-from-idea-to-grafana-chart/)\n\n[Why does NunDB we have a Debug command?](https://mateusfreira.github.io/@mateusfreira-the-nun-db-debug-command/)\n\n[Code Coverage for Rust Projects with GitHub Actions](https://blog.freira.dev/@mateusfreira-2024-06-15-code-coverage-for-rust-projects-with-github-actions/)\n[![codecov](https://codecov.io/gh/mateusfreira/nun-db/graph/badge.svg?token=yPiTap1MPJ)](https://codecov.io/gh/mateusfreira/nun-db)\n\n## Diagram\n\n```bash\n\n                                      .------------------------------------------.\n                                      |                     |                    |              .---------------.\n                                      |       http_ops      |                    |-------------\u003e|  Disck        |      \n                                      |                     |   disk_ops         |-------------\u003e|               |      \n                                      |_____________        |                    |              .---------------.\n                                      |            |        ._________________.__|                                     \n                                      |            |        |                 |  |                                     \n                                      |            |        | replication_ops |  |                                     \n                                      |   tcp_ops   \\_______+--------+--------+  |                                     \n                                      |             |       |        |           |                                     \n.----------------.                    |             |       |        |           |      .---------------. \n|   Frontends/   | ---text----------\u003e |_____________|parse \u003c.\u003e core  | db_ops    |-----\u003e|   Memory      | \n|  Clients       | \u003c---text---------- |             |       |        |           |\u003c-----| Repository    | \n.----------------.                    |             |       |        |           |      |               | \n                                      |             \\_______|_______/________.   |      ._______________.\n                                      |    ws_ops           |                |   |                                    \n                                      |                     |   election_ops |   |                                    \n                                      |_____________________._______________/____|                                    \n                                      |                                          |                                    \n                                      |                                          |                                    \n                                      |                 monitoring               |                                    \n                                      |                                          |                                    \n                                      |                                          |                                    \n                                      .------------------------------------------.                                    \n\n```\n\n## Connectors \n* Http\n    Port: 3013\n* Socket\n    Port: 3014\n    \n* Web Socket\n    Port: 3012\n\n\n\n## Having any issue or needs help?\n\nOpen an issue I will follow as soon as possible.\n\n## Want to use it in production ?\n\nTalk to me @mateusfreira I will help you get it to the scale you need.\n\n## Nun Query language (NQL)\n\n### Auth\n#### Context\n- [ ] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? ()\n\ne.g:\n```\nauth $user $pwd\n```\n\n### UseDb\n#### Context\n- [ ] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? ()\n\ne.g:\n```\nuse-db $db-name $db-pwd\n```\n#### Alias\n`use`\n\ne.g:\n```\nuse $db-name $db-pwd\n```\n\ne.g: \n```\nuse-db $db-bame $user-name $token\n```\n\n### CreateDb\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [x] Replicate? How? (create-db)\n- [ ] Register Oplog? How? (Update)\nCreates a new database\n\ne.g.,\n\n```\n# create simple db\ncreate-db test test-pwd;\nresponse: \nempty\n```\n\n```\n# create db with arbiter conlifct resolution strategy\ncreate-db test test-pwd arbiter;\nresponse:\nempty\n```\n\n### Get\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How? ()\n\n### Set\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (replicate)\n\n### SetSafe\nChanges the value of a key, guarantees consistency by version\nSoon this will be the default way to set values to a key, since we are moving to a leader less replication model.\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (replicate)\nExamples:\n```\nset-safe $key $version $value\nset-safe name 10 Mateus # Sets key name to mateus if version is equal or minor than \n```\n\n\n### Remove\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (replicate)\n\n### Snapshot $reclaim_space(true|false) $db_names_pipe_separated (Empty means will snapshot the current database only)\n#### Context\n- [x] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (replicate-snapshot)\n\nE.g:\n\n##### Snapshot the databases to disk, faster method will store only the difference will use more disk space.\n```\nsnapshot false db1|db2|db3|db4\n```\n\n##### Snapshot the database to disk, slower method will store all data again to disk and reorganize the values file\n```\nsnapshot true db1|db2|db3|db4\n```\n##### Snapshot the database to disk only the current database\n```\nsnapshot true\n```\n\n### UnWatch\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How? \n\n### UnWatchAll\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How? \n\n\n### Watch\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How? \n\n### Keys\nReturn the list of keys for the database.\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How?\n\ne.gs: \n* Return all keys\n`keys`\n* Return keys starting with name\n`keys name*`\n* Return keys ending with name\n`keys *name`\n* Return keys containing with name\n`keys name`\n\n#### Alias\n`ls`\n\n\n### SetPrimary\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? \n\n### ElectionWin\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? \n\n### Join\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [x] Replicate? How? (replicate-join)\n\n### ReplicateSince\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? \n\n### Increment\n#### Context\n- [ ] Require admin auth\n- [ ] Require db auth\n- [x] Replicate? How? (replicate-increment)\n- [x] Register Oplog? How? (Update)\n\nIncrements an integer key value, if the key does not exist start the value with 0, if the key is integer returns an error\ne.gs: \nIncrements in 1\n```\nincrement visits\n```\n\nIncrements by 10\n\n```\nincrement visits 10\n```\n\n### Acknowledge\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? (replicate-increment)\n- [ ] Register Oplog? How? (Update)\n\nInternally used to acknowledge messages processed by the replicas\n\ne.g\n```\n  ack 1 replica1:181\n```\n\n### ClusterState\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? (replicate-increment)\n- [ ] Register Oplog? How? (Update)\n\nReturns the cluster state, useful for debugging or admin purposes\n\ne.gs\n```\n# request\ncluster-state;\nresponse: \ncluster-state  127.0.0.1:3017:Primary, 127.0.0.1:3018:Secoundary,\n```\n\n### MetricsState\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? (replicate-increment)\n- [ ] Register Oplog? How? (Update)\n\nReturns the oplog and query metrics state, useful for debugging or admin proposes\n\ne.gs\n```\n# request\nmetrics-state;\nresponse: \nmetrics-state pending_ops: 0, op_log_file_size: 0, op_log_count: 0,replication_time_moving_avg: 0.0, get_query_time_moving_avg: 0.0\n```\n\n### Debug\n#### Context\n- [x] Require admin auth\n- [ ] Require db auth\n- [ ] Replicate? How? (replicate-increment)\n- [ ] Register Oplog? How? (Update)\n\nThe debug command holds admin queries for NunDB, such as, checking the messages that are pending replication from a specific node in the cluster.\n\ne.gs\n```\ndebug pending-ops\n# result\npending-ops #list-of-pending-ops#\n\ndebug pending-conflicts\n# result (Must be connected to a database with use *** ***)\npending-conflicts #list-of-pending-conflicts#\n\ndebug list-dbs\n# result (List the name and the strategy of the dbs one on each line)\ndbs-list\nsample : none\nvue : none\ntest : arbiter\nanalitcs-blog : none\n$admin : newer\n\ndebug force-election\n# Forces run an election immediately, which is useful if some node in the cluster is not responsive or to debug latency problems between nodes. Elections in NunDB are predictable and, unless the primary node is slow, the primary should not change even if you force an election.\n```\n\n### Arbiter\n#### Context\n- [ ] Require admin auth\n- [x] Require db auth\n- [ ] Replicate? How? ()\n- [ ] Register Oplog? How? (no)\n\nWith the arbiter command one client register it self as an arbiter for conflicts, this client must be connected to the primary database.\nInternally this command is equivalent to watching the key $conflict.\n\ne.gs\n```\narbiter\n#result\nok|error\n```\n\n### CreateUser\n#### Context\n- [x] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (Set command to security key)\n- [x] Register Oplog? How? (As key value)\n\nCreates a user for the selected user\ne.gs\n```\ncreate-user $user $token\n```\n\n\n### SetPermission\n#### Context\n- [x] Require admin auth\n- [x] Require db auth\n- [x] Replicate? How? (Set command to security key)\n- [x] Register Oplog? How? (As key value)\n\nCreates a user for the selected user\ne.gs\n```\nset-permissions $user_name $permissions $key_pattern|$permissions $key_pattern;\n\nset-permissions foo rw test-*|rwi count;\n# Grant permission to user called foo to read and write keys starting with test-, and read write and increment count key.\n# Permissions latter\n# r = Read,\n# w = Write,\n# i = Increment,\n# x = Remove,\n\n```\n\n\n## Special keys\n\nAll special keys will have a `$` symbol in the first letter of the name.\n\n### $connections\n\nCount the number of connections to a database.\n\n```\n$connections\n```\n\n### $conflicts\n* Key used to register client as arbiter for conflict resolution\n\n## Secure keys\n* All keys prefixed with `$$` will be considered secure by NunDB and will only allow database admin authentication to `set`, `get`, or `remove` them. These are useful if admins want to store information that should not be leaked to any client. \n* The key `$$token` cannot be removed even with admin credentials.\n\n\n\n## Configurations\n\n1. **NUN_USER**\n   - **Default Value:** `nun`\n   - **Description:** Username for NunDB, which can be overridden by command line.\n   - **Environment Variable:** `NUN_USER`\n* This is the cluster admin user to be used in NunDB in real situation you shall not use the default value. It can be overridden by command line argument `--user`.\n\n2. **NUN_PWD**\n   - **Default Value:** `nun-pwd`\n   - **Description:** Password for NunDB, which can be overridden by command line.\n   - **Environment Variable:** `NUN_PWD`\n* This is the cluster admin password to be used in NunDB in real situation you shall not use the default value. It can be overridden by command line argument `-p`.\n\n3. **NUN_DBS_DIR**\n   - **Default Value:** `dbs`\n   - **Description:** Directory where NunDB databases are stored.\n   - **Environment Variable:** `NUN_DBS_DIR`\n\n4. **NUN_WS_ADDR**\n   - **Default Value:** `0.0.0.0:3012`\n   - **Description:** Address for NunDB WebSocket service.\n   - **Environment Variable:** `NUN_WS_ADDR`\n\n5. **NUN_HTTP_ADDR**\n   - **Default Value:** `0.0.0.0:3013`\n   - **Description:** Address for NunDB HTTP service.\n   - **Environment Variable:** `NUN_HTTP_ADDR`\n\n6. **NUN_TCP_ADDR**\n   - **Default Value:** `0.0.0.0:3014`\n   - **Description:** Address for NunDB TCP service.\n   - **Environment Variable:** `NUN_TCP_ADDR`\n\n7. **NUN_REPLICATE_ADDR**\n   - **Default Value:** `\"\"`\n   - **Description:** Address for NunDB replication service.\n   - **Environment Variable:** `NUN_REPLICATE_ADDR`\n\n8. **NUN_LOG_LEVEL**\n   - **Default Value:** `Info`\n   - **Description:** Logging level for NunDB (Options: Off, Error, Warn, Info, Debug, Trace).\n   - **Environment Variable:** `NUN_LOG_LEVEL`\n\n9. **NUN_ELECTION_TIMEOUT**\n   - **Default Value:** `1000` (milliseconds)\n   - **Description:** Timeout for leader election in NunDB clusters.\n   - **Environment Variable:** `NUN_ELECTION_TIMEOUT`\n* Configurations are available to define the timeout period for elections to wait until they are acknowledged from all nodes. It is important to note that you should rarely change this variable since doing so could make elections slower. The value of this variable should be at least twice the latency value to ensure that the election process runs smoothly.\n\n10. **NUN_MAX_OP_LOG_SIZE**\n    - **Default Value:** `1073741824` (1GB)\n    - **Description:** Maximum operations log size before rotation.\n    - **Environment Variable:** `NUN_MAX_OP_LOG_SIZE`\n* Defines the max op log size in bytes, default value is 1G. NunDb keeps up to 10 uplog files each one containing at max 10% of the max size in bytes of the Oplog max size. When there are more than 10 oplog files the oldest one will be removed in the next time the declutter disk runs. Declutter run on a scheduler evert every 300s (5 minutes), that can be changed by the environment var NUN_DECLUTTER_INTERVAL that defines the interval second. That means when cleaning up space 10% of the operations are cleaned at a time.\n\n11. **NUN_DECLUTTER_INTERVAL**\n    - **Default Value:** `300` (seconds)\n    - **Description:** Interval for decluttering, at decluttering time NunDB will snapshot pending databases to disk (or s3) and remove old not needed times.\n    - **Environment Variable:** `NUN_DECLUTTER_INTERVAL`\n* Defines the interval in seconds that the declutter process will run, default value is 300s (5 minutes).\n\n12. **NUN_STORAGE_STRATEGY**\n    - **Default Value:** `disk`\n    - **Description:** Storage strategy for NunDB (Options: disk, s3).\n    - **Environment Variable:** `NUN_STORAGE_STRATEGY`\n\n* Defines how NunDB will store the data disk or s3, in case `s3` it will require may others configurations like `NUN_S3_API_URL`, `NUN_S3_BUCKET`, `NUN_S3_PREFIX`, `NUN_S3_KEY_ID`, `NUN_S3_SECRET_KEY` and it still requires configurations to a path to store data. E.g NunDB still stores oplog locally on the each node, when using s3 this file does not need to be durable if the process restarts.\n\n13. **NUN_S3_KEY_ID**\n    - **Default Value:** `nun-db`\n    - **Description:** AWS S3 Key ID for S3 storage.\n    - **Environment Variable:** `NUN_S3_KEY_ID`\n\n14. **NUN_S3_SECRET_KEY**\n    - **Default Value:** `nun-db-something-more`\n    - **Description:** AWS S3 Secret Key for S3 storage.\n    - **Environment Variable:** `NUN_S3_SECRET_KEY`\n\n15. **NUN_S3_BUCKET**\n    - **Default Value:** `nun-db`\n    - **Description:** S3 bucket name for NunDB storage.\n    - **Environment Variable:** `NUN_S3_BUCKET`\n\n16. **NUN_S3_PREFIX**\n    - **Default Value:** `nun-db-base`\n    - **Description:** Prefix for NunDB data in the S3 bucket.\n    - **Environment Variable:** `NUN_S3_PREFIX`\n\n17. **NUN_S3_API_URL**\n    - **Default Value:** `http://127.0.0.1:9000`\n    - **Description:** API URL for accessing S3-compatible services.\n    - **Environment Variable:** `NUN_S3_API_URL`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmateusfreira%2Fnun-db","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmateusfreira%2Fnun-db","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmateusfreira%2Fnun-db/lists"}