{"id":34637917,"url":"https://github.com/lovelindhoni/lally","last_synced_at":"2025-12-24T17:12:18.134Z","repository":{"id":273369494,"uuid":"918938382","full_name":"lovelindhoni/lally","owner":"lovelindhoni","description":"Lally is a distributed in-memory key-value database, written in Rust and Tokio","archived":false,"fork":false,"pushed_at":"2025-11-11T18:51:36.000Z","size":153,"stargazers_count":18,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-11T20:35:03.895Z","etag":null,"topics":["distributed-systems","key-value-database","rust","tokio"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lovelindhoni.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-01-19T09:26:07.000Z","updated_at":"2025-11-11T18:51:40.000Z","dependencies_parsed_at":"2025-01-20T14:36:21.157Z","dependency_job_id":"6e754509-b636-4937-b9d9-adc9af1db1d2","html_url":"https://github.com/lovelindhoni/lally","commit_stats":null,"previous_names":["lovelindhoni/lally"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lovelindhoni/lally","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovelindhoni%2Flally","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovelindhoni%2Flally/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovelindhoni%2Flally/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovelindhoni%2Flally/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lovelindhoni","download_url":"https://codeload.github.com/lovelindhoni/lally/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovelindhoni%2Flally/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28005413,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","key-value-database","rust","tokio"],"created_at":"2025-12-24T17:09:16.156Z","updated_at":"2025-12-24T17:12:18.125Z","avatar_url":"https://github.com/lovelindhoni.png","language":"Rust","readme":"# Lally\n\n[![Lint](https://github.com/lovelindhoni/lally/actions/workflows/lint.yml/badge.svg)](https://github.com/lovelindhoni/lally/actions/workflows/lint.yml)\n\nLally is a distributed in-memory key-value database, written in Rust and Tokio, loosely inspired by Riak and Project Voldermort.\nIt prioritizes high availability, adhering to the AP (Availability and Partition Tolerance) aspects of the CAP theorem.\nLally is being designed for high concurrency, extendability via hooks, and features persistent append-only file logging for reliable crash recovery.\n\n\u003e [!NOTE]\n\u003e Lally is currently in the early stages of development and is not yet ready for serious production use.\n\n## Demo\n\nA Lally cluster consisting of three globally distributed EC2 instances.\n\nhttps://github.com/user-attachments/assets/6ec0d3df-a9d7-4a50-846b-08045bf3ebb5\n\n## Features\n\n- **High Performance**: A highly concurrent, thread-safe, in-memory key-value database optimized to utilize all available CPU cores responsibly.\n- **Extensible Design**: Hooks-based architecture allows custom behaviors to be invoked during any key-value operation.\n- **Crash Recovery**: Supports append-only file (AOF) logging for robust crash recovery, easy backup, replay, and data restoration.\n- **Cluster Joining**: Enables seamless IP-based cluster joining through a seed node for distributed operation.\n- **Data Replication**: Achieves data replication across cluster nodes using lightweight and efficient Protocol Buffers through gossipping\n- **Connection Pooling**: Reduces overhead by pooling gRPC connections, avoiding repeated connection establishment for inter-node communication.\n- **Quorum Flexibility**: Configurable read and write quorum settings to match the size and needs of the cluster.\n- **Flexible Configuration**: YAML-based configuration that can be overridden using command-line arguments for customization.\n- **Graceful Shutdown**: Ensures proper cluster exit and resource cleanup during shutdown, maintaining cluster stability.\n- **Comprehensive Logging**: Includes extensive logging capabilities for debugging and tracing operations effectively.\n- **User-friendly HTTP API**: Offers an intuitive and straightforward HTTP API for managing the key-value store.\n- **Read Repair Mechanism**: Automatically resolves stale or outdated data during read operations to maintain consistency.\n- **Safe and Reliable (👀)**: Free from unsafe code blocks\n\n## Configuration\n\nYou can configure Lally by passing the following CLI arguments:\n\n- `--config`: Path to the configuration file (e.g., lally.yml).\n- `--fresh`: Wipes previous WAL (Write-Ahead Log) data and starts fresh. (This will clear the existing AOF file, skipping replay of previous operations)\n- `--replay-log`: Path to a custom AOF file for replay. The contents of this file will replace and replay the default AOF file used by Lally.\n- `--seed-node`: IPv4 address with the port of the seed node. Required for joining a cluster via the seed node.\n- `--http-port`: Custom port for the HTTP server (default: 3000).\n- `--grpc-port`: Custom port for the gRPC server (default: 50071).\n- `--read-quorum`: Specifies the number of nodes required for a successful read operation (default: 1).\n- `--write-quorum`: Specifies the number of nodes required for a successful write operation (default: 1).\n- `--aof_flush_interval`: Interval (in milliseconds) at which logs are flushed to disk (default: 100).\n- `--help`: Displays detailed usage information.\n\nLally also supports configuration through a YAML file for greater flexibility and ease of use.\n\nBy default, Lally looks for a lally.yml file in the current directory where the binary is executed. Alternatively, you can specify a custom configuration file using the `--config` CLI argument, like this:\n\n```bash\nlally --config /path/to/lally.yml\n```\n\n### Example lally.yml with Default Values\n\n```yaml\nfresh: false # Start fresh, wiping the previous AOF log (default: false)\nreplay_log: None # Path to a custom AOF log file for replay\nseed_node: None # IPv4 address and port of the seed node (if joining a cluster)\ngrpc_port: 50071 # Port for the gRPC server\nhttp_port: 3000 # Port for the HTTP server\nread_quorum: 1 # Number of nodes required for a successful read operation\nwrite_quorum: 1 # Number of nodes required for a successful write operation\naof_flush_interval: 100 # Flush interval for logs to disk, in milliseconds.\n```\n\n### Priority of Configuration\n\nCommand-line arguments (`--config`, `--http-port`, etc.) always override the values specified in the YAML file.\nThis allows for quick runtime adjustments without the need to modify the configuration file.\nFor example, even if http_port is set to 3000 in the YAML file, specifying `--http-port 8080` via CLI will use 8080 for that session.\n\n## HTTP Endpoints\n\nLally provides a simple HTTP API for interacting with the distributed key-value store.\n\n### POST /get\n\nFetches the latest value associated with the given key. If the value is stale, it triggers a read repair to ensure eventual consistency.\n\n#### Request\n\n```jsonc\n{\n  \"key\": \"example_key\",\n}\n```\n\n#### Expected Response\n\n```jsonc\n{\n  \"status\": \"success | partial | error\",\n  \"key\": \"example_key\",\n  \"value\": \"example_value | null\",\n  \"timestamp\": \"RFC3339 timestamp | null\",\n  \"quorum\": {\n    \"required\": 2, // Number of nodes needed for quorum\n    \"achieved\": 2, // Number of nodes that responded\n  },\n  \"message\": \"Human-readable explanation\",\n}\n```\n\n### POST /add\n\nAdds a key-value pair to the cluster. If the key already exists, its value will be updated.\n\n#### Request\n\n```jsonc\n{\n  \"key\": \"example_key\",\n  \"value\": \"example_value\",\n}\n```\n\n#### Expected Response\n\n```jsonc\n{\n  \"status\": \"success | partial\",\n  \"key\": \"example_key\",\n  \"value\": \"example_value\",\n  \"timestamp\": \"RFC3339 timestamp\",\n  \"quorum\": {\n    \"required\": 2, // Number of nodes needed for quorum\n    \"achieved\": 2, // Number of nodes that responded\n  },\n  \"message\": \"Operation completed successfully.\",\n}\n```\n\n### DELETE /remove\n\nFetches the latest value associated with the given key. If the value is stale, it triggers a read repair to ensure consistency.\n\n#### Request\n\n```jsonc\n{\n  \"key\": \"example_key\",\n}\n```\n\n#### Expected Response\n\n```jsonc\n{\n  \"status\": \"success | partial | error\",\n  \"key\": \"example_key\",\n  \"value\": \"example_value | null\", // Removed value (if applicable)\n  \"timestamp\": \"RFC3339 timestamp | null\", // Timestamp of the removal operation\n  \"quorum\": {\n    \"required\": 2, // Number of nodes needed for quorum\n    \"achieved\": 1, // Number of nodes that responded\n  },\n  \"message\": \"Key successfully removed.\", // Human-readable explanation\n}\n```\n\n### GET /nodes\n\nRetrieves a list of all network addresses of nodes in the cluster, excluding the node handling the request.\n\n#### Expected Response\n\n```jsonc\n{\n  \"status\": \"success | error\",\n  \"nodes\": [\"192.168.1.1:50071\", \"192.168.1.2:50071\"],\n}\n```\n\n### Key Notes\n\n**Quorum State**: The status field in responses indicates the quorum state:\n\n- **success**: The quorum is fully satisfied.\n- **partial**: Quorum partially satisfied; some nodes failed to respond.\n- **error**: Error :|\n\n**Timestamp Format**: All timestamps are in RFC3339 format for standardization.\n\n**Quorum Details**: The quorum field provides insight into the required and achieved votes during the operation, helping debug cluster consistency\n\n## Development\n\nTo get started with developing or running Lally, ensure you have the required dependencies and follow the setup instructions below.\n\n### Dependencies\n\nA suitable Rust toolchain.\nProtocol Buffers Compiler (protoc) version 3 or higher\nProtocol Buffers development headers.\n\n### Setup instructions\n\n1. Clone the repository\n\n```bash\ngit clone https://github.com/lovelindhoni/lally.git\ncd lally\n```\n\n2. Configure network settings:\n\nAllow inbound and outbound TCP traffic on the ports used by Lally.\n\nDefault HTTP port: 3000\nDefault gRPC port: 50071\nEnsure your firewall settings permit these ports or any custom ports you’ve configured.\n\n3. Debug mode\n\n```bash\ncargo run\n```\n\n## Production Build\n\nBuild the release binary:\n\n```bash\ncargo build --release\n```\n\nLocate the compiled binary in target/release/lally\n\n### Example exectution\n\n```bash\n./lally --config lally.yml --http_port 4001\n```\n\nThis binary is optimized for performance and ready for deployment.\n\n### Docker Image\n\nBuild the docker image:\n\n```bash\ncd lally\ndocker build -t lally\n```\n\nRun the container\n\n```bash\ndocker run --rm -v \"/path/to/lally-folder\":/root/.local/share/lally -v \"/path/to/config-file\":/app/lally.yml -v \"/path/to/replayfile\":/app/your-aof.txt -p 3000:3089 -p 50071:50071 lally:0.1.0 --replay-log /app/your-aof.txt\n```\n\n## Roadmap (tentative)\n\n- Kubernetes compatibility for deployment and scalability\n- Testing suite and performance benchmarking 👀\n- Periodic health checks to monitor the status of transparent channels between nodes in the pool\n- Background reconciliation (anti-entropy) process for automatic key value repairs, to fulfill eventual consistency\n- CI pipeline for building, testing, and releasing Lally across all platforms\n- Enhanced, structured trace logging for better observability\n- Web-based UI for intuitive interaction with Lally\n- Support for additional data structures—currently, only string-to-string pairs are supported, but we aim to streamline support for other data structures via more efficient deserialization\n\n## Contribution\n\nPlease submit pull requests if you had to, after your code is linted and formatted using Clippy and Rustfmt.\nYou can find the required checks in the ci.yml file, which lists all the validation processes\n\n## LICENSE\n\n**MIT**\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovelindhoni%2Flally","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flovelindhoni%2Flally","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovelindhoni%2Flally/lists"}