{"id":50837609,"url":"https://github.com/openlake-project/openlake","last_synced_at":"2026-06-14T05:01:32.866Z","repository":{"id":358065128,"uuid":"1222120468","full_name":"openlake-project/openlake","owner":"openlake-project","description":"High performance storage system for LLM Inference and GPU Training. Feed your GPUs at blazing fast speeds","archived":false,"fork":false,"pushed_at":"2026-06-13T13:29:15.000Z","size":1427,"stargazers_count":1345,"open_issues_count":64,"forks_count":201,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-06-13T15:21:39.962Z","etag":null,"topics":["blackwell","gpt","gpu","high-performance","llm","llm-training","model-serving","rdma","rust","storage","throughput"],"latest_commit_sha":null,"homepage":"https://theopenlake.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openlake-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-27T04:07:33.000Z","updated_at":"2026-06-13T15:15:48.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/openlake-project/openlake","commit_stats":null,"previous_names":["openlake-project/openlake"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/openlake-project/openlake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlake-project%2Fopenlake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlake-project%2Fopenlake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlake-project%2Fopenlake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlake-project%2Fopenlake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openlake-project","download_url":"https://codeload.github.com/openlake-project/openlake/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openlake-project%2Fopenlake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34309655,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-14T02:00:07.365Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blackwell","gpt","gpu","high-performance","llm","llm-training","model-serving","rdma","rust","storage","throughput"],"created_at":"2026-06-14T05:00:59.079Z","updated_at":"2026-06-14T05:01:32.860Z","avatar_url":"https://github.com/openlake-project.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\n\u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/openlake-project/openlake/refs/heads/main/assets/openlake-wordmark-dark-8192.png\"\u003e\n  \u003cimg alt=\"OpenLake\" src=\"https://raw.githubusercontent.com/openlake-project/openlake/refs/heads/main/assets/openlake-wordmark-light-8192.png\" width=55%\u003e\n\u003c/picture\u003e\n\n\n\u003ch3 align=\"center\"\u003e\nThe shortest path from NVMe to GPU memory.\n\u003c/h3\u003e\n\nDistributed object storage for GPU workloads. Built on Rust on `io_uring`, OpenLake is a state of the art storage engine delivering 6x throughput and million+ iops within 1ms.\n\n\n[Discord](https://discord.gg/TNXqVSnP6x)\u0026nbsp;·\u0026nbsp;[Website](https://theopenlake.com)\u0026nbsp;·\u0026nbsp;[Comparison](https://theopenlake.com/compare.html)\u0026nbsp;·\u0026nbsp;[Architecture](https://github.com/openlake-project/openlake/tree/main/docs)\u0026nbsp;·\u0026nbsp;[Quickstart](#quickstart)\n\n\n\n[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)\n[![Rust](https://img.shields.io/badge/rust-1.91%2B-orange.svg)](rust-toolchain.toml)\n[![Discord](https://img.shields.io/badge/community-discord-5865F2?logo=discord\u0026logoColor=white)](https://discord.gg/TNXqVSnP6x)\n[![Web](https://img.shields.io/badge/web-theopenlake.com-1d4ed8.svg)](https://theopenlake.com)\n\n\n\n\u003c/div\u003e\n\n\n---\n\n## What is OpenLake?\n\nOpenLake is an object store for AI infrastructure. Training and inference clusters spend a large fraction of their wall clock time moving bytes from storage into GPU memory, most object stores put the host CPU, the page cache, and several userspace copies directly in that path. OpenLake is a high throughput, low latency storage engine for high throughput GPU workloads.\n\n- **`io_uring`, thread per core.** Built on the [`compio`](https://github.com/compio-rs/compio) completion based runtime. One runtime per core, pinned, no work stealing. The HTTP frontend and the storage engine run on the *same* thread, so a request never crosses a core boundary on the hot path.\n - **No kernel involvement.** GPUDirect Storage and RDMA, data moves from peer NIC into GPU VRAM zerocopy, eliminating host memory and the page cache. see [Architecture](https://github.com/openlake-project/openlake#quickstart).\n - **Erasure coded.** SIMD Reed Solomon across striped EC. Reduced storage cost for replication, high throughput without the CPU cost of conventional replication.\n - **PacedRDMA.** Novel congestion control algorithm for high throughput RDMA. Credit based memory management to absorb request bursts, minimizing tail latencies. (Supporting S3 over RDMA)\n \u003cbr\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/get_readme_p50_512.png\" width=\"1400\"\u003e\n  \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\u003csub\u003eOpenLake sustains 225 MiB/s GET at sub 10 ms p50, 3x MinIO and 9x RustFS at c=512.\u003c/sub\u003e\u003c/p\u003e\n\n  ## Quickstart\n\n  ### Prerequisites\n\n  Stable Rust 1.91 or newer (pinned via `rust-toolchain.toml`). Linux gives you the `io_uring` driver; macOS builds and runs against `kqueue` for development.\n\n  ```sh\n  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n  rustup default stable\n  ```\n\n  ### Build\n\n  Clone the repo and build the workspace in release mode.\n\n  ```sh\n  git clone \u003crepo-url\u003e openlake \u0026\u0026 cd openlake\n  cargo build --release --workspace\n  ```\n\n  ### Benchmark\n\n  The `openlake` CLI drives a `LocalFsBackend` directly for diagnostics and microbenchmarks. Not an S3 client, but the quickest way to confirm the build works and see local throughput.\n\n  ```sh\n  ./target/release/openlake bench --n 100000 --size 4096 --concurrency 64\n  ```\n\n  ### Start Cluster\n\n  Write one TOML file per node. The full schema lives at the top of [`crates/openlake_server/src/config.rs`](crates/openlake_server/src/config.rs).\n\n  Start `openlaked` on each host with its own config, then talk to the cluster with any S3 client.\n\n  ```sh\n  ./target/release/openlaked --config node0.toml\n\n  aws --endpoint-url http://10.0.0.10:9000 s3 mb s3://demo\n  aws --endpoint-url http://10.0.0.10:9000 s3 cp ./checkpoint.safetensors s3://demo/\n  aws --endpoint-url http://10.0.0.10:9000 s3 ls s3://demo/\n  ```\n## Contributing\n\nWe welcome and value any contributions and collaborations.\nPlease check out [Contributing to OpenLake](https://github.com/openlake-project/openlake/blob/main/CONTRIBUTING.md) for how to get involved.\n\n## Contact Us\n\n  - For technical support, please reach out on [discord](https://discord.gg/TNXqVSnP6x).\n  - For technical issues, bugs, and feature requests, please open an issue on [GitHub](https://github.com/openlake-project/openlake/issues).\n  - For everything else, visit the [website](https://theopenlake.com) or reach out to the maintainers on discord.\n\n## License\n\n[Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenlake-project%2Fopenlake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenlake-project%2Fopenlake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenlake-project%2Fopenlake/lists"}