{"id":21657822,"url":"https://github.com/facebookincubator/nimble","last_synced_at":"2025-07-17T20:32:15.460Z","repository":{"id":236168402,"uuid":"730832824","full_name":"facebookincubator/nimble","owner":"facebookincubator","description":"New file format for storage of large columnar datasets.","archived":false,"fork":false,"pushed_at":"2024-11-22T02:58:27.000Z","size":533,"stargazers_count":452,"open_issues_count":15,"forks_count":31,"subscribers_count":21,"default_branch":"main","last_synced_at":"2024-11-22T03:27:43.687Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookincubator.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-12T19:16:56.000Z","updated_at":"2024-11-22T02:58:31.000Z","dependencies_parsed_at":"2024-09-10T00:32:45.076Z","dependency_job_id":"c71d9bba-cfff-4b57-8fde-c852904633ac","html_url":"https://github.com/facebookincubator/nimble","commit_stats":null,"previous_names":["facebookexternal/nimble","facebookincubator/nimble"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookincubator%2Fnimble","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookincubator%2Fnimble/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookincubator%2Fnimble/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookincubator%2Fnimble/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookincubator","download_url":"https://codeload.github.com/facebookincubator/nimble/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226304082,"owners_count":17603504,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T09:28:10.123Z","updated_at":"2025-07-17T20:32:15.448Z","avatar_url":"https://github.com/facebookincubator.png","language":"C++","funding_links":[],"categories":["C++","HarmonyOS","Related formats"],"sub_categories":["Windows Manager","Tests"],"readme":"# The Nimble File Format\n\nNimble (formerly known as _“Alpha”_) is a new columnar file format for large\ndatasets created by Meta. Nimble is meant to be a replacement for file formats\nsuch as Apache Parquet and ORC.\n\nWatch [this talk](https://www.youtube.com/watch?v=bISBNVtXZ6M) to learn more\nabout Nimble’s internals.\n\nNimble has the following design principles:\n\n- **Wide:** Nimble is better suited for workloads that are wide in nature, such\n  as tables with thousands of columns (or streams) which are commonly found in\n  feature engineering workloads and training tables for machine learning.\n\n- **Extensible:** Since the state-of-the-art in data encoding evolves faster\n  than the file layout itself, Nimble decouples stream encoding from the\n  underlying physical layout. Nimble allows encodings to be extended by library\n  users and recursively applied (cascading).\n\n- **Parallel:** Nimble is meant to fully leverage highly parallel hardware by\n  providing encodings which are SIMD and GPU friendly. Although this is not\n  implemented yet, we intend to expose metadata to allow developers to better\n  plan decoding trees and schedule kernels without requiring the data streams\n  themselves.\n\n- **Unified:** More than a specification, Nimble is a product. We strongly\n  discourage developers to (re-)implement Nimble’s spec to prevent environmental\n  fragmentation issues observed with similar projects in the past. We encourage\n  developers to leverage the single unified Nimble library, and create\n  high-quality bindings to other languages as needed.\n\nNimble has the following features:\n\n- Lighter metadata organization to efficiently support thousands to tens of\n  thousands of columns and streams.\n\n- Use Flatbuffers instead of thrift/protobuf to more efficiently access large\n  metadata sections.\n\n- Use block encoding instead of stream encoding to provide predictable memory\n  usage while decoding/reading.\n\n- Supports many encodings out-of-the-box, and additional encodings can be added\n  as needed.\n\n- Supports cascading (recursive/composite) encoding of streams.\n\n- Supports pluggable encoding selection policies.\n\n- Provide extensibility APIs where encodings and other aspects of the file can\n  be extended.\n\n- Clear separation between logical and physical encoded types.\n\n- And more.\n\nNimble is a work in progress, and many of these features above are still under\ndesign and/or active development. As such, Nimble does not provide stability or\nversioning guarantees (yet). They will be eventually provided with a future\nstable release. Use it at your own risk.\n\n## Build\n\nNimble’s CMake build system is self-sufficient and able to either locate its\nmain dependencies or compile them locally. In order to compile it, one can\nsimply:\n\n```shell\n$ git clone git@github.com:facebookincubator/nimble.git\n$ cd nimble\n$ make\n```\n\nTo override the default behavior and force the build system to, for example,\nbuild a dependency locally (bundle it), one can:\n\n```shell\n$ folly_SOURCE=BUNDLED make\n```\n\nNimble builds have been tested using clang 15 and 16. It should automatically\ncompile the following dependencies: gtest, glog, folly, abseil, and velox. You\nmay need to first install the following system dependencies for these to compile\n(example from Ubuntu 22.04):\n\n```shell\n$ sudo apt install -y \\\n    git \\\n    cmake \\\n    flatbuffers-compiler \\\n    protobuf-compiler \\\n    libflatbuffers-dev \\\n    libgflags-dev \\\n    libunwind-dev \\\n    libgoogle-glog-dev \\\n    libdouble-conversion-dev \\\n    libevent-dev \\\n    liblz4-dev \\\n    liblzo2-dev \\\n    libelf-dev \\\n    libdwarf-dev \\\n    libsnappy-dev \\\n    libssl-dev \\\n    bison \\\n    flex \\\n    libfl-dev \\\n    pkg-config \\\n    clang \\\n    clang-format\n```\n\nAlthough Nimble’s codebase is today closely coupled with velox, we intend to\ndecouple them in the future.\n\n## License\n\nNimble is licensed under the Apache 2.0 License. A copy of the license\n[can be found here.](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookincubator%2Fnimble","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookincubator%2Fnimble","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookincubator%2Fnimble/lists"}