{"id":14154874,"url":"https://github.com/Oxen-AI/Oxen","last_synced_at":"2025-08-06T00:33:24.528Z","repository":{"id":65751923,"uuid":"483904733","full_name":"Oxen-AI/Oxen","owner":"Oxen-AI","description":"Oxen.ai's core rust library, server, and CLI","archived":false,"fork":false,"pushed_at":"2024-08-16T20:16:13.000Z","size":116096,"stargazers_count":172,"open_issues_count":11,"forks_count":11,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-08-16T20:18:52.841Z","etag":null,"topics":["artificial-intelligence","data-science","database","machine-learning","version-control"],"latest_commit_sha":null,"homepage":"https://oxen.ai","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Oxen-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-21T04:26:22.000Z","updated_at":"2024-08-16T13:38:08.000Z","dependencies_parsed_at":"2024-04-08T05:32:34.843Z","dependency_job_id":"5e5f5ccd-e24c-4fe8-924f-b2acd469686f","html_url":"https://github.com/Oxen-AI/Oxen","commit_stats":null,"previous_names":[],"tags_count":211,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oxen-AI%2FOxen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oxen-AI%2FOxen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oxen-AI%2FOxen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Oxen-AI%2FOxen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Oxen-AI","download_url":"https://codeload.github.com/Oxen-AI/Oxen/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":215735785,"owners_count":15923388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-intelligence","data-science","database","machine-learning","version-control"],"created_at":"2024-08-17T08:01:30.633Z","updated_at":"2024-12-09T02:30:55.035Z","avatar_url":"https://github.com/Oxen-AI.png","language":"Rust","funding_links":[],"categories":["Rust","artificial-intelligence","1. Core Frameworks \u0026 Libraries","Data \u0026 Index Versioning","Data Management"],"sub_categories":["Frameworks"],"readme":"# 🐂 Oxen\n\nCreate a world where everyone can contribute to an Artificial General Intelligence, starting with the data.\n\n# 🌾 What is Oxen?\n\nOxen at it's core is a data version control library, written in Rust. It's goals are to be fast, reliable, and easy to use. It's designed to be used in a variety of ways, from a simple command line tool, to a remote server to sync to, to integrations into other ecosystems such as [python](https://github.com/Oxen-AI/oxen-release).\n\n# 📚 Documentation\n\nThe documentation for the Oxen.ai tool chain can be found [here](https://docs.oxen.ai).\n\n# ✅ TODO\n\n- [ ] Hugging face compatible APIs\n  - [ ] Upload model to hub\n  - [ ] Download model with `transformers` library\n  - [ ] Upload dataset to hub\n  - [ ] Download dataset with `datasets` library\n- [ ] Configurable storage backends\n  - [x] Local filesystem\n  - [ ] S3\n  - [ ] GCS\n  - [ ] Azure\n  - [ ] Backblaze\n- [ ] Block level deduplication\n\n# 🔨 Build \u0026 Run\n\n## Install Dependencies\n\nOxen is purely written in Rust 🦀. You should install the Rust toolchain with rustup: https://www.rust-lang.org/tools/install.\n\n```\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh\n```\n\nIf you are a developer and want to learn more about adding code or the overall architecture [start here](docs/dev/AddLibraryCode.md). Otherwise, a quick start to make sure everything is working follows.\n\n## Build\n\n```\ncargo build\n```\n\nIf on intel mac, you may need to build with the following\n\n```\n$ rustup target install x86_64-apple-darwin\n$ cargo build --target x86_64-apple-darwin\n```\n\nIf on Windows, you may need to add the following directories to the 'INCLUDE' environment variable\n\n```\n\"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\include\"\n\n\"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.27023\\include\"\n\n\"C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\Llvm\\lib\\clang\\12.0.0\\include\"\n```\nThese are example paths and will vary between machines. If you install 'C++ Clang tools for Windows' through [Microsoft Visual Studio Build Tools](https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019), the directories can be located from the Visual Studio installation under 'BuildTools\\VC\\Tools'\n\n## Speed up the build process\n\nYou can use the [mold](https://github.com/rui314/mold) linker to speed up builds (The MIT-licensed macOS version is [sold](https://github.com/bluewhalesystems/sold)).\n\nUse the following instructions to\ninstall sold and configure cargo to use it for building Oxen:\n\n```\ngit clone --depth=1 --single-branch https://github.com/bluewhalesystems/sold.git\n\nmkdir sold/build\ncd sold/build\ncmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=c++ ..\ncmake --build . -j $(nproc)\nsudo cmake --install .\n```\n\nThen create `.cargo/config.toml` in your Oxen repo root with the following\ncontent:\n\n```\n[target.x86_64-unknown-linux-gnu]\nrustflags = [\"-C\", \"link-arg=-fuse-ld=/usr/local/bin/ld64.mold\"]\n\n[target.x86_64-apple-darwin]\nrustflags = [\"-C\", \"link-arg=-fuse-ld=/usr/local/bin/ld64.mold\"]\n\n```\n\n**For macOS with Apple Silicon**, you can use the [lld](https://lld.llvm.org/) linker.\n\n```\nbrew install llvm\n```\n\nThen create `.cargo/config.toml` in your Oxen repo root with the following:\n\n```\n[target.aarch64-apple-darwin]\nrustflags = [ \"-C\", \"link-arg=-fuse-ld=/opt/homebrew/opt/llvm/bin/ld64.lld\", ]\n\n```\n\n# Run\n\n## CLI\n\nTo run Oxen from the command line, add the `Oxen/target/debug` directory to the 'PATH' environment variable\n\n```\nexport PATH=\"$PATH:/path/to/Oxen/target/debug\"\n```\n\nOn Windows, you can use\n\n```\n$env:PATH += \";/path/to/Oxen/target/debug\"\n```\n\nInitialize a new repository or clone an existing one\n\n```\noxen init\noxen clone https://hub.oxen.ai/namespace/repository\n```\n\nThis will create the `.oxen` dir in your current directory and allow you to run Oxen CLI commands\n\n```\noxen status\noxen add images/\noxen commit -m \"added images\"\noxen push origin main\n```\n\n\n## Oxen Server\n\nTo run a local Oxen Server, generate a config file and token to authenticate the user\n\n```\n./target/debug/oxen-server add-user --email ox@oxen.ai --name Ox --output user_config.toml\n```\n\nCopy the config to the default locations\n\n```\nmkdir ~/.oxen\n```\n\n```\nmv user_config.toml ~/.oxen/user_config.toml\n```\n\n```\ncp ~/.oxen/user_config.toml data/test/config/user_config.toml\n```\n\nSet where you want the data to be synced to. The default sync directory is `./data/` to change it set the SYNC_DIR environment variable to a path.\n\n```\nexport SYNC_DIR=/path/to/sync/dir\n```\n\nYou can also create a .env.local file in the /src/server directory which can contain the SYNC_DIR variable to avoid setting it every time you run the server.\n\nRun the server\n\n```\n./target/debug/oxen-server start\n```\n\nTo run the server with live reload, first install cargo-watch\n\n```\ncargo install cargo-watch\n```\n\nOn Windows, you may need to use `cargo-watch --locked`\n\n```\ncargo install cargo-watch --locked\n```\n\nThen run the server like this\n\n```\ncargo watch -- cargo run --bin oxen-server start\n```\n\n\n## Nix Flake\n\nIf you have [Nix installed](https://github.com/DeterminateSystems/nix-installer)\nyou can use the flake to build and run the server. This will automatically\ninstall and configure the required build toolchain dependencies for Linux \u0026 macOS.\n\n```\nnix build .#oxen-server\nnix build .#oxen-cli\nnix build .#liboxen\n```\n\n```\nnix run .#oxen-server -- start\nnix run .#oxen-cli -- init\n```\n\nTo develop with the standard rust toolchain in a Nix dev shell:\n\n```\nnix develop -c $SHELL\ncargo build\ncargo run --bin oxen-server start\ncargo run --bin oxen start\n```\n\nThe flake also provides derviations to build OCI (Docker) images with the minimal\nset of dependencies required to build and run `oxen` \u0026 `oxen-server`.\n\n```\nnix build .#oci-oxen-server\nnix build .#oci-oxen-cli\n```\n\nThis will export the OCI image and can be loaded with:\n\n```\ndocker load -i result\n```\n\n# Unit \u0026 Integration Tests\n\nMake sure your user is configured and server is running on the default port and host, by following these setup steps:\n\n```bash\n# Configure a user\nmkdir ./data/test/runs\n./target/debug/oxen-server add-user --email ox@oxen.ai --name Ox --output user_config.toml\ncp user_config.toml data/test/config/user_config.toml\n# Start the oxen-server\n./target/debug/oxen-server start\n```\n\n*Note:* tests open up a lot of file handles, so limit num test threads if running everything.\n\nYou an also increase the number of open files your system allows ulimit before running tests:\n\n```\nulimit -n 10240\n```\n\n```\ncargo test -- --test-threads=$(nproc)\n```\n\nIt can be faster (in terms of compilation and runtime) to run a specific test. To run a specific library test:\n\n```\ncargo test --lib test_get_metadata_text_readme\n```\n\nTo run a specific integration test\n\n```\ncargo test --test test_rm test_rm_directory_restore_directory\n```\n\nTo run with all debug output and run a specific test\n\n```\nenv RUST_LOG=warn,liboxen=debug,integration_test=debug cargo test -- --nocapture test_command_push_clone_pull_push\n```\n\nTo set a different test host you can set the `OXEN_TEST_HOST` environment variable\n\n```\nenv OXEN_TEST_HOST=0.0.0.0:4000 cargo test\n```\n\n# Oxen Server\n\n## Structure\n\nRemote repositories have the same internal structure as local ones, with the caviate that all the data is in the .oxen dir and not duplicated into a \"local workspace\".\n\n# APIs\n\nServer defaults to localhost 3000\n\n```\nset SERVER 0.0.0.0:3000\n```\n\nYou can grab your auth token from the config file above (~/.oxen/user_config.toml)\n\n```\nset TOKEN \u003cYOUR_TOKEN\u003e\n```\n\n## List Repositories\n\n```\ncurl -H \"Authorization: Bearer $TOKEN\" \"http://$SERVER/api/repos\"\n```\n\n## Create Repository\n\n```\ncurl -H \"Authorization: Bearer $TOKEN\" -X POST -d '{\"name\": \"MyRepo\"}' \"http://$SERVER/api/repos\"\n```\n\n# Docker\n\nCreate the docker image\n\n```\ndocker build -t oxen/server:0.6.0 .\n```\n\nRun a container on port 3000 with a local filesystem mounted from /var/oxen/data on the host to /var/oxen/data in the container.\n\n```\ndocker run -d -v /var/oxen/data:/var/oxen/data -p 3000:3001 --name oxen oxen/server:0.6.0\n```\n\nOr use docker compose\n\n```\ndocker-compose up -d reverse-proxy\n```\n\n```\ndocker-compose up -d --scale oxen=4 --no-recreate\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOxen-AI%2FOxen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOxen-AI%2FOxen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOxen-AI%2FOxen/lists"}