{"id":26103668,"url":"https://github.com/aleph-alpha/blended-dataset-loop","last_synced_at":"2025-03-09T20:41:59.887Z","repository":{"id":253950191,"uuid":"772468588","full_name":"Aleph-Alpha/blended-dataset-loop","owner":"Aleph-Alpha","description":"Compute a balanced order of dataset indices for training","archived":false,"fork":false,"pushed_at":"2024-08-20T12:47:14.000Z","size":30,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-08-21T14:48:56.413Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Aleph-Alpha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-15T08:53:39.000Z","updated_at":"2024-08-20T12:47:17.000Z","dependencies_parsed_at":"2024-08-20T14:49:04.378Z","dependency_job_id":"c59df661-656c-4aa6-a7d0-26be8724f419","html_url":"https://github.com/Aleph-Alpha/blended-dataset-loop","commit_stats":null,"previous_names":["aleph-alpha/blended-dataset-loop"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fblended-dataset-loop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fblended-dataset-loop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fblended-dataset-loop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Aleph-Alpha%2Fblended-dataset-loop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Aleph-Alpha","download_url":"https://codeload.github.com/Aleph-Alpha/blended-dataset-loop/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242750781,"owners_count":20179256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-09T20:41:59.357Z","updated_at":"2025-03-09T20:41:59.880Z","avatar_url":"https://github.com/Aleph-Alpha.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Blended Dataset Loop\n\nThis repository contains a simple loop to compute a balanced ordering of dataset indices to train on.\nThe resulting ordering ensures the data distribution is similar among batches.\nThe loop is implemented in Rust for performance reasons and can be consumed as part of a Python package.\n\nThe package uses `cffi` (rather than e.g. `PyO3`) in order to be compatible with different Python versions.\n\n## Requirements\n\n* Conda (for Python)\n* Cargo with nightly Rust\n\n## Setup\n\nCreate a conda environment as follows:\n\n```sh\nconda create -n blended_dataset_loop python=3.9 -y\nconda activate blended_dataset_loop\n```\n\nInstall Rust nightly\n\n```sh\nrustup override set nightly-2024-02-03\n```\n\nInstall the Python dev-dependencies:\n\n```sh\npip install 'maturin[patchelf]'\npip install '.[dev]'\n```\n\n## Develop\n\nAfter changing the Rust code, run:\n\n```sh\nmaturin develop\n```\n\nor, for release mode\n\n```sh\nmaturin develop --release\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fblended-dataset-loop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faleph-alpha%2Fblended-dataset-loop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faleph-alpha%2Fblended-dataset-loop/lists"}