https://github.com/aleph-alpha/blended-dataset-loop
Compute a balanced order of dataset indices for training
https://github.com/aleph-alpha/blended-dataset-loop
Last synced: over 1 year ago
JSON representation
Compute a balanced order of dataset indices for training
- Host: GitHub
- URL: https://github.com/aleph-alpha/blended-dataset-loop
- Owner: Aleph-Alpha
- License: other
- Created: 2024-03-15T08:53:39.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-20T12:47:14.000Z (almost 2 years ago)
- Last Synced: 2024-08-21T14:48:56.413Z (almost 2 years ago)
- Language: Rust
- Size: 29.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Blended Dataset Loop
This repository contains a simple loop to compute a balanced ordering of dataset indices to train on.
The resulting ordering ensures the data distribution is similar among batches.
The loop is implemented in Rust for performance reasons and can be consumed as part of a Python package.
The package uses `cffi` (rather than e.g. `PyO3`) in order to be compatible with different Python versions.
## Requirements
* Conda (for Python)
* Cargo with nightly Rust
## Setup
Create a conda environment as follows:
```sh
conda create -n blended_dataset_loop python=3.9 -y
conda activate blended_dataset_loop
```
Install Rust nightly
```sh
rustup override set nightly-2024-02-03
```
Install the Python dev-dependencies:
```sh
pip install 'maturin[patchelf]'
pip install '.[dev]'
```
## Develop
After changing the Rust code, run:
```sh
maturin develop
```
or, for release mode
```sh
maturin develop --release
```