{"id":51240111,"url":"https://github.com/commaai/gigashuffle","last_synced_at":"2026-06-28T23:32:21.998Z","repository":{"id":361505068,"uuid":"1160505630","full_name":"commaai/gigashuffle","owner":"commaai","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-18T01:39:06.000Z","size":180,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-18T03:22:00.538Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/commaai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-02-18T02:49:29.000Z","updated_at":"2026-06-17T17:05:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/commaai/gigashuffle","commit_stats":null,"previous_names":["commaai/gigashuffle"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/commaai/gigashuffle","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commaai%2Fgigashuffle","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commaai%2Fgigashuffle/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commaai%2Fgigashuffle/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commaai%2Fgigashuffle/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/commaai","download_url":"https://codeload.github.com/commaai/gigashuffle/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/commaai%2Fgigashuffle/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34907985,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-28T02:00:05.809Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-28T23:32:21.877Z","updated_at":"2026-06-28T23:32:21.988Z","avatar_url":"https://github.com/commaai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gigashuffle\n\nShuffle-buffer dataloading for PyTorch training.\n\n## Usage\n\nStart a redis server before constructing a loader:\n\n```bash\nredis-server\n```\n\nUse in your training script:\n\n```python\nimport os\nimport torch\nimport torch.distributed as dist\nfrom torch.utils.data import IterableDataset\nfrom gigashuffle import DataloaderConfig, MultiprocessShuffledDataloader\n\nclass MyDataset(IterableDataset):\n  def __iter__(self):\n    while True:\n      x = torch.randn(64, 3, 224, 224)\n      y = torch.randint(0, 1000, (64,))\n      yield [{'x': x, 'y': y}]\n\nconfig = DataloaderConfig(\n  bs=32,\n  shuffle_size=1000,\n  num_writers=2,\n  num_readers=2,\n  local_rank=int(os.environ.get('LOCAL_RANK', '0')),\n  global_rank=int(os.environ.get('RANK', '0')),\n  local_world_size=int(os.environ.get('LOCAL_WORLD_SIZE', '1')),\n  global_world_size=int(os.environ.get('WORLD_SIZE', '1')),\n  queue_name='my-training-loader', # must be unique across your dataloaders\n  fill_once=False, # see below\n)\n\nloader = MultiprocessShuffledDataloader(MyDataset(), config=config)\ndummy_batch = loader.get_dummy_batch()\n\nfor batch in loader:\n  x = batch[0]['x']\n  y = batch[0]['y']\n  break\n```\n\nDatasets yield a `Buffer`: `list[dict[str, Tensor | ndarray]]`. Every tensor or array in a sample must share the same first dimension; that is the input batch size. `config.bs` is the output batch size.\n\nCall `loader.get_dummy_batch()` when you need a sanity-check batch before the shuffle buffer reaches `min_mixing`. It repeats the initial sample to `config.bs` and does not advance the normal shuffled iterator.\n\nSet `config.fill_once=True`, `config.min_mixing=1`, and `config.num_readers=1` to populate the shuffle buffer once, then yield one ordered pass over it without returning indices to the writers. Reader waits for the full `shuffle_size` before yielding batches in this mode.\n\nBy default (`config.evict_on_read=True`) a buffer index is returned to the writers as soon as it is read, so each sample is consumed once. Set `config.evict_on_read=False` to keep read entries in the buffer for re-sampling and have the client manage eviction with `loader.evict(indices)`. Never evicting will freeze the buffer once it fills, and evicting faster than writers can refill will make reads wait. Every batch carries the buffer index of each row in `batch[0][INDEX_KEY]` (a `(bs,)` int64 tensor). `config.evict_on_read=False` is not supported with `fill_once=True`.\n\n## Notes\n\nEach Dataloader owns one shared CPU shuffle buffer. The owner writer (`proc_idx=0`) allocates the buffer with `Tensor.share_memory_()` using PyTorch's `file_descriptor` CPU sharing strategy, publishes only an `AF_UNIX` attach-socket path in Redis, and sends the shared tensor objects over that socket; this follows the same fd-transfer mechanism PyTorch uses for CPU tensor IPC. The simpler alternatives do not work well here: a `multiprocessing.Queue` object cannot be shared across independent `torchrun` ranks, Redis cannot transmit process-local file descriptors, explicit `/dev/shm` or `torch.from_file` names can leak after killed workers, and `/proc/\u003cpid\u003e/fd/\u003cfd\u003e` attachment is Linux/container-permission fragile and races owner death.\n\nThe output batches are views into reusable shared-memory reader buffers, not copies. Their contents may be overwritten after the iterator advances or the batch is released, so callers that need to keep a batch must clone/copy it before requesting another batch or dropping the iterator.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommaai%2Fgigashuffle","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcommaai%2Fgigashuffle","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcommaai%2Fgigashuffle/lists"}