{"id":20040009,"url":"https://github.com/google-parfait/dataset_grouper","last_synced_at":"2025-08-13T23:07:36.334Z","repository":{"id":171095268,"uuid":"645908527","full_name":"google-parfait/dataset_grouper","owner":"google-parfait","description":"Libraries for efficient and scalable group-structured dataset pipelines.","archived":false,"fork":false,"pushed_at":"2025-06-18T21:23:48.000Z","size":62,"stargazers_count":26,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-08T03:29:28.528Z","etag":null,"topics":["apache-beam","datasets","federated-learning","jax","pytorch","tensorflow","tensorflow-datasets"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-parfait.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-05-26T18:20:41.000Z","updated_at":"2025-06-18T21:23:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"55e656c9-41eb-43eb-8b79-707c709e2e80","html_url":"https://github.com/google-parfait/dataset_grouper","commit_stats":{"total_commits":36,"total_committers":5,"mean_commits":7.2,"dds":0.3055555555555556,"last_synced_commit":"b656a7a9a43e7ae5c4aa9a73f7a153890dd5c415"},"previous_names":["google-research/dataset_grouper","google-parfait/dataset_grouper"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/google-parfait/dataset_grouper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-parfait%2Fdataset_grouper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-parfait%2Fdataset_grouper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-parfait%2Fdataset_grouper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-parfait%2Fdataset_grouper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-parfait","download_url":"https://codeload.github.com/google-parfait/dataset_grouper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-parfait%2Fdataset_grouper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":270330595,"owners_count":24565816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-13T02:00:09.904Z","response_time":66,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-beam","datasets","federated-learning","jax","pytorch","tensorflow","tensorflow-datasets"],"created_at":"2024-11-13T10:40:04.125Z","updated_at":"2025-08-13T23:07:36.318Z","avatar_url":"https://github.com/google-parfait.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dataset Grouper - Scalable Dataset Pipelines for Group-Structured Learning\n\n![PyPI version](https://img.shields.io/pypi/v/dataset_grouper)\n\nDataset Grouper is a library for creating, writing, and iterating over datasets\nwith group-level structure. It is primarily intended for creating large-scale\ndatasets for federated learning research.\n\n## Installation\n\nWe recommend installing via PyPI. Please check the [PyPI](https://pypi.org/project/dataset-grouper/) page for up-to-date version requirements, including python version requirements.\n\n```\npip install --upgrade pip\npip install dataset-grouper\n```\n\n## Getting Started\n\nBelow is a simple starting example, that partitions MNIST across 10 clients by\nlabel.\n\nFirst we import some necessary packages, including [Apache Beam](https://beam.apache.org/), which will be used to run the Dataset Grouper's pipelines.\n\n```python\nimport apache_beam as beam\nimport dataset_grouper as dsgp\nimport tensorflow_datasets as tfds\n```\n\nNext, we download and prepare the MNIST dataset.\n\n```python\ndataset_builder = tfds.builder('mnist')\ndataset_builder.download_and_prepare(...)\n```\n\nWe now write a function that assigns each MNIST example a client identifier\n(generally a `bytes` object). In this case, we will partition examples according\nto their label, but you can use much more interesting partition functions as\nwell.\n\n```python\ndef label_partition(x):\n  label = x['label'].numpy()\n  return str(label).encode('utf-8')\n```\n\nFinally, we build a pipeline that will partition MNIST according to this\nfunction, and run it using Beam's\n[Direct Runner](https://beam.apache.org/documentation/runners/direct/).\n\n```python\nmnist_pipeline = dsgp.tfds_to_tfrecords(\n    dataset_builder=dataset_builder,\n    split='test',\n    get_key_fn=label_partition,\n    file_path_prefix=...\n)\nwith beam.Pipeline() as root:\n  mnist_pipeline(root)\n```\n\nThis will save a version of MNIST that has been partitioned according to labels\nto a [TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) format.\nWe can also load it to iterate over client datasets.\n\n```python\npartitioned_dataset = dsgp.PartitionedDataset(\n  file_pattern=...,\n  tfds_features='mnist')\n\nfor group_dataset in partitioned_dataset.build_group_stream():\n  pass\n```\n\nGenerally, `PartitionedDataset.build_group_stream()` is a `tf.data.Dataset` that\nyields datasets, each of which is contains all the examples held by one group.\nIf you'd like to use these datasets with NumPy, you can simply do:\n\n```python\ngroup_dataset_numpy = group_dataset.as_numpy_iterator()\n```\n\n## What Else?\n\nThe example above is primarily for educational purposes. MNIST is a relatively\nsmall dataset, and can generally fit entirely into memory. For more interesting\nexamples, check out the examples folder.\n\nDataset Grouper is intended more for large-scale datasets, especially those\ndatasets that do not fit into memory. For these datasets, we recommend using\nmore sophisticated [Beam runners](https://beam.apache.org/documentation/runners/capability-matrix/), in order to partition the data in a distributed fashion.\n\n## Disclaimers\n\nThis is not an officially supported Google product.\n\nThis is a utility library that downloads and prepares public datasets. We do\nnot host or distribute these datasets, vouch for their quality or fairness, or\nclaim that you have license to use the dataset. It is your responsibility to\ndetermine whether you have permission to use the dataset under the dataset's\nlicense.\n\nIf you're interested in learning more about responsible AI practices, please\nsee Google AI's [Responsible AI Practices](https://ai.google/education/responsible-ai-practices).\n\nDataset Grouper is Apache 2.0 licensed. See the [`LICENSE`](LICENSE) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-parfait%2Fdataset_grouper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-parfait%2Fdataset_grouper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-parfait%2Fdataset_grouper/lists"}