{"id":15037108,"url":"https://github.com/apple/ml-flair","last_synced_at":"2025-10-19T22:32:47.596Z","repository":{"id":37845099,"uuid":"320045553","full_name":"apple/ml-flair","owner":"apple","description":"A large labelled image dataset for benchmarking in federated learning","archived":false,"fork":false,"pushed_at":"2024-03-06T16:50:48.000Z","size":6783,"stargazers_count":94,"open_issues_count":2,"forks_count":15,"subscribers_count":10,"default_branch":"main","last_synced_at":"2025-01-30T07:42:36.578Z","etag":null,"topics":["dataset","differential-privacy","federated-learning"],"latest_commit_sha":null,"homepage":"https://proceedings.neurips.cc/paper_files/paper/2022/file/f64e55d03e2fe61aa4114e49cb654acb-Paper-Datasets_and_Benchmarks.pdf","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-12-09T18:31:43.000Z","updated_at":"2025-01-16T06:29:17.000Z","dependencies_parsed_at":"2023-02-13T19:17:01.468Z","dependency_job_id":null,"html_url":"https://github.com/apple/ml-flair","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-flair","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-flair/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-flair/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-flair/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-flair/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237224875,"owners_count":19275102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","differential-privacy","federated-learning"],"created_at":"2024-09-24T20:33:23.708Z","updated_at":"2025-10-19T22:32:40.512Z","avatar_url":"https://github.com/apple.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Federated Learning Annotated Image Repository (FLAIR): A large labelled image dataset for benchmarking in federated learning\n\n**Paper:** [link](https://proceedings.neurips.cc/paper_files/paper/2022/file/f64e55d03e2fe61aa4114e49cb654acb-Paper-Datasets_and_Benchmarks.pdf)\n\n**Update:** FLAIR is now available in [pfl-research](https://github.com/apple/pfl-research/tree/develop/benchmarks/flair)!\n\nFLAIR is a large dataset of images that captures a number of characteristics encountered in federated learning and privacy-preserving ML tasks. \nThis dataset comprises approximately 430,000 images from 51,000 Flickr users, which will better reflect federated learning problems arising in practice, and it is being released to aid research in the field.\n\n![alt text](assets/FLAIR_sample.jpeg)\n\n## Image Labels\nThese images have been annotated by humans and assigned labels from a taxonomy of more than 1,600 fine-grained labels. \nAll main subjects present in the images have been labeled, so images may have multiple labels. \nThe taxonomy is hierarchical where the fine-grained labels can be mapped to 17 coarse-grained categories.\nThe dataset includes both fine-grained and coarse-grained labels so researchers can vary the complexity of a machine learning task.\n\n## User Labels and their use for Federated Learning\nWe have used image metadata to extract artist names/IDs for the purposes of creating user datasets for federated learning. \nWhile optimization algorithms for machine learning are often designed under the assumption that each example is an independent sample from the distribution, federated learning applications deviate from this assumption in a few different ways that are reflected in our user-annotated examples. \nDifferent users differ in the number of images they have, as well as the number of classes represented in their image collection. \nFurther, images of the same class but taken by different users are likely to have some distribution shift. \nThese properties of the dataset better reflect federated learning applications, and we expect that benchmark tasks on this dataset will benefit from algorithms designed to handle such data heterogeneity.\n\n## Getting Started\n### Prerequisites\nPlease make sure you have python \u003e= 3.8 and have the required packages installed (see below).\n```sh\npython3 -m pip install -r requirements.txt\n```\n### Download the dataset\nEnsure you have a good network connection to download the ~6GB of image data, and enough local space to store and decompress it.\nDownload the dataset with the following command:\n```sh\npython3 download_dataset.py --dataset_dir=/path/to/data \n```\nThe images and metadata will be saved to the provided `dataset_dir`.\nBy default the script will download the down-sized images (size = 256 x 256). \nThe images are split and compressed into dozens of tarball archives and will be decompressed after downloading.\nIf you wish to download the full-size raw images, add `--download_raw` flag in the above command.\\\n ⚠️ Warning: the raw images take up to ~1.2TB disk space to store after decompressing.\n\nAfter downloading and decompressing, the `dataset_dir` will have the following layout:\n```\ndataset_dir\n├── labels_and_metadata.json      # a list of labels and metadata for each image\n├── label_relationship.txt        # a list of `(fine-grained label, label)` pair\n├── small_images\n│   └── *.jpg                     # all down-sized images\n└── raw_images                    # exists if you added `--download_raw` flag\n    └── *.jpg                     # all raw images\n```\n\n### Dataset split\nWe include a standard train/val/test split in `labels_and_metadata.json`.\nThe partition is based on user ids with ratio 8:1:1, i.e. train, val and test sets have disjoint users.\nBelow are the numbers for each partition:\n\n| Partition        | Train   | Val    | Test   |\n| ---------------- | ------- | ------ | ------ |\n| Number of users  | 41,131  | 5,141  | 5,142  |\n| Number of images | 345,879 | 39,239 | 43,960 |\n\nWe recommend using the provided split for reproducible benchmarks.\n\n### Explore the dataset\nBelow is an example metadata and label for one image from `labels_and_metadata.json`:\n```json\n{\n    \"user_id\": \"59769174@N00\",\n    \"image_id\": \"14913474848\",\n    \"fine_grained_labels\": [\n        \"bag\",\n        \"document\",\n        \"furniture\",\n        \"material\",\n        \"printed_page\"\n    ],\n    \"labels\": [\n        \"equipment\",\n        \"material\",\n        \"structure\"\n    ],\n    \"partition\": \"train\"\n}\n```\nField `image_id` is the Flickr PhotoID and `user_id` is the Flickr NSID that owns the image.\nField `partition` denotes which `train/dev/test` partition the image belongs to.\nField `fine_grained_labels` is a list of annotated labels presenting the subjects in the image and `labels` is the list of coarse-grained labels obtained by  mapping fine-grained labels to higher-order categories.\nThe file `label_relationship.txt` includes the mapping from ~1,600 fine-grained labels to 17 higher-order categories.\n\nWe provide scripts to explore the images and labels in more detail. First you need to start a jupyter notebook:\n```sh\njupyter notebook\n```\n- To explore the downloaded images, open in jupyter notebook [`explore_images.ipynb`](./explore_images.ipynb) which displays the images with corresponding metadata and labels.\n- To explore the labels, open in jupyter notebook  [`explore_labels.ipynb`](./explore_labels.ipynb) which displays the statistics of the user and label distribution.\n\n### (Optional) Prepare the dataset in HDF5\nWe provide a script to prepare the dataset in HDF5 format for more efficient processing and training:\n```sh\npython3 prepare_dataset.py --dataset_dir=/path/to/data --output_file=/path/to/hdf5\n```\nBy default the script will group the images and labels by train/val/test split and then by user ids, making it suitable for federated learning experiments.\nWith the flag `--not_group_data_by_user`, the script will simply group the images and labels by train/val/test split and ignore the user ids, which is the typical setup for centralized training. \\\n⚠️ Warning: the hdf5 file take up to ~80GB disk space to store after processing.\n\n## Benchmark FLAIR with TensorFlow Federated\n\n### Prepare the dataset in TFRecords\nWe provide a script to prepare the dataset in TFRecords format for benchmarking with TensorFlow Federated:\n```sh\npython3 prepare_tfrecords.py --dataset_dir=/path/to/data --tfrecords_dir=/path/to/tfrecords\n```\nWhen the above script finishes, the `tfrecords_dir` will have the following layout:\n```\ntfrecords_dir\n├── label_index.json             # a mapping from class label to index    \n├── train\n│   └── \u003cuser-id\u003e.tfrecords      # tfrecords for all train users\n├── dev\n│   └── \u003cuser-id\u003e.tfrecords      # tfrecords for all dev users\n└── test                    \n    └── \u003cuser-id\u003e.tfrecords      # tfrecords for all test users\n```\n### Training in centralized setting\nIn centralized setting, user split is ignored and all users' data are concatenated.\nCentralized model training can be done in TensorFlow Keras with the following command:\n```sh\npython3 -m benchmark.central_main --tfrecords_dir=/path/to/tfrecords\n```\nTo view all available arguments, please use the following command: \n```sh\npython3 -m benchmark.central_main --help\n```\nPlease refer to our [benchmark paper](https://arxiv.org/abs/2207.08869) for the recommended hyperparameters.\n\n### Training in federated setting\nIn federated setting, sampled users train on their own data locally and then share the model updates with the central server.\nFederated model training can be simulated in TensorFlow Federated with the following command:\n```sh\npython3 -m benchmark.federated_main --tfrecords_dir=/path/to/tfrecords\n```\nTo view all available arguments, please use the following command: \n```sh\npython3 -m benchmark.federated_main --help\n```\nPlease refer to our [benchmark paper](https://arxiv.org/abs/2207.08869) for the recommended hyperparameters.\n\n### Training in federated setting with differential privacy\nTo provide a formal privacy guarantee, we use [DP-SGD](https://arxiv.org/abs/1607.00133)\nin the [federated context](https://arxiv.org/abs/1710.06963) which is supported in TensorFlow Federated.\nThe following command enables federated learning with differential privacy:\n```sh\npython3 -m benchmark.federated_main --tfrecords_dir=/path/to/tfrecords --epsilon=2.0 --l2_norm_clip=0.1\n```\nwhere `epsilon` is the privacy budget and `l2_norm_clip` is the L2 norm clipping bound for Gaussian mechanism.\nBy default, we use [adaptive clipping](https://arxiv.org/abs/1905.03871v3) to tune the L2 norm clipping bound automatically by setting `--target_unclipped_quantile=0.1`.\n\n### Fine-tuning a pretrained ImageNet model\nAbove commands are all for training from a random initialized model.\nWe also provide a ResNet model pretrained on ImageNet, which can be downloaded with the following command: \n```sh\nwget -O /path/to/model https://docs-assets.developer.apple.com/ml-research/datasets/flair/models/resnet18.h5\n```\nThe pretrained model is originally from [torch vision](https://download.pytorch.org/models/resnet18-f37072fd.pth) and converted to Keras format.\nTo use the pretrained model, please add the argument `--restore_model_path=/path/to/model` in the above training commands.\n\n### Training a binary classifier for a single label\nBy default, we train a multi-label classification model where the output is a multi-hot vector indicating which labels presented in the input image.\nWe also provide the option to train a simpler binary classifier for a single label. \nFor example, adding the argument `--binary_label=structure` trains a model only to predict whether `structure` label presented in the image. \n\n## Disclaimer\nThe annotations and Apple’s other rights in the dataset are licensed under CC-BY-NC 4.0 license. \nThe images are copyright of the respective owners, the license terms of which can be found using the links provided in ATTRIBUTIONS.TXT (by matching the Image ID). \nApple makes no representations or warranties regarding the license status of each image and you should verify the license for each image yourself.\n\n## Citing FLAIR\n\n```\n@article{song2022flair,\n  title={FLAIR: Federated Learning Annotated Image Repository},\n  author={Song, Congzheng and Granqvist, Filip and Talwar, Kunal},\n  journal={Advances in Neural Information Processing Systems},\n  volume={35},\n  pages={37792--37805},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-flair","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-flair","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-flair/lists"}