{"id":21124866,"url":"https://github.com/harveyslash/ms-celeb-extractor","last_synced_at":"2025-07-08T22:31:03.780Z","repository":{"id":37221087,"uuid":"279435316","full_name":"harveyslash/ms-celeb-extractor","owner":"harveyslash","description":"Extraction tool to parse MS Celeb dataset","archived":false,"fork":false,"pushed_at":"2022-12-08T11:08:51.000Z","size":38,"stargazers_count":3,"open_issues_count":12,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-03-04T12:42:21.533Z","etag":null,"topics":["data-science","dataset","dataset-manager","face-recognition","microsoft-research"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harveyslash.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-13T23:45:21.000Z","updated_at":"2022-07-27T17:28:14.000Z","dependencies_parsed_at":"2023-01-25T12:01:18.005Z","dependency_job_id":null,"html_url":"https://github.com/harveyslash/ms-celeb-extractor","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harveyslash%2Fms-celeb-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harveyslash%2Fms-celeb-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harveyslash%2Fms-celeb-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harveyslash%2Fms-celeb-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harveyslash","download_url":"https://codeload.github.com/harveyslash/ms-celeb-extractor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225465605,"owners_count":17478592,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","dataset","dataset-manager","face-recognition","microsoft-research"],"created_at":"2024-11-20T04:18:18.700Z","updated_at":"2024-11-20T04:18:19.391Z","avatar_url":"https://github.com/harveyslash.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ms-celeb-extractor\nExtraction tool to parse MS Celeb dataset\n\nThe [MS Celeb Dataset](https://github.com/EB-Dodo/C-MS-Celeb) is a database of faces with 6,464,018\n images. \n \n Due to some error, the original [dataset is gone](https://github.com/EB-Dodo/C-MS-Celeb/issues/1) . \n However, there is a torrent availble for use [here](https://academictorrents.com/details/9e67eb7cc23c9417f39778a8e06cca5e26196a97/tech\u0026hit=1\u0026filelist=1) \n It contains a tsv file with the images encoded as base64 strings. \n \n This extraction tool helps read through the tsv and place images of the same person in their respective folders. \n As it reads through the tsv file, it deletes the already read entries, meaning it requires no extra disk space to save the processed files.\n \n The reasoning for this is: \n \n1. Most libraries have built in helper functions to parse such a structure, including [pytorch](https://pytorch.org/docs/stable/torchvision/datasets.html#datasetfolder) and \n [keras/tensorflow](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image_dataset_from_directory)\n2. Modern file systems hash their files, so if the path of the file is known, reading it is O(1) time \n3. Storing as the original jpeg files give a reduction in size from 95 GB to 57 GB \n\n\n## Installing\n`pip install -r requirements.txt`\n\n\n## Usage \n\n```\nUsage: extractor.py [OPTIONS] COMMAND [ARGS]...\n\n  Utility to help extract MS Celeb data into manageable fils.\n\nOptions:\n  --help  Show this message and exit.\n\nCommands:\n  combine  Combine clean_list_128Vec_WT051_P010.txt and...\n  process  Read lines from the MS Celeb TSV file and save into a directory...\n  ```\n\n\nFirst use the combine command to combine the two text files provided in the dataset. Details of \nwhy to combine will be clear on referring to Section \"How to use C-MS-Celeb\" at \nhttps://github.com/EB-Dodo/C-MS-Celeb. Further, the 2 txt files are not found in the torrent but in\nhttps://github.com/EB-Dodo/C-MS-Celeb/blob/master/clean_list.7z\n\n\n\n\n```\nUsage: extractor.py combine [OPTIONS]\n\n  Combine clean_list_128Vec_WT051_P010.txt and relabel_list_128Vec_T058.txt\n  together.\n\n  The output of this file is used by the process command.\n\nOptions:\n  --clean_list_128_path FILENAME  Path of clean_list_128Vec_WT051_P010.txt\n                                  [required]\n\n  --relabel_list_128_path FILENAME\n                                  Path of relabel_list_128Vec_T058  [required]\n  --output_path FILE              Path of output file  [required]\n  --help                          Show this message and exit.\n  ```\n\n\nThen use the generated combined txt file into the process command to start extracting\nlines from the tsv and saving to jpeg files.\n\n```\n  Usage: extractor.py process [OPTIONS]\n\n  Read lines from the MS Celeb TSV file and save into a directory structure.\n  The files will be put in this format:\n\n      root/person_x/xxx.jpg     root/person_x/xxy.jpg\n      root/person_x/xxz.jpg\n\n      root/person_y/123.jpg     root/person_y/817.jpg\n      root/person_y/some.jpg\n\n  !NOTE!: As this command reads the TSV, it will delete the lines already\n  read.\n\nOptions:\n  --tsv_location FILENAME    Location of the entire MS Celeb tsv file\n                             [required]\n\n  --output_dir PATH          Output directory for images  [required]\n  --combined_file_path FILE  Location of the file generated by combine command\n                             [required]\n\n  --chunk_size INTEGER       Number of bytes to read from the tsv at once\n  --num_threads INTEGER\n  --help                     Show this message and exit.\n  ```\n\n\nExample:\n\n```bash\npython ms-celeb-extractor/extractor.py process --tsv_location=head.tsv --output_dir out --combined_file_path combined.txt\n89it [00:03, 23.58it/s]\n```\n\n## Contributing \nFeel free to add issues or pull requests\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharveyslash%2Fms-celeb-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharveyslash%2Fms-celeb-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharveyslash%2Fms-celeb-extractor/lists"}