{"id":31004514,"url":"https://github.com/ryanrudes/wikimedia","last_synced_at":"2025-09-13T01:56:52.265Z","repository":{"id":112061556,"uuid":"376836098","full_name":"ryanrudes/wikimedia","owner":"ryanrudes","description":"A dataset comprised of over 40 million images sourced from Wikimedia Commons","archived":false,"fork":false,"pushed_at":"2023-10-22T22:02:10.000Z","size":15,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-02T03:55:26.446Z","etag":null,"topics":["computer-vision","data-science","data-scraping","dataset","datasets","deep-learning","gans","image","images","machine-learning","wikimedia","wikimedia-commons"],"latest_commit_sha":null,"homepage":"https://www.kaggle.com/ryanrudes/wikimedia","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanrudes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-06-14T13:39:12.000Z","updated_at":"2024-07-08T16:23:04.000Z","dependencies_parsed_at":null,"dependency_job_id":"cad08a5d-09ab-476f-b626-b8bbe776048b","html_url":"https://github.com/ryanrudes/wikimedia","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ryanrudes/wikimedia","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanrudes%2Fwikimedia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanrudes%2Fwikimedia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanrudes%2Fwikimedia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanrudes%2Fwikimedia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanrudes","download_url":"https://codeload.github.com/ryanrudes/wikimedia/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanrudes%2Fwikimedia/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274907848,"owners_count":25371822,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-12T02:00:09.324Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","data-science","data-scraping","dataset","datasets","deep-learning","gans","image","images","machine-learning","wikimedia","wikimedia-commons"],"created_at":"2025-09-13T01:55:55.926Z","updated_at":"2025-09-13T01:56:52.257Z","avatar_url":"https://github.com/ryanrudes.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://user-images.githubusercontent.com/18452581/121902225-1f5a5680-ccf5-11eb-8343-2a42720616d7.png\" width=\"100%\"\u003e\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/Ryan-Rudes/ad5bc9481ffb268e1cacaf3808d395e5/wikimedia-dataset-demo.ipynb)\n\n### Introduction\n\nWikimedia Commons Image Dataset is comprised of over 40 million URLs to Wikimedia Commons images.\n\n### Requirements\n\nThese are only required if you plan to run the data scraper yourself, which is unnecessary.\n\n- [`tqdm`](https://github.com/tqdm/tqdm)\n- [`bs4`](https://github.com/waylan/beautifulsoup)\n\nThese are the requirements for the PyTorch `DataLoader`:\n\n- [`torch`](https://github.com/pytorch/pytorch)\n- [`scikit-image`](https://github.com/scikit-image/scikit-image)\n- [`pillow`](https://github.com/python-pillow/Pillow)\n- [`opencv-python`](https://github.com/opencv/opencv)\n- [`torchvision`](https://github.com/pytorch/vision)\n\n### Data\n\nData is represented in a certain compressed format. URLs are newline-delimited.\n\nURLs to Wikimedia Commons images are formatted as follows: \\\n`https://upload.wikimedia.org/wikipedia/commons/thumb/\u003cID-1\u003e/\u003cID-2\u003e/\u003cFILENAME\u003e` \\\nor \\\n`https://upload.wikimedia.org/wikipedia/commons/\u003cID-1\u003e/\u003cID-2\u003e/\u003cFILENAME\u003e` (no /thumb/)\n\n`\u003cID-1\u003e` is 1 character in length, and `\u003cID-2\u003e` is 2\n\nFor each URL, it is compressed as follows:\n`\u003cTHUMB\u003e\u003cID-1\u003e\u003cID-2\u003e\u003cFILENAME\u003e`\n\nwhere `\u003cTHUMB\u003e` is a binary integer, indicating whether `/thumb/` is a component of the path.\n\nThere are 41666578 URLs in total, equating to 4.73 GB.\n\n### Usage\n\nIncluded is a:\n- [x] PyTorch `Dataset` and `DataLoader`\n- [ ] TensorFlow `Dataset`\n\n#### PyTorch `DataLoader` Usage\nTo demo the PyTorch `DataLoader`, first `cd` to the main directory. Then, download the dataset:\n```\nkaggle datasets download -d ryanrudes/wikimedia --unzip\n```\nThen, run the script:\n```\npython loaders/pytorch.py\n```\nYou can use this dataset by simply importing the `DataLoader` class, for example:\n```python\nfrom loaders.pytorch import WikimediaCommonsLoader\n\nloader = WikimediaCommonsLoader()\n\nfor batch in loader:\n    print (batch.shape)\n\n\u003e\u003e\u003e torch.Size([32, 3, 256, 256])\n\u003e\u003e\u003e torch.Size([32, 3, 256, 256])\n\u003e\u003e\u003e torch.Size([32, 3, 256, 256])\n\u003e\u003e\u003e torch.Size([32, 3, 256, 256])\n\u003e\u003e\u003e torch.Size([32, 3, 256, 256])\n...\n```\nYou can modify the following arguments of `WikimediaCommonsLoader`. Their default values are given below:\n```python\npath        = 'filtered.txt'\nverbose     = True\nmax_retries = None\ntimeout     = None\nshuffle     = True\nmax_buffer  = 4096\nworkers     = 8\ntransform   = None\nbatch_size  = 32\nresize_to   = 512\ncrop_to     = 256\n```\nOr, you can use the backbone `WikimediaCommonsDataset` class, which returns the raw images, one by one, without applying any transformations, whereas `WikimediaCommonsLoader` performs resizing and random cropping:\n```python\nfrom loaders.pytorch import WikimediaCommonsDataset\n\ndataset = WikimediaCommonsDataset()\n\nfor image in dataset:\n    print (image.shape)\n\n\u003e\u003e\u003e (120, 100, 3)\n\u003e\u003e\u003e (80, 120, 3)\n\u003e\u003e\u003e (120, 80, 3)\n\u003e\u003e\u003e (98, 120, 3)\n\u003e\u003e\u003e (120, 97, 3)\n\u003e\u003e\u003e (120, 120, 3)\n...\n```\nThe `WikimediaCommonsDataset` class takes almost the same arguments as `WikimediaCommonsLoader`, excluding `batch_size`, `resize_to`, and `crop_to`.\n\n### Links and Further Info\n\nThe dataset is available on [Kaggle](https://www.kaggle.com/ryanrudes/wikimedia)\n\nThis is licensed under the MIT license. Click [here](https://github.com/Ryan-Rudes/wikimedia/blob/master/LICENSE.txt) to learn more. All image links in this dataset are in the public domain. The only exception would be links to Wikimedia Foundation logos, which were already filtered prior to the data upload.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanrudes%2Fwikimedia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanrudes%2Fwikimedia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanrudes%2Fwikimedia/lists"}