{"id":19291579,"url":"https://github.com/jovianhq/opendatasets","last_synced_at":"2025-04-04T11:16:14.043Z","repository":{"id":40605064,"uuid":"296247014","full_name":"JovianHQ/opendatasets","owner":"JovianHQ","description":"A Python library for downloading datasets from Kaggle, Google Drive, and other online sources.","archived":false,"fork":false,"pushed_at":"2023-11-04T14:23:49.000Z","size":27197,"stargazers_count":333,"open_issues_count":13,"forks_count":141,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-03-28T10:08:43.506Z","etag":null,"topics":["data-science","datasets","machine-learning","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JovianHQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-17T07:07:35.000Z","updated_at":"2025-03-18T18:01:07.000Z","dependencies_parsed_at":"2024-06-18T14:08:14.358Z","dependency_job_id":null,"html_url":"https://github.com/JovianHQ/opendatasets","commit_stats":null,"previous_names":["jovianml/opendatasets"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JovianHQ%2Fopendatasets","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JovianHQ%2Fopendatasets/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JovianHQ%2Fopendatasets/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JovianHQ%2Fopendatasets/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JovianHQ","download_url":"https://codeload.github.com/JovianHQ/opendatasets/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166169,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","datasets","machine-learning","python"],"created_at":"2024-11-09T22:26:18.368Z","updated_at":"2025-04-04T11:16:14.020Z","avatar_url":"https://github.com/JovianHQ.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# opendatasets\n\n`opendatasets` is a Python library for downloading datasets from online sources like [Kaggle](https://www.kaggle.com/datasets) and Google Drive using a simple Python command. \n\n\n### Installation\n\nInstall the library using `pip`:\n\n```\npip install opendatasets --upgrade\n```\n\n### Usage - Downloading a dataset\n\nDatasets can be downloaded within a Jupyter notebook or Python script using the `opendatasets.download` helper function. Here's some sample code for downloading the [US Elections Dataset](https://www.kaggle.com/tunguz/us-elections-dataset):\n\n```\nimport opendatasets as od\ndataset_url = 'https://www.kaggle.com/tunguz/us-elections-dataset'\nod.download('https://www.kaggle.com/tunguz/us-elections-dataset')\n```\n\n`dataset_url` can also point to a public Google Drive link or a raw file URL.\n\n### Kaggle Credentials\n\n`opendatasets` uses the [Kaggle Official API](https://github.com/Kaggle/kaggle-api) for donwloading dataset from Kaggle.  Follow these steps to find your API credentials:\n\n1. Go to [https://kaggle.com/me/account](https://kaggle.com/me/account) (sign in if required).\n\n2. Scroll down to the \"API\" section and click \"Create New API Token\". This will download a file `kaggle.json` with the following contents:\n\n```\n{\"username\":\"YOUR_KAGGLE_USERNAME\",\"key\":\"YOUR_KAGGLE_KEY\"}\n```\n\n3. When you run `opendatsets.download`, you will be asked to enter your username \u0026 Kaggle API, which you can get from the file downloaded in step 2.\n\nNote that you need to download the `kaggle.json` file only once. You can also place the `kaggle.json` file in the same directory as the Jupyter notebook, and the credentials will be read automatically.\n\n**IMPORTANT NOTE**: If you're downloading a competition dataset, make sure to first accept the rules of the competition.\n\n### Some interesting datasets\n\nYou can find interesting datasets on Kaggle: https://www.kaggle.com/datasets\n\n*You can also create a new dataset on Kaggle by uploading a CSV file here: https://www.kaggle.com/datasets?new=true (make sure to keep your dataset public, otherwise it will not be downloadable)*\n\n- Video Games sales: https://www.kaggle.com/gregorut/videogamesales\n- World University Rankings: https://www.kaggle.com/mylesoneill/world-university-rankings\n- Netflix Tv shows and Movies: https://www.kaggle.com/shivamb/netflix-shows/notebooks\n- StackOverflow Developer Survey: https://www.kaggle.com/stackoverflow/stack-overflow-2018-developer-survey\n- Google Play Store Android Apps Data: https://www.kaggle.com/lava18/google-play-store-apps\n- Indian Stock Market Data: https://www.kaggle.com/rohanrao/nifty50-stock-market-data\n- Indian Air Quality: https://www.kaggle.com/rohanrao/air-quality-data-in-india\n- Worldwide Covid-19 Cases: https://www.kaggle.com/imdevskp/corona-virus-report\n- USA Covid-19 Cases: https://www.kaggle.com/sudalairajkumar/covid19-in-usa\n- US Election Results (2012): https://www.kaggle.com/tunguz/us-elections-dataset\n- US Stock Market: https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs/\n- Crop production in India: https://www.kaggle.com/srinivas1/agricuture-crops-production-in-india\n- Agricultural raw material prices: https://www.kaggle.com/kianwee/agricultural-raw-material-prices-19902020\n- Agricultural land values: https://www.kaggle.com/jmullan/agricultural-land-values-19972017\n- Digital payments in India: https://www.kaggle.com/lazycipher/upi-usage-statistics-aug16-to-feb20\n- US Unemployment Rate Data: https://www.kaggle.com/jayrav13/unemployment-by-county-us\n- India Road accident Data: https://community.data.gov.in/statistics-of-road-accidents-in-india/\n- Data Science Jobs Data:\n    - https://www.kaggle.com/sl6149/data-scientist-job-market-in-the-us\n    - https://www.kaggle.com/jonatancr/data-science-jobs-around-the-world\n    - https://www.kaggle.com/rkb0023/glassdoor-data-science-jobs\n- Youtube Trending Videos: https://www.kaggle.com/datasnaek/youtube-new\n- Asteroid Dataset: https://www.kaggle.com/sakhawat18/asteroid-dataset\n- Solar flares Data: https://www.kaggle.com/khsamaha/solar-flares-rhessi\n- F-1 Race Data: https://www.kaggle.com/cjgdev/formula-1-race-data-19502017\n- Automobile Insurance: https://www.kaggle.com/aashishjhamtani/automobile-insurance\n- PUBG video game matches: https://www.kaggle.com/skihikingkevin/pubg-match-deaths\n- CounterStrike GO (video game)\n    - https://www.kaggle.com/mateusdmachado/csgo-professional-matches\n    - https://www.kaggle.com/skihikingkevin/csgo-matchmaking-damage\n- Dota 2 (video game): https://www.kaggle.com/devinanzelmo/dota-2-matches\n- Cricket One-Day Internationals Data: https://www.kaggle.com/jaykay12/odi-cricket-matches-19712017\n- Cricket Indian Premier League Data: https://www.kaggle.com/nowke9/ipldata\n- Basketball (NCAA): https://www.kaggle.com/ncaa/ncaa-basketball\n- Basketball NBA Players Stats: https://www.kaggle.com/ncaa/ncaa-basketball\n- Football datasets: \n    - https://www.kaggle.com/martj42/international-football-results-from-1872-to-2017\n    - https://www.kaggle.com/abecklas/fifa-world-cup\n    - https://www.kaggle.com/egadharmawan/uefa-champion-league-final-all-season-19552019\n- Hotel Booking Demand: https://www.kaggle.com/jessemostipak/hotel-booking-demand\n- New York Airbnb listings: https://www.kaggle.com/dgomonov/new-york-city-airbnb-open-data\n\n\nOther sources to look for datasets: \n- [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/index.php)\n- [awesome-public-datasets](https://github.com/awesomedata/awesome-public-datasets)\n- [Google Dataset Search](https://datasetsearch.research.google.com)\n\n*If you use an external source other than Kaggle, you'll create a new dataset on Kaggle by uploading a CSV file here: https://www.kaggle.com/datasets?new=true (make sure to keep your dataset public, otherwise it will not be downloadable using `opendatasets`)*\n\n\n\n## Curated Datasets\n\n`opendatasets` also provides some curated datsets that you can download by passing the Dataset ID to `opendatasets.download`. Here's an example:\n\n```\nimport opendatasets\nopendatasets.download('stackoverflow-developer-survey-2020')\n```\n\nThe following datasets are available for download.\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eDataset ID\u003c/th\u003e\n        \u003cth\u003eDescription\u003c/th\u003e\n        \u003cth\u003eSource\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003estackoverflow-developer-survey-2020\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eStack Overflow Developer Survey 2020\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://insights.stackoverflow.com/survey/\"\u003eStack Overflow\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003eowid-covid-19-latest\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eCovid-19 Stats by Our World in Data\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://github.com/owid/covid-19-data/tree/master/public/data\"\u003eOur World in Data\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003estate-of-javascript-2016\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eState of Javascript Annual Survey 2016\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://www.kaggle.com/sachag/state-of-js-2019\"\u003eStateOfJS\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003estate-of-javascript-2017\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eState of Javascript Annual Survey 2017\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://www.kaggle.com/sachag/state-of-js-2019\"\u003eStateOfJS\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003estate-of-javascript-2018\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eState of Javascript Annual Survey 2018\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://www.kaggle.com/sachag/state-of-js-2019\"\u003eStateOfJS\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003estate-of-javascript-2019\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eState of Javascript Annual Survey 2019\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://www.kaggle.com/sachag/state-of-js-2019\"\u003eStateOfJS\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ccode\u003ecountries-languages-spoken\u003c/code\u003e\u003c/td\u003e\n        \u003ctd\u003eLanguages Spoken in Different Countries\u003c/td\u003e\n        \u003ctd\u003e\n            \u003ca href=\"https://www.infoplease.com/world/countries/languages-spoken-in-each-country-of-the-world\"\u003eInfoplease\u003c/a\u003e\n        \u003c/td\u003e\n    \u003c/tr\u003e\n\u003c/table\u003e\n\nMore datasets will be added soon..\n\n## Contributing\n\nThis is an open source project and we welcome contributions.\n\n### Local Development Setup\n\n1. Clone the repository:\n\n```\ngit clone https://github.com/JovianML/opendatasets.git\n```\n\n2. Setup the Python environment for development\n\n```\nconda create -n opendatasets python=3.5\nconda activate opendatasets\npip install -r requirements.txt\n```\n\n3. Open up the project in VS code and make your changes. Make sure to install the Python Extension for VS Code and select the `opendatasets` conda environment.\n\nThis package is developed and maintained by the [Jovian](https://www.jovian.ai) team.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjovianhq%2Fopendatasets","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjovianhq%2Fopendatasets","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjovianhq%2Fopendatasets/lists"}