awesome-datascience
:memo: An awesome Data Science repository to learn and apply for real world problems.
https://github.com/academic/awesome-datascience
Last synced: 9 days ago
JSON representation
Agents
Frameworks
- ADK-Rust - Production-ready AI agent development kit for Rust with model-agnostic design (Gemini, OpenAI, Anthropic), multiple agent types (LLM, Graph, Workflow), MCP support, and built-in telemetry.
- Lumen - Agent framework for chatting with data, turning natural language into SQL, transformation pipelines and visualizations. Outputs are declarative specs that can be inspected, edited, reopened in a notebook or composed into a dashboard.
Research & Knowledge Retrieval
- Chunk Tuner - Open-source Python library and MCP server to benchmark document chunking strategies for RAG, score retrieval quality, and recommend configurations for a corpus.
- II-Commons - Daily-updated skill and CLI for deterministic retrieval across arXiv, PubMed/PMC, and supported US policy corpora.
- Suppr - AI literature search, document translation, and deep-research workspace for researchers.
- Spraay x402 Gateway - x402 payment gateway with 23 Research & Reference endpoints for AI agents: Wikipedia, arXiv, PubMed, Wikidata, academic citation lookup, entity extraction, and more. Pay-per-call in USDC on Base & Solana — no API keys or subscriptions. Also serves 150+ endpoints across 39 categories including geospatial, AI inference, DeFi, and compute. GitHub
Tools
- DeepAlpha - AI crypto trading framework using LightGBM + XGBoost ensemble with 72 ML features. 70.9% walk-forward validated accuracy on out-of-sample data. Supports Bybit and Binance. MIT licensed, available on PyPI.
- CAJAL - Local AI agent for generating publication-ready scientific papers with real arXiv citations, IMRaD structure, and tribunal scoring. Runs 100% offline via Ollama with 4B-9B models. MIT licensed. HuggingFace
- ai-evaluation - Open-source LLM and agent evaluation framework with 50+ metrics, LLM-as-Judge augmentation, and guardrail scanners (jailbreak, PII, prompt-injection). Useful for scoring RAG outputs, agent trajectories, and function-calling behavior in data-science workflows.
- Kitaru - Open-source platform that records real AI agent runs, replays them against changes, and evaluates outcomes before deployment.
Workflow
Fun
Comics
Datasets
- Academic Torrents
- ADS-B Exchange - Specific datasets for aircraft and Automatic Dependent Surveillance-Broadcast (ADS-B) sources.
- Public Big Data Sets
- data.gov - The home of the U.S. Government's open data
- United States Census Bureau
- usgovxml.com
- datahub.io
- datacite.org
- The official portal for European data
- NASDAQ:DATA - Nasdaq Data Link A premier source for financial, economic and alternative datasets.
- figshare.com
- GeoLite Legacy Downloadable Databases
- Quora's Big Datasets Answer
- Kaggle Datasets
- A Deep Catalog of Human Genetic Variation
- World Bank Data
- Open Data Philly
- grouplens.org
- research-quality data sets
- National Centers for Environmental Information
- ClimateData.us
- r/datasets
- MapLight - provides a variety of data free of charge for uses that are freely available to the general public. Click on a data set below to learn more
- GHDx - Institute for Health Metrics and Evaluation - a catalog of health and demographic datasets from around the world and including IHME results
- St. Louis Federal Reserve Economic Data - FRED
- New Zealand Institute of Economic Research – Data1850
- UNICEF Data
- undata
- NASA SocioEconomic Data and Applications Center - SEDAC
- The GDELT Project
- StackExchange Data Explorer - an open source tool for running arbitrary queries against public data from the Stack Exchange network.
- SocialGrep - a collection of open Reddit datasets.
- San Fransisco Government Open Data
- IBM Asset Dataset
- Open data Index
- Public Git Archive
- Microsoft Research Open Data
- Open Government Data Platform India
- Google Dataset Search (beta)
- IBB Open Portal
- The Humanitarian Data Exchange
- GHTorrent
- enigma.com - Navigate the world of public data - Quickly search and analyze billions of public records published by governments, companies and organizations.
- Hugging Face Datasets
- Open Data Sources
- NASA SocioEconomic Data and Applications Center - SEDAC
- StackExchange Data Explorer - an open source tool for running arbitrary queries against public data from the Stack Exchange network.
- Open data Index
- Open Government Data Platform India
- NAYN.CO Turkish News with categories
- Covid-19
- Covid-19 Google
- 5000 Images of Clothes
- IBB Open Portal
- 250k+ Job Postings - An expanding dataset of historical job postings from Luxembourg from 2020 to today. Free with 250k+ job postings hosted on AWS Data Exchange.
- FinancialData.Net - Financial datasets (stock market data, financial statements, sustainability data, and more).
- notesjor corpus-collection - Free corpora (over 6 billion tokens) mostly German (both historically and in contemporary German).
- CLARIN-Repository - CLARIN is a European repository for scientific datasets.
- IBM Asset Dataset
- AI Displacement Tracker - Structured dataset tracking 92 AI-attributed workforce reduction events affecting 453,748 workers across 12 countries and 11 sectors. JSON and CSV formats. CC-BY-4.0 licensed.
- GBIF - Global Biodiversity Information Facility: 2.4B+ species occurrence records. Free, open API for ecological modeling and ML research.
- FirstData - The world's most comprehensive authoritative data source knowledge base. 210+ curated sources from governments, international organizations, and research institutions. MCP integration for AI agents. MIT licensed.
- latamdata-py - Python package for one-line access to 38 open research datasets from Latin America (health, neuroscience, mental health, economics). pip install latamdata-py.
- Japan Neighborhoods - English dataset of Tokyo crime statistics across 5,078 neighborhoods × 7 years (36,222 records, 2018-2024), sourced from Tokyo Metropolitan Police open data. Includes interactive crime map, safety grading, and cost-of-living index. CC BY licensed.
- ZipCheckup - Free ZIP-level environmental safety data for 42,000+ US ZIP codes: water quality, air quality, PFAS contamination, radon, lead, flood risk, and 11 more verticals. Public REST API, npm/PyPI packages, CC BY 4.0.
- Helium - Real-time news corpus with structured bias features across 15+ dimensions (3.2M+ articles, 5,000+ sources), live financial market data (stocks, ETFs, crypto) with AI-generated analysis, ML options pricing with probability metrics and full Greeks, historical options chain data for quantitative research; available via MCP server or REST API.
- Packrift Packaging Optimization Benchmark Corpus - Public packaging product dataset generated from 1,000 exact-spec SKU records, with downloadable CSV and JSON files for ecommerce fulfillment and warehouse analysis.
- The Quiet-Broke Index - A 30-metro composite ranking of how much of a $400K household income gets consumed by housing, taxes, childcare, healthcare, and transport. Open methodology, free, no email gate.
- US Truck-Involved Fatal Crashes (FARS) 2018-2024 - Filtered subset of NHTSA Fatality Analysis Reporting System covering 33,898 fatal crashes involving medium and heavy commercial trucks across all 50 US states, 2018-2024. Includes interactive Vision Zero Report Card comparing 19 cities, reproducible Python pipeline on GitHub, and HuggingFace mirror. Permanent DOI, CC BY 4.0.
- Encyclopedia of Life - Open structured data on 1.9M+ species, including traits, classification, and media. Free API and bulk downloads for biodiversity and species-classification tasks.
- Chinese Tea Dataset - Curated open dataset of 100+ Chinese teas with category, origin, caffeine level, flavor notes, oxidation, and brewing parameters. Available as JSON and CSV.
- College ROI Dataset - Lifetime return-on-investment estimates for 29,700 US bachelor's programs across 3,392 colleges, built from FREOPP, IPEDS, and BEA regional price data. 5 CSVs with data dictionary, CC BY 4.0, Zenodo DOI.
- US Truck-Involved Fatal Crashes (FARS) 2018-2024 - Filtered subset of NHTSA Fatality Analysis Reporting System covering 33,898 fatal crashes involving medium and heavy commercial trucks across all 50 US states, 2018-2024. Includes interactive Vision Zero Report Card comparing 19 cities, reproducible Python pipeline on GitHub, and HuggingFace mirror. Permanent DOI, CC BY 4.0.
- State of Peptides 2026 - Structured reference dataset of 156 peptide and peptide-adjacent compounds, each with a regulatory status bucket, category, route, half-life, molecular weight, CAS number, reference count, and PubChem/DrugBank/Wikidata IDs. CSV and JSON, no login, CC BY 4.0.
- NASA SocioEconomic Data and Applications Center - SEDAC
- BDE Score - AI-powered multi-market stock analysis with transparent BDE scoring across 73 stocks (US/HK/A-share). EU AI Act Art.50 compliant. MIT license.
- Verified Supplement Evidence - Evidence-graded dietary-supplement dataset covering dosing, bioavailability by form, drug-nutrient interactions, NHANES deficiency prevalence, FDA FAERS adverse-event signals, and cost-per-effective-dose, with every clinical claim citing a PubMed PMID. CC BY 4.0, DOI 10.57967/hf/9356.
- FAOSTAT - UN FAO statistics on food production, trade, land use, and emissions for 245+ countries. Free API and bulk download.
- A community-curated database of well-known people, places, and things
- HDD Price Index - Daily open dataset of the cheapest new internal 3.5" SATA hard-drive price per terabyte (USD/TB) by capacity tier on Amazon US, with a historical time series. CSV, JSON and JSONL, no login, CC BY 4.0.
- Pokemon Card Centering Measurements - 320 measured PSA-style centering annotations (left/right and top/bottom border percentages, tilt) across 302 real eBay-listed Pokemon cards. CSV, CC BY 4.0, Zenodo DOI.
- Pokemon Card Sold-Price Reference by Grade - Median sold price by grade (raw, PSA 9, PSA 10) for 486 Pokemon cards, with sample size and confidence flag per card. CSV, CC BY 4.0, Zenodo DOI.
- Evidaxis Momentum Snapshots - Weekly snapshots of public development and citation activity for open-source and research-native AI systems, content-addressed and byte-reproducible from public inputs. JSON and CSV per snapshot date, CC0, DOI 10.5281/zenodo.21076011.
- Congressional Stock Brain - Free AI-powered tool that scores U.S. congressional STOCK Act trade disclosures by significance. Machine-scored signals from 537 lawmakers's public trade filings.
Infographics
- <img src="https://i.imgur.com/0OoLaa5.png" width="150" /> - differences-of-a-data-scientist-vs-data-engineer) |
- <img src="https://cloud.githubusercontent.com/assets/182906/19517857/604f88d8-960c-11e6-97d6-16c9738cb824.png" width="150" />
- <img src="https://i.imgur.com/W2t2Roz.png" width="150" />
- <img src="https://i.imgur.com/rb9ruaa.png" width="150" /> - a-data-scientist/). |
- <img src="https://i.imgur.com/XBgKF2l.png" width="150" />
- <img src="https://i.imgur.com/l9ZGtal.jpg" width="150" />
- <img src="https://i.imgur.com/TWkB4X6.png" width="150" />
- <img src="https://i.imgur.com/gtTlW5I.png" width="150" />
- <img src="https://scikit-learn.org/stable/_static/ml_map.png" width="150" />
- <img src="https://i.imgur.com/3JSyUq1.png" width="150" />
- <img src="https://i.imgur.com/DQqFwwy.png" width="150" />
- <img src="https://www.springboard.com/blog/wp-content/uploads/2016/03/20160324_springboard_vennDiagram.png" width="150" height="150" /> - science-career-paths-different-roles-industry/) by Springboard |
- <img src="https://data-literacy.geckoboard.com/assets/img/data-fallacies-to-avoid-preview.jpg" width="150" alt="Data Fallacies To Avoid" /> - data scientist/non-statistician colleagues how to avoid mistakes with data. From Geckoboard's Data Literacy Lessons. |
- <img src="https://scikit-learn.org/1.5/_downloads/b82bf6cd7438a351f19fac60fbc0d927/ml_map.svg" width="150" /> - learn.org/1.5/machine_learning_map.html#choosing-the-right-estimator) |
Literature and Media
Bloggers
- Wes McKinney - Wes McKinney Archives.
- Matthew Russell - Mining The Social Web.
- Greg Reda - Greg Reda Personal Blog
- Julia Evans - Recurse Center alumna
- Hakan Kardas - Personal Web Page
- Sean J. Taylor - Personal Web Page
- Drew Conway - Personal Web Page
- Hilary Mason - Personal Web Page
- Noah Iliinsky - Personal Blog
- Matt Harrison - Personal Blog
- Vamshi Ambati - AllThings Data Sciene
- Prash Chan - Tech Blog on Master Data Management And Every Buzz Surrounding It
- Clare Corthell - The Open Source Data Science Masters
- Paul Miller
- Data Science London - profit organization dedicated to the free, open, dissemination of data science.
- Datawrangling
- Quora Data Science - Data Science Questions and Answers from experts
- Siah
- Machine Learning Mastery
- Daniel Forsyth - Personal Blog
- Data Science Weekly - Weekly News Blog
- Revolution Analytics - Data Science Blog
- R Bloggers - R Bloggers
- The Practical Quant
- Yet Another Data Blog
- Spenczar - building to reporting.
- KD Nuggets
- Meta Brown - Personal Blog
- Data Scientist
- WhatSTheBigData
- Tevfik Kosar - Magnus Notitia
- New Data Scientist
- Harvard Data Science - Thoughts on Statistical Computing and Visualization
- Data Science 101 - Learning To Be A Data Scientist
- Kaggle Past Solutions
- datascientistjourney
- Learning Lover
- Dataists
- Data-Mania
- Data-Magnum
- P-value - Musings on data science, machine learning, and stats.
- Digital transformation
- Data Mania Blog - The File Drawer - Chris Said's science blog
- Emilio Ferrara's web page
- DataNews
- Reddit TextMining
- Periscopic
- Hilary Parker
- Data Science Lab
- Meaning of
- Adventures in Data Land
- DATA MINERS BLOG
- Dataclysm
- FlowingData - Visualization and Statistics
- Calculated Risk
- O'reilly Learning Blog
- Dominodatalab
- i am trask - A Machine Learning Craftsmanship Blog
- Vademecum of Practical Data Science - Handbook and recipes for data-driven solutions of real-world problems
- Dataconomy - A blog on the newly emerging data economy
- Springboard - A blog with resources for data science learners
- Analytics Vidhya - A full-fledged website about data science and analytics study material.
- Occam's Razor - Focused on Web Analytics.
- Data School - Data science tutorials for beginners!
- Colah's Blog - Blog for understanding Neural Networks!
- Sebastian's Blog - Blog for NLP and transfer learning!
- Chris Albon's Website - Data Science and AI notes
- Andrew Carr - Data Science with Esoteric programming languages
- floydhub - Blog for Evolutionary Algorithms
- Jingles - Review and extract key concepts from academic papers
- nbshare - Data Science notebooks
- Deep and Shallow - All things Deep and Shallow in Data Science
- Loic Tetrel - Data science blog
- Chip Huyen's Blog - ML Engineering, MLOps, and the use of ML in startups
- Maria Khalusova - Data science blog
- Aditi Rastogi - ML,DL,Data Science blog
- Santiago Basulto - Data Science with Python
- Akhil Soni - ML, DL and Data Science
- Akhil Soni - ML, DL and Data Science
- Dataclysm
- Dataclysm
- Dataclysm
- Dataclysm
- Dataclysm
- Dataclysm
- Dataclysm