{"id":20081398,"url":"https://github.com/prirai/rare-diseases-data-scraping","last_synced_at":"2026-06-04T22:32:02.903Z","repository":{"id":64925562,"uuid":"578963398","full_name":"prirai/rare-diseases-data-scraping","owner":"prirai","description":"This repository aims to be a central place for all data scraping and analysis related to rare diseases.","archived":false,"fork":false,"pushed_at":"2022-12-21T06:26:27.000Z","size":27237,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-13T01:42:26.368Z","etag":null,"topics":["csv","ontology","python","rare-disease"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prirai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-12-16T10:10:05.000Z","updated_at":"2023-03-15T10:57:55.000Z","dependencies_parsed_at":"2023-01-30T02:46:27.746Z","dependency_job_id":null,"html_url":"https://github.com/prirai/rare-diseases-data-scraping","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prirai%2Frare-diseases-data-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prirai%2Frare-diseases-data-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prirai%2Frare-diseases-data-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prirai%2Frare-diseases-data-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prirai","download_url":"https://codeload.github.com/prirai/rare-diseases-data-scraping/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241515803,"owners_count":19975139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["csv","ontology","python","rare-disease"],"created_at":"2024-11-13T15:38:56.762Z","updated_at":"2026-06-04T22:32:02.712Z","avatar_url":"https://github.com/prirai.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# rare-diseases-data-scraping\n\nData scraped from - https://rarediseases.info.nih.gov\n\n`data1.csv` is the currently incomplete detailed description about the diseases.\n\n## File and their purpose\n----\n|File|Purpose|\n|--|--|\n`links.md` | links of interest\n`disease_links_scraper.py` | extracts the big list from https://rarediseases.info.nih.gov/diseases.\n`disease_links.csv` | data extracted using the above script\n`scrape_specific_page.ipynb` | Scrapes a given page for disease details (names, symptoms and causes).\n`reqmul.py` | Requests with multithreading - made to save all the 5910 pages offline as HTML for easy scraping later.\n\n## Contents of `pages` folder\n\nThis directory contains all the files, the links for which were already scraped and are included in `disease_links.csv`. The naming convention is just the part after the last `/` in the corresponding url.\n`reqmul.py` checks from previously downloaded files and doesn't overwrite. Current script allows somewhere around 700 pages after which it gets a server restriction. The previous versions were even worse and could only get 100 to max 200 at a time and that too in a long time. wget, axel, aria2, selenium have been already tried.\n\nScraped Disease at a glance section, people affected,  symptoms, categories, ages and causes for all diseases using the offline HTML pages (code in `page_details_extractor-offline.ipynb`). They are saved in the `disease_details.csv` file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprirai%2Frare-diseases-data-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprirai%2Frare-diseases-data-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprirai%2Frare-diseases-data-scraping/lists"}