{"id":16778775,"url":"https://github.com/ritvik19/scrapers","last_synced_at":"2025-04-10T20:51:26.733Z","repository":{"id":37060873,"uuid":"184921030","full_name":"Ritvik19/Scrapers","owner":"Ritvik19","description":null,"archived":false,"fork":false,"pushed_at":"2023-02-08T00:45:14.000Z","size":703,"stargazers_count":5,"open_issues_count":7,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-24T18:21:20.104Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://ritvik19.github.io","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ritvik19.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-04T16:48:18.000Z","updated_at":"2024-04-18T10:41:32.000Z","dependencies_parsed_at":"2024-10-13T07:28:37.987Z","dependency_job_id":"e14ad143-0154-49ed-830c-2c9a816be80c","html_url":"https://github.com/Ritvik19/Scrapers","commit_stats":null,"previous_names":["ritvik19/scrapers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ritvik19%2FScrapers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ritvik19%2FScrapers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ritvik19%2FScrapers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ritvik19%2FScrapers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ritvik19","download_url":"https://codeload.github.com/Ritvik19/Scrapers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248296645,"owners_count":21080302,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T07:28:38.588Z","updated_at":"2025-04-10T20:51:26.713Z","avatar_url":"https://github.com/Ritvik19.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lord-Varys\n\n_The Master of Whisperers_\n\n\u003e My little birds are everywhere, and they whisper to me the strangest stories...\n\nWeb Scrapers for various websites\n\nDependencies:\n* Anaconda Distribution for Python\n* Scrapy\n* Scrapy-Selenium\n* PRAW\n* Selenium\n* Pdfkit (For saving Webpages as pdf)\n* Spotipy\n\nThis repo consists of chrome extensions,  notebooks, scripts for automated browsers and crawlers that scrape:\n* [Youtube-Downloader](https://www.youtube.com): Downloads YouTube Videos ```Chrome Extension```\n* Sherlock-Holmes: General Purpose Webpage scrapper ```Chrome Extension```\n\n* [AltNews](https://www.altnews.in)\n  \n      scrapy crawl education\n      scrapy crawl news\n      scrapy crawl politics\n      scrapy crawl religion\n      scrapy crawl science\n      scrapy crawl society\n\n* [ANI](https://aninews.in/)\n  \n      scrapy crawl news\n\n* [Books2Scrape](https://books.toscrape.com)\n \n      scrapy crawl books\n\n* [Cambridge Dictionary](https://dictionary.cambridge.org/dictionary/): scrapes the defintion of word\n\n      scrapy crawl dict -a query=\u003cquery\u003e\n\n* [CodeChef](https://www.codechef.com/)\n      \n      scrapy crawl all\n      scrapy crawl beginner\n      scrapy crawl easy\n      scrapy crawl medium\n      scrapy crawl hard\n      scrapy crawl challenge\n      scrapy crawl peer\n\n* [CodeForces](https://codeforces.com/)\n      \n      scrapy crawl problems\n\n* [CodeGrepper](https://www.codegrepper.com/)\n\n      scrapy crawl codesnippets\n\n* Emails: Crawl emails mentioned on a complete website\n\n      scrapy crawl emails -a start_url=\u003chomepage_url\u003e -a thresh=\u003cnum bw 1 \u0026 100\u003e -a restrict_domain=\u003c0 or 1\u003e\n\n* [Github](https://github.com)\n  \n      scrapy crawl profileheatmap -a username=\u003cusername\u003e\n\n* [IMDB](https://www.imdb.com/chart/top)\n\n      scrapy crawl best_movies\n\n* [IndiaTV](https://www.indiatvnews.com/)\n\n      scrapy crawl auto\n      scrapy crawl brandcontent\n      scrapy crawl business\n      scrapy crawl crime\n      scrapy crawl education\n      scrapy crawl entertainment\n      scrapy crawl fyi\n      scrapy crawl health\n      scrapy crawl india\n      scrapy crawl jobs\n      scrapy crawl lifestyle\n      scrapy crawl politics\n      scrapy crawl science\n      scrapy crawl sports\n      scrapy crawl technology\n      scrapy crawl trending\n      scrapy crawl world\n\n* [Inshorts](https://inshorts.com/en/read)\n\n      scrapy crawl automobile\n      scrapy crawl business\n      scrapy crawl entertainment\n      scrapy crawl hatke\n      scrapy crawl home\n      scrapy crawl miscellaneous\n      scrapy crawl national\n      scrapy crawl politics\n      scrapy crawl science\n      scrapy crawl sports\n      scrapy crawl sports\n      scrapy crawl startup\n      scrapy crawl technology\n      scrapy crawl world\n\n* [JantaKaReporter](http://www.jantakareporter.com/)\n\n      scrapy crawl news\n\n* [Keep Inspiring](https://www.keepinspiring.me/category/quotes/)\n\n      scrapy crawl quotes\n\n* [LiveCoin](https://www.livecoin.net/en)\n\n      scrapy crawl livecoin\n\n* [Medium Stats](https://medium.com/): Stats for the articles of various publications\n\n      scrapy crawl analyticsvidhya -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl betterhuman -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl bettermarketing -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl betterprogramming -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl codeburst -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl dailyjs -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl datadriveninvestor -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl devbits -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl every30days -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl forge -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl freecodecamp -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl googledevelopers -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl javascriptinplainenglish -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl jupyterblog -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl learningnewstuff -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl levelupcoding -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl lifeofthought -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl omgfacts -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl onezero -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl pythonfeatures -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl pythoninplainenglish -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl pythonpandemonium -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl therenaissancedeveloper -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl thestartup -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl thewritingcooperative -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl towardsdatascience -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n      scrapy crawl uxcollective -a sdate=\u003cyyyy-mm-dd\u003e -a edate=\u003cyyyy-mm-dd\u003e\n\n* [MensXP](https://www.mensxp.com/)\n\n      scrapy crawl culture\n      scrapy crawl entertainment\n      scrapy crawl fashion\n      scrapy crawl grooming\n      scrapy crawl relationships\n      scrapy crawl social\n      scrapy crawl sports\n      scrapy crawl technology\n\n* [NASA APOD](https://api.nasa.gov/)\n\n      scrapy crawl apod\n\n* [OneLineFun](https://onelinefun.com)\n\n      scrapy crawl oneliners\n      \n* [OpIndia](https://www.opindia.com/)\n      \n      scrapy crawl opindia\n\n      scrapy crawl crime\n      scrapy crawl economy\n      scrapy crawl entertainment\n      scrapy crawl explainer\n      scrapy crawl factcheck\n      scrapy crawl government\n      scrapy crawl law\n      scrapy crawl media\n      scrapy crawl opinions\n      scrapy crawl politicalhistory\n      scrapy crawl politics\n      scrapy crawl sports\n      scrapy crawl variety\n      scrapy crawl virtualworld\n\n* [PostcardNews](https://postcard.news/timeline)\n      \n      scrapy crawl timeline\n\n* [ProjectEuler](https://projecteuler.net/archives)\n      \n      scrapy crawl problems\n\n* [Pypi](https://pypi.org)\n      \n      scrapy crawl packagedependency -a package=\u003cpackagename\u003e\n\n* [Quotes2Scrape](https://quotes.toscrape.com)\n\n      scrapy crawl quotes\n\n* [Reddit](http://reddit.com/)\n\n      scrapy crawl reddit -a subreddit=\u003csubreddit-name\u003e\n\n* [RVCJ](http://rvcj.com/)\n\n      scrapy crawl rvcj\n\n* [ScoopWhoop](https://www.scoopwhoop.com/)\n\n      scrapy crawl stories\n\n* [Shine_com](https://shine.com/)\n\n      scrapy crawl accounting-jobs\n      scrapy crawl finance-accounts-investment-banking-jobs\n\n* [SixWordStories](https://http://www.sixwordstories.net/)\n      \n      scrapy crawl stories\n\n* [SlickDeals](https://slickdeals.net/deal-categories/)\n\n      scrapy crawl computer\n\n* [Swarajya](https://swarajyamag.com/)\n\n      scrapy crawl api\n\n* [Syntax DB](https://syntaxdb.com/reference)\n\n      scrapy crawl snippets\n\n* [TFIPost](https://tfipost.com/)\n\n      scrapy crawl posts\n\n* [TheWeek](https://www.theweek.in/home.html)\n\n      scrapy crawl stories\n\n* [TheWire](https://thewire.in/)\n\n      scrapy crawl posts\n\n* [They Said So](https://theysaidso.com/api/)\n\n      scrapy crawl qod\n\n* [TutorialsPoint](https://www.tutorialspoint.com/programming_examples/)\n\n      scrapy crawl snippets\n\n* [ViralNova](https://viralnova.com/)\n\n      scrapy crawl viralnova\n\n* [ViralStories](http://viralstories.in/)\n\n      scrapy crawl viralstories\n\n* [Weather](https://openweathermap.org/)\n\n      scrapy crawl weather -a city=\u003ccity\u003e\n      scrapy crawl forecast -a city=\u003ccity\u003e\n\n* [Brainy Quote](https://www.brainyquote.com) ```Automated Browser```\n* [CSS Tricks](https://css-tricks.com/snippets/)\n* [Crazy Programmer](https://www.thecrazyprogrammer.com)\n* [Cricbuzz](https://www.cricbuzz.com): live cricket scores\n* [Goodreads-Quotes](https://www.goodreads.com/quotes)\n* [Google Images Scrapper](https://www.google.com/imghp?hl=en) ```Automated Browser```\n* Image Crawler: Downloads all images from a webpage\n* [IncludeHelp code-snippets](https://www.includehelp.com/code-snippets/)\n* [Java](https://jaxenter.com/15-useful-code-snippets-java-developers-131796.html)\n* [Jonas John](http://www.jonasjohn.de/snippets/all.htm)\n* [jQuery](https://www.thecrazyprogrammer.com/2015/01/useful-jquery-code-snippets.html)\n* [Medium-collections](https://medium.com/collections)\n* OnThisDay: crawls through [On This Day](https://www.onthisday.com/) and [Britannica](https://www.britannica.com/on-this-day) to fetch data into json format\n* [ProjectGutenberg](https://www.gutenberg.org/)\n* [Python Snyppets](https://snippets.readthedocs.io/en/latest/)\n* [Seb Sauvage](https://sebsauvage.net/python/snyppets/)\n* [Snipplr](https://snipplr.com/popular/language)\n* [Spotify](https://www.spotify.com/in/) ```API```\n* [Stackoverflow](https://stackoverflow.com): scrapes the result for a query\n* Website Info: provides various info about a website using [Alexa](https://www.alexa.com/siteinfo/) and [Whois](https://www.whois.com/)\n\nInstructions:\n* Crawlers:\n      Traverse to the directory in the command line and type\n```scrapy crawl \u003ccrawler name\u003e -o \u003coutfilename\u003e```\n* Notebooks:\n      Just run them in any jupyter environment\n* Automated Browser:\n    Make sure you have got chrome driver installed from [here](https://sites.google.com/a/chromium.org/chromedriver/downloads) before running the notebook\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fritvik19%2Fscrapers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fritvik19%2Fscrapers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fritvik19%2Fscrapers/lists"}