{"id":16551126,"url":"https://github.com/michaelcurrin/scroll-and-scrape","last_synced_at":"2026-01-04T12:06:22.092Z","repository":{"id":93179965,"uuid":"195697768","full_name":"MichaelCurrin/scroll-and-scrape","owner":"MichaelCurrin","description":"Store tweets from a Twitter search results, using browser scraping","archived":false,"fork":false,"pushed_at":"2024-07-13T07:21:27.000Z","size":42,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-18T20:54:08.162Z","etag":null,"topics":["python","python3","scraping","selenium","tweets","twitter","twitter-search","webscraping"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MichaelCurrin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-07T21:16:29.000Z","updated_at":"2024-11-14T16:00:23.000Z","dependencies_parsed_at":"2024-10-11T19:36:32.708Z","dependency_job_id":"82fa7499-bb4e-46c6-8d8c-f7d19950fff3","html_url":"https://github.com/MichaelCurrin/scroll-and-scrape","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelCurrin%2Fscroll-and-scrape","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelCurrin%2Fscroll-and-scrape/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelCurrin%2Fscroll-and-scrape/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MichaelCurrin%2Fscroll-and-scrape/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MichaelCurrin","download_url":"https://codeload.github.com/MichaelCurrin/scroll-and-scrape/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245108310,"owners_count":20562021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","python3","scraping","selenium","tweets","twitter","twitter-search","webscraping"],"created_at":"2024-10-11T19:36:26.268Z","updated_at":"2026-01-04T12:06:22.023Z","avatar_url":"https://github.com/MichaelCurrin.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Scroll and Scrape\n\u003e Store tweets from a Twitter search results, using browser scraping\n\n[![GitHub tag](https://img.shields.io/github/tag/MichaelCurrin/scroll-and-scrape?include_prereleases=\u0026sort=semver)](https://github.com/MichaelCurrin/scroll-and-scrape/releases/)\n[![License](https://img.shields.io/badge/License-MIT-blue)](#license)\n\n[![Made with Python](https://img.shields.io/badge/Python-\u003e=3.6-blue?logo=python\u0026logoColor=white)](https://python.org)\n[![Made for Twitter](https://img.shields.io/badge/Made_for-Twitter-blue?logo=twitter\u0026logoColor=white)](https://twitter.com)\n[![dependency - selenium](https://img.shields.io/badge/dependency-selenium-blue)](https://pypi.org/project/selenium)\n[![dependency - beautifulsoup4](https://img.shields.io/badge/dependency-beautifulsoup4-blue)](https://pypi.org/project/beautifulsoup4)\n\n\n## Purpose\n\nThis application uses Python and the browser (controlled through Selenium) to load a page and scrape the contents and save the data. The approach here loads the DOM (not possible when using plain HTTP request) and also adds in waiting and scrolling logic so that dynamic elements can be pulled in.\n\nThis project is aimed at scraping Tweets on a search, where scrolling is needed to load more. It could be used for a user timeline though.\n\n\n## Requirements\n\n- [Python 3](https://python.org/)\n- Chrome browser\n\n\n## Clone\n\n```sh\n$ git clone git@github.com:MichaelCurrin/scroll-and-scrape.git\n$ cd scroll-and-scrape\n```\n\n\n## Installation\n\nInstall Python 3.\n\nCreate a virtual environment and activate it.\n\nInstall project packages:\n\n```sh\n$ pip install -r requirements.txt\n$ pip install -r requirements-dev.txt\n```\n\n\n## Usage\n\n```sh\n$ cd scrollscrape\n$ python main.py\n```\n\n\n## Background\n\nThis is a simple Python 3 application, based on existing scripts which are included in the [research](/research/) directory. See also [gist](https://gist.github.com/artjomb/07209e859f9bf0206f76).\n\nThe goal is to get all the Twitter tweets for a search query, going back as far as possible. \n\nUsing the Twitter API is restrictive - it only gives a week worth of data. Note that to keep this application simple, only the tweet ID needs to be stored and none of the tweet or author data. Since, once you have a tweet ID no matter how old, you can use the Twitter API to get all tweet and author data for a given tweet ID.\n\n\n## Future development\n\nSo far the main script works okay, but it not getting all known tweets for a particular known so it needs improvement. The format can be improved - for now it is printing to stdout which can be redirected to a text file.\n\nAlso, this could be improved for efficiency using a headless browser. It could also use a more modern library such as [requests-html](https://github.com/kennethreitz/requests-html).\n\n\n## License\n\nReleased under [MIT](/LICENSE) by [@MichaelCurrin](https://github.com/MichaelCurrin).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelcurrin%2Fscroll-and-scrape","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichaelcurrin%2Fscroll-and-scrape","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichaelcurrin%2Fscroll-and-scrape/lists"}