{"id":31761544,"url":"https://github.com/stephanakkerman/x-timeline-scraper","last_synced_at":"2025-10-09T21:53:33.137Z","repository":{"id":315876792,"uuid":"1060831264","full_name":"StephanAkkerman/x-timeline-scraper","owner":"StephanAkkerman","description":"Scrapes X (formerly Twitter) timeline","archived":false,"fork":false,"pushed_at":"2025-09-21T09:35:23.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-21T11:33:04.376Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StephanAkkerman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["StephanAkkerman"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2025-09-20T17:21:43.000Z","updated_at":"2025-09-21T09:35:26.000Z","dependencies_parsed_at":"2025-09-21T11:33:10.325Z","dependency_job_id":"4deee071-b835-4efa-a7d3-9d44eca13492","html_url":"https://github.com/StephanAkkerman/x-timeline-scraper","commit_stats":null,"previous_names":["stephanakkerman/x-timeline-scraper"],"tags_count":null,"template":false,"template_full_name":"StephanAkkerman/template","purl":"pkg:github/StephanAkkerman/x-timeline-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StephanAkkerman","download_url":"https://codeload.github.com/StephanAkkerman/x-timeline-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002043,"owners_count":26083286,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-09T21:53:30.669Z","updated_at":"2025-10-09T21:53:33.132Z","avatar_url":"https://github.com/StephanAkkerman.png","language":"Python","funding_links":["https://github.com/sponsors/StephanAkkerman"],"categories":[],"sub_categories":[],"readme":"# X-Timeline Scraper\nA Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command.\n\n\u003c!-- Add a banner here like: https://github.com/StephanAkkerman/fintwit-bot/blob/main/img/logo/fintwit-banner.png --\u003e\n\n---\n\u003c!-- Adjust the link of the second badge to your own repo --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3.13-blue.svg\" alt=\"Supported versions\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/license/StephanAkkerman/x-timeline-scraper.svg?color=brightgreen\" alt=\"License\"\u003e\n  \u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" alt=\"Code style: black\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Introduction\n\nThis project provides a Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command. It leverages asynchronous programming for efficient data retrieval and includes features for parsing tweet data.\n\n## Table of Contents 🗂\n\n- [Installation](#installation)\n- [Usage](#usage)\n- [Citation](#citation)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Installation ⚙️\n\u003c!-- Adjust the link of the second command to your own repo --\u003e\n\nTo install the X-Timeline Scraper, you can use pip:\n\n```bash\npip install xtimeline\n```\n\n## Usage ⌨️\n\nTo use the X-Timeline Scraper, you need to provide a cURL command that accesses the desired X timeline. The instructions can be found in [curl_example.txt](curl_example.txt). Then, you can use the `XTimelineClient` class to fetch and parse tweets.\n\nHere's a simple example of how to use the client:\n\n```python\nimport asyncio\nfrom src.xclient import XTimelineClient\n\nasync with XTimelineClient(\n        \"curl.txt\", persist_last_id_path=\"state/last_id.txt\"\n    ) as xc:\n        tweets = await xc.fetch_tweets(update_last_id=False)\n        for t in tweets:\n            print(t.to_markdown())\n```\n\nYou can also stream new tweets in real-time:\n\n```python\nimport asyncio\nfrom src.xclient import XTimelineClient\nasync with XTimelineClient(\n        \"curl.txt\", persist_last_id_path=\"state/last_id.txt\"\n    ) as xc:\n        async for t in xc.stream(interval_s=5.0):\n            print(t.to_markdown())\n```\n\n## Citation ✍️\n\u003c!-- Be sure to adjust everything here so it matches your name and repo --\u003e\nIf you use this project in your research, please cite as follows:\n\n```bibtex\n@misc{project_name,\n  author  = {Stephan Akkerman},\n  title   = {X-Timeline Scraper},\n  year    = {2025},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/StephanAkkerman/x-timeline-scraper}}\n}\n```\n\n## Contributing 🛠\n\u003c!-- Be sure to adjust the repo name here for both the URL and GitHub link --\u003e\nContributions are welcome! If you have a feature request, bug report, or proposal for code refactoring, please feel free to open an issue on GitHub. We appreciate your help in improving this project.\\\n![https://github.com/StephanAkkerman/x-timeline-scraper/graphs/contributors](https://contributors-img.firebaseapp.com/image?repo=StephanAkkerman/x-timeline-scraper)\n\n## License 📜\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper/lists"}