{"id":31761544,"url":"https://github.com/stephanakkerman/x-timeline-scraper","last_synced_at":"2026-07-19T01:07:00.148Z","repository":{"id":315876792,"uuid":"1060831264","full_name":"StephanAkkerman/x-timeline-scraper","owner":"StephanAkkerman","description":"Scrapes X (formerly Twitter) timeline","archived":false,"fork":false,"pushed_at":"2025-09-21T09:35:23.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-21T11:33:04.376Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StephanAkkerman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":["StephanAkkerman"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"otechie":null,"lfx_crowdfunding":null,"custom":null}},"created_at":"2025-09-20T17:21:43.000Z","updated_at":"2025-09-21T09:35:26.000Z","dependencies_parsed_at":"2025-09-21T11:33:10.325Z","dependency_job_id":"4deee071-b835-4efa-a7d3-9d44eca13492","html_url":"https://github.com/StephanAkkerman/x-timeline-scraper","commit_stats":null,"previous_names":["stephanakkerman/x-timeline-scraper"],"tags_count":null,"template":false,"template_full_name":"StephanAkkerman/template","purl":"pkg:github/StephanAkkerman/x-timeline-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StephanAkkerman","download_url":"https://codeload.github.com/StephanAkkerman/x-timeline-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StephanAkkerman%2Fx-timeline-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002043,"owners_count":26083286,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-09T21:53:30.669Z","updated_at":"2026-07-19T01:07:00.137Z","avatar_url":"https://github.com/StephanAkkerman.png","language":"Python","funding_links":["https://github.com/sponsors/StephanAkkerman"],"categories":[],"sub_categories":[],"readme":"# X-Timeline Scraper\nA Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command.\n\n\u003c!-- Add a banner here like: https://github.com/StephanAkkerman/fintwit-bot/blob/main/img/logo/fintwit-banner.png --\u003e\n\n---\n\u003c!-- Adjust the link of the second badge to your own repo --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/python-3.13-blue.svg\" alt=\"Supported versions\"\u003e\n  \u003cimg src=\"https://img.shields.io/pypi/dm/xtimeline\" alt=\"pypi downloads\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/license/StephanAkkerman/x-timeline-scraper.svg?color=brightgreen\" alt=\"License\"\u003e\n  \u003ca href=\"https://github.com/psf/black\"\u003e\u003cimg src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" alt=\"Code style: black\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Introduction\n\nThis project provides a Python client to scrape tweets from X (formerly Twitter) timelines using a cURL command. It leverages asynchronous programming for efficient data retrieval and includes features for parsing tweet data.\n\n## Table of Contents 🗂\n\n- [Installation](#installation)\n- [Usage](#usage)\n- [Citation](#citation)\n- [Contributing](#contributing)\n- [License](#license)\n\n## Installation ⚙️\n\u003c!-- Adjust the link of the second command to your own repo --\u003e\n\nTo install the X-Timeline Scraper, you can use pip:\n\n```bash\npip install xtimeline\n```\n\n## Usage ⌨️\n\nTo use the X-Timeline Scraper, you need to provide a cURL command that accesses the desired X timeline. The instructions can be found in [curl_example.txt](curl_example.txt). Then, you can use the `XTimelineClient` class to fetch and parse tweets.\n\n### Fetching tweets once\n\n```python\nimport asyncio\nfrom xclient import XTimelineClient\n\nasync def main():\n    async with XTimelineClient(\"curl.txt\") as xc:\n        tweets = await xc.fetch_tweets()\n        for t in tweets:\n            print(t.to_markdown())\n\nasyncio.run(main())\n```\n\n### Streaming new tweets\n\n```python\nimport asyncio\nfrom xclient import XTimelineClient\n\nasync def main():\n    async with XTimelineClient(\n        \"curl.txt\", persist_last_id_path=\"state/last_id.txt\"\n    ) as xc:\n        async for t in xc.stream():\n            print(t.to_markdown())\n\nasyncio.run(main())\n```\n\nBy default, `stream()` now polls every ~30 seconds with built-in jitter (fuzzy interval) so requests do not follow an identical cadence.\n\n```python\n# 30s base with +-20% jitter (default)\nasync for t in xc.stream():\n    process(t)\n\n# Custom base interval and jitter\nasync for t in xc.stream(interval_s=45.0, jitter_ratio=0.15):\n    process(t)\n\n# Disable jitter if you need a fixed cadence\nasync for t in xc.stream(interval_s=30.0, jitter_ratio=0.0):\n    process(t)\n```\n\n### Fetch modes\n\nBoth `fetch_tweets()` and `stream()` accept a `mode` parameter that controls which tweets are returned:\n\n| Mode | Behaviour |\n|---|---|\n| `\"new_only\"` (default) | Only returns tweets newer than the last-seen cursor. The cursor advances so the same tweet is never emitted twice. |\n| `\"all\"` | Returns every tweet in each response. Nothing is filtered. Useful when your own store (e.g. a SQLite database) handles deduplication. |\n| `\"with_updates\"` | Returns new tweets **and** re-emits previously seen tweets whenever their metrics change (likes, retweets, views). Re-emitted tweets have `is_update=True`. |\n\n```python\n# Hand all deduplication to your own store\nasync for t in xc.stream(mode=\"all\"):\n    upsert_to_db(t)\n\n# Only new tweets, cursor persisted across restarts\nasync with XTimelineClient(\n    \"curl.txt\", persist_last_id_path=\"state/last_id.txt\"\n) as xc:\n    async for t in xc.stream(mode=\"new_only\"):\n        process(t)\n\n# New tweets + engagement updates\nasync for t in xc.stream(mode=\"with_updates\"):\n    if t.is_update:\n        update_metrics_in_db(t)\n    else:\n        insert_new_tweet(t)\n```\n\n### Tweet fields\n\nEach `Tweet` object contains:\n\n| Field | Type | Description |\n|---|---|---|\n| `id` | `int` | Tweet ID |\n| `text` | `str` | Full text, HTML entities unescaped, t.co links expanded, long-form tweets supported |\n| `user_name` | `str` | Display name |\n| `user_screen_name` | `str` | @handle (without @) |\n| `user_img` | `str` | Profile image URL |\n| `url` | `str` | Canonical tweet URL |\n| `created_at` | `str` | Post time in ISO 8601 format (`2026-04-01T19:15:49Z`) |\n| `likes` | `int` | Like count |\n| `retweets` | `int` | Retweet count |\n| `replies` | `int` | Reply count |\n| `views` | `int` | View count |\n| `media` | `list[MediaItem]` | Attached photos/videos |\n| `tickers` | `list[str]` | Uppercased `$TICKER` symbols |\n| `hashtags` | `list[str]` | Uppercased hashtags |\n| `title` | `str` | Human-readable summary, e.g. `\"TraderSZ retweeted Jelle\"` |\n| `is_update` | `bool` | `True` if this tweet was seen in a previous fetch this session |\n\n## Citation ✍️\n\u003c!-- Be sure to adjust everything here so it matches your name and repo --\u003e\nIf you use this project in your research, please cite as follows:\n\n```bibtex\n@misc{project_name,\n  author  = {Stephan Akkerman},\n  title   = {X-Timeline Scraper},\n  year    = {2025},\n  publisher = {GitHub},\n  journal = {GitHub repository},\n  howpublished = {\\url{https://github.com/StephanAkkerman/x-timeline-scraper}}\n}\n```\n\n## Contributing 🛠\n\u003c!-- Be sure to adjust the repo name here for both the URL and GitHub link --\u003e\nContributions are welcome! If you have a feature request, bug report, or proposal for code refactoring, please feel free to open an issue on GitHub. We appreciate your help in improving this project.\\\n![https://github.com/StephanAkkerman/x-timeline-scraper/graphs/contributors](https://contributors-img.firebaseapp.com/image?repo=StephanAkkerman/x-timeline-scraper)\n\n## License 📜\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstephanakkerman%2Fx-timeline-scraper/lists"}