{"id":15199034,"url":"https://github.com/rufiorogue/xdotcom-media-scraper","last_synced_at":"2026-03-06T17:39:00.412Z","repository":{"id":254039768,"uuid":"845276484","full_name":"rufiorogue/xdotcom-media-scraper","owner":"rufiorogue","description":"Scrape (download) mediafiles from x.com (twtter) accounts","archived":false,"fork":false,"pushed_at":"2024-08-22T19:00:36.000Z","size":39,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-03T14:11:15.633Z","etag":null,"topics":["download","images","media","python","selenium","twitter","videos","xcom"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rufiorogue.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-20T23:55:51.000Z","updated_at":"2025-02-02T22:54:49.000Z","dependencies_parsed_at":"2024-08-22T20:46:08.630Z","dependency_job_id":null,"html_url":"https://github.com/rufiorogue/xdotcom-media-scraper","commit_stats":{"total_commits":4,"total_committers":1,"mean_commits":4.0,"dds":0.0,"last_synced_commit":"a30df467bc36cda39b1febd1f5858a9bfbd35a24"},"previous_names":["rufiorogue/xdotcom-media-scraper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rufiorogue/xdotcom-media-scraper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rufiorogue%2Fxdotcom-media-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rufiorogue%2Fxdotcom-media-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rufiorogue%2Fxdotcom-media-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rufiorogue%2Fxdotcom-media-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rufiorogue","download_url":"https://codeload.github.com/rufiorogue/xdotcom-media-scraper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rufiorogue%2Fxdotcom-media-scraper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30188781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T17:33:53.563Z","status":"ssl_error","status_checked_at":"2026-03-06T17:33:51.678Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["download","images","media","python","selenium","twitter","videos","xcom"],"created_at":"2024-09-28T02:00:31.416Z","updated_at":"2026-03-06T17:39:00.391Z","avatar_url":"https://github.com/rufiorogue.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# x.com (twitter) media scraper\n\nWith provided user ID, this program will scrape statuses containing mediafiles and download mediafile resources.\nCurrently only image resources are supported. Uses selenium so no API knowledge required but may break in future if markup changes.\n\n## Build\n\n```\npoetry install\n```\n\n## Usage\n\nOn the initial run, cache login information:\n\n```\nx_media_scraper --cache-directory=cache login\n```\nIn selenium window log in to website then return to terminal and press Enter.\n\nYou now should be able to use scrape command line, for example:\n\n```\nx_media_scraper --cache-directory=cache scrape --user=TWITTER_USER_ID --output-directory=out\n```\n\n\n## selenium.common.exceptions.TimeoutException\n\nAt some point you will face the Elmo's notorious rate-limiter. The website just stops returning any meaningful data\nand then you get the above exception. In such case simply run the application again and it will pick where it left.\nTo force re-download existing items again delete the file `cache/visited.sqlite3`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frufiorogue%2Fxdotcom-media-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frufiorogue%2Fxdotcom-media-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frufiorogue%2Fxdotcom-media-scraper/lists"}