{"id":15434025,"url":"https://github.com/sebinsua/scrape-twitter","last_synced_at":"2025-04-09T18:16:27.294Z","repository":{"id":46790167,"uuid":"76662235","full_name":"sebinsua/scrape-twitter","owner":"sebinsua","description":"🐦 Access Twitter data without an API key. [DEPRECATED]","archived":false,"fork":false,"pushed_at":"2019-01-24T00:59:55.000Z","size":536,"stargazers_count":174,"open_issues_count":17,"forks_count":36,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-09T18:16:23.001Z","etag":null,"topics":["cli","conversation","likes","scraper","streams","timeline","tweets","twitter"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sebinsua.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-16T15:02:26.000Z","updated_at":"2024-12-26T11:27:28.000Z","dependencies_parsed_at":"2022-08-25T11:02:08.903Z","dependency_job_id":null,"html_url":"https://github.com/sebinsua/scrape-twitter","commit_stats":null,"previous_names":[],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebinsua%2Fscrape-twitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebinsua%2Fscrape-twitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebinsua%2Fscrape-twitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sebinsua%2Fscrape-twitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sebinsua","download_url":"https://codeload.github.com/sebinsua/scrape-twitter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248085322,"owners_count":21045139,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","conversation","likes","scraper","streams","timeline","tweets","twitter"],"created_at":"2024-10-01T18:36:45.781Z","updated_at":"2025-04-09T18:16:27.270Z","avatar_url":"https://github.com/sebinsua.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# `scrape-twitter` [![Build Status](https://travis-ci.org/sebinsua/scrape-twitter.png)](https://travis-ci.org/sebinsua/scrape-twitter) [![npm version](https://badge.fury.io/js/scrape-twitter.svg)](https://npmjs.org/package/scrape-twitter)\n\u003e 🐦 Access Twitter data without an API key\n\n**DEPRECATED**: The next version of Twitter auto-generates all of the class names and therefore is going to be much more difficult to screen scrape. I might come up with a better solution...\n\n___\n\nThis module provides command line interfaces to scrape: profiles, timelines, connections, likes, search and conversations.\n\nIt also exposes both streams and a promise returning function to help accessing Twitter in your own applications.\n\nReal-time firehoses can be created using the companion module [`monitor-head-stream`](https://github.com/sebinsua/monitor-head-stream#example). \n\n## Features\n\n- [x] Get Twitter data without being required to configure an API key.\n- [x] Twitter can't constrain access as easily as they can to an API or an individual API key. Any constraints introduced would apply to their public site. A scraper can be fixed; you are no longer beholden to Twitter.\n- [x] Grab timelines, whole conversations, profiles, connections, likes, etc.\n- [x] Automatically pages to fetch all tweets.\n- [x] Provides metadata on how tweet replies are linked together. *e.g. `isReplyToId`*\n\n## Example\n\n### Get profile\n\n```sh\n$ scrape-twitter profile sebinsua\n# ...\n```\n\n### Get timeline\n\n```sh\n$ scrape-twitter timeline nouswaves\n# ...\n```\n\n### Get likes\n\nThis command requires a valid login. It will check for the following environment variables: `TWITTER_USERNAME`, `TWITTER_PASSWORD`, `TWITTER_KDT`. But can also pick these up from a [`dotenv`](https://github.com/motdotla/dotenv) file at the path `~/.scrape-twitter`. The first time you login you will be asked to store the `TWITTER_KDT` - this is used by Twitter to recognise your device.\n\n```sh\n$ scrape-twitter likes sebinsua\n# ...\n```\n\n### Get connections\n\nThis command also requires a valid login.\n\n```sh\n$ scrape-twitter connections sebinsua --type=following\n# ...\n```\n\n### Get conversation\n\n```sh\n$ scrape-twitter conversation ctbeiser 691766715835924484\n# ...\n```\n\n### Search\n\n```sh\n$ scrape-twitter search --query \"from:afoolswisdom motivation\" --type latest\n# ...\n```\n\n### Get list\n\n```sh\n$ scrape-twitter list nouswaves list\n# ...\n```\n\n#### JSON interface plays nicely with CLI tools like [`jq`](https://github.com/stedolan/jq), [`coreutils/gshuf`](https://github.com/wertarbyte/coreutils) and [`terminal-notifier`](https://github.com/julienXX/terminal-notifier)\n\nFor example, a [MOTD-like](https://en.wikipedia.org/wiki/Motd_(Unix)) script might contain:\n\n```sh\nscrape-twitter search --query=\"from:afoolswisdom knowledge\" | jq -r '.[].text' | gshuf -n 1 | terminal-notifier -title \"Knowledge (MOTD)\"\n```\n\n## Install\n\n*With `yarn`*:\n```sh\nyarn global add scrape-twitter\n```\n\n*With `npm`*:\n```sh\nnpm install -g scrape-twitter\n```\n\n## API\n\n### `new TimelineStream(username: string, { retweets: boolean, replies: boolean, count: ?number })`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the timeline of a `username`.\n\n### `new LikeStream(username: string, { count: ?number, env: process.env })`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the likes of a `username`.\n\n### `new ConnectionStream(username: string, type: 'following' | 'followers', process.env)`\n\nCreate a `ReadableStream\u003cUserConnection\u003e` for the connections of a `username`.\n\n### `new ConversationStream(username: string, id: string, { count: ?number })`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the conversation that belongs to a `username` and tweet `id`.\n\n### `new ThreadedConversationStream(id: string)`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the thread that belongs to a tweet `id`.\n\n### `new TweetStream(query: string, type: 'top' | 'latest', { count: ?number })`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the tweets that match a `query` and `type`.\n\n### `new ListStream(username: string, list: string, { count: ?number })`\n\nCreate a `ReadableStream\u003cTweet\u003e` for the `username`'s `list`.\n\n### `getUserProfile(username: string)`\n\nGet a `Promise\u003cUserProfile\u003e` for a particular `username`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebinsua%2Fscrape-twitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsebinsua%2Fscrape-twitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsebinsua%2Fscrape-twitter/lists"}