{"id":20085347,"url":"https://github.com/saasify-sh/twitter-flock","last_synced_at":"2026-03-11T00:03:05.511Z","repository":{"id":38266281,"uuid":"272350521","full_name":"saasify-sh/twitter-flock","owner":"saasify-sh","description":"Simple \u0026 robust workflow automations for Twitter.","archived":false,"fork":false,"pushed_at":"2023-01-07T19:08:31.000Z","size":5706,"stargazers_count":15,"open_issues_count":15,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-06T01:38:03.795Z","etag":null,"topics":["automation","followers-twitter","saasify","twitter","twitter-api","workflow"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saasify-sh.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-15T05:32:57.000Z","updated_at":"2022-02-11T17:26:24.000Z","dependencies_parsed_at":"2023-02-07T22:15:16.783Z","dependency_job_id":null,"html_url":"https://github.com/saasify-sh/twitter-flock","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/saasify-sh/twitter-flock","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saasify-sh%2Ftwitter-flock","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saasify-sh%2Ftwitter-flock/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saasify-sh%2Ftwitter-flock/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saasify-sh%2Ftwitter-flock/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saasify-sh","download_url":"https://codeload.github.com/saasify-sh/twitter-flock/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saasify-sh%2Ftwitter-flock/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30362722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T21:41:54.280Z","status":"ssl_error","status_checked_at":"2026-03-10T21:40:59.357Z","response_time":106,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","followers-twitter","saasify","twitter","twitter-api","workflow"],"created_at":"2024-11-13T15:55:42.614Z","updated_at":"2026-03-11T00:03:05.476Z","avatar_url":"https://github.com/saasify-sh.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Twitter Flock\n\n\u003e Simple \u0026 robust workflows to export your flock of followers from Twitter.\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://saasify.sh\" title=\"Saasify\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/saasify-sh/twitter-flock/master/media/twitter.jpg\" alt=\"Twitter Flock\" /\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n[![Build Status](https://travis-ci.com/saasify-sh/twitter-flock.svg?branch=master)](https://travis-ci.com/saasify-sh/twitter-flock) [![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n\n## How it works\n\nAll automations are built around Twitter OAuth which gives us higher rate limits and access to private user actions like tweeting and sending DMs.\n\n### BatchJob\n\nThe core automation functionality is built around the [BatchJob](./lib/batch-job.js) class.\n\nThe goal of `BatchJob` is to ensure that potentially large batches of Twitter API calls are **serializable** and **resumable**.\n\nA `BatchJob` stores all of the state it would need to continue resolving its async batched operation in the event of an error. `BatchJob` instances support serializing their state in order to store them in a database of JSON file on disk.\n\nHere's an example batch job in action:\n\n```js\n// fetches the user ids for all of your followers\nconst job = BatchJobFactory.createBatchJobTwitterGetFollowers({\n  params: {\n    // assumes that you already have user oauth credentials\n    accessToken: twitterAccessToken,\n    accessTokenSecret: twitterAccessTokenSecret,\n\n    // only fetch your first 10 followers for testing purposes\n    maxLimit: 10,\n    count: 10\n  }\n})\n\n// process as much of this job as possible until it either completes or errors\nawait job.run()\n\n// job.status: 'active' | 'done' | 'error'\n// job.results: string[]\nconsole.log(job.status, job.results)\n\n// store this job to disk\nfs.writeFileSync('out.json', job.serialize())\n\n// ...\n\n// read the job from disk and resume processing\nconst jobData = fs.readFileSync('out.json')\nconst job = BatchJobFactory.deserialize(jobData)\n\nif (job.status === 'active') {\n  await job.run()\n}\n```\n\nThis example also shows how to serialize and resume a job.\n\n### Workflow\n\nSequences of `BatchJob` instances can be connected together to form a [Workflow](./lib/workflow.js).\n\nHere's an example workflow:\n\n```js\nconst job = new Workflow({\n  params: {\n    // assumes that you already have user oauth credentials\n    accessToken: twitterAccessToken,\n    accessTokenSecret: twitterAccessTokenSecret,\n    pipeline: [\n      {\n        type: 'twitter:get-followers',\n        label: 'followers',\n        params: {\n          // only fetch your first 10 followers for testing purposes\n          maxLimit: 10,\n          count: 10\n        }\n      },\n      {\n        type: 'twitter:lookup-users',\n        label: 'users',\n        connect: {\n          // connect the output of the first job to the `userIds` param for this job\n          userIds: 'followers'\n        },\n        transforms: ['sort-users-by-fuzzy-popularity']\n      },\n      {\n        type: 'twitter:send-direct-messages',\n        connect: {\n          // connect the output of the second job to the `users` param for this job\n          users: 'users'\n        },\n        params: {\n          // handlebars template with access to the current twitter user object\n          template: `Hey @{{user.screen_name}}, I'm testing an open source Twitter automation tool and you happen to be my one and only lucky test user.\\n\\nSorry for the spam. https://github.com/saasify-sh/twitter-flock`\n        }\n      }\n    ]\n  }\n})\n\nawait job.run()\n```\n\nThis workflow is comprised of three jobs:\n\n- `twitter:get-followers` - Fetches the user ids of all of your followers.\n  - Batches twitter API calls to [twitter followers/ids](https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids)\n- `twitter:lookup-users` - Expands these user ids into user objects.\n  - Batches twitter API calls to [users/lookup](https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup)\n- `twitter:send-direct-messages` - Sends a template-based direct message to each of these users.\n  - Batches twitter API calls to [direct_messages/events/new](https://developer.twitter.com/en/docs/direct-messages/sending-and-receiving/api-reference/new-event)\n\nNote that `Workflow` derives from `BatchJob`, so workflows are also serializable and resumable. Huzzah!\n\n## Future work\n\nA more robust, scalable version of this project would use a solution like [Apache Kafka](https://kafka.apache.org). [Kafka.js](https://kafka.js.org) looks useful as a higher-level Node.js wrapper.\n\nKafka would add quite a bit of complexity, but it would also handle a lot of details and be significantly more efficient. In particular, Kafka would solve the producer / consumer model, give us more robust error handling, horizontal scalability, storing and committing state, and enable easy interop with different data sources and sinks.\n\nThis project was meant as a quick prototype, however, and our relatively simple `BatchJob` abstraction works pretty well all things considered.\n\n### Producer / Consumer\n\nOne of the disadvantages of the current design is that a `BatchJob` needs to complete before any dependent jobs can run, whereas we'd really like to model this as a [Producer-Consumer problem](https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem).\n\n### DAGs\n\nAnother shortcoming of the current design is that `Workflows` can only combine sequences of jobs where the output of one job feeds into the input of the next job.\n\nA more extensible design would allow for workflows comprised of [directed acyclic graphs](https://en.wikipedia.org/wiki/Directed_acyclic_graph).\n\n## MVP TODO\n\n- [x] resumable batch jobs\n- [x] resumable workflows (sequences of batch jobs)\n- [x] twitter:get-followers batch job\n- [x] twitter:lookup-users batch job\n- [x] twitter:send-direct-messages batch job\n- [x] test workflow which combines these three batch jobs\n- [x] test rate limits\n  - twitter:get-followers 75k / 15 min\n  - twitter:lookup-users 90k / 15 min\n  - twitter:send-direct-messages 1k / day\n  - twitter:send-tweets 300 / 3h -\u003e 2.4k / day\n- [x] large account test\n- [x] gracefully handle twitter rate limits\n- [x] experiment with extracting public emails\n- [x] add default persistent storage\n  - via [leveldb](https://github.com/Level/level)\n- [x] support commiting batch job updates\n- [x] user-friendly cli\n- [x] add cli support for different output formats\n  - via [sheetjs/xlsx](https://github.com/SheetJS/sheetjs#supported-output-formats)\n  - json, csv, xls, xlsx, html, txt, etc\n- [x] gracefully handle process exit\n- [ ] initial set of cli commands\n- [ ] cli oauth support\n- [ ] unit tests for snapshotting, serializing, deserializing\n- [ ] unit tests for workflows\n- [ ] convert transforms to batchjob\n- [ ] more dynamic rate limit handling\n- [ ] support bring-your-own-api-key\n- [ ] basic docs and demo video\n- [ ] hosted saasify version\n\n## License\n\nMIT © [Saasify](https://saasify.sh)\n\nSupport my OSS work by \u003ca href=\"https://twitter.com/transitive_bs\"\u003efollowing me on twitter \u003cimg src=\"https://storage.googleapis.com/saasify-assets/twitter-logo.svg\" alt=\"twitter\" height=\"24px\" align=\"center\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaasify-sh%2Ftwitter-flock","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaasify-sh%2Ftwitter-flock","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaasify-sh%2Ftwitter-flock/lists"}