{"id":23061810,"url":"https://github.com/rrwen/twitter2mongodb-cli","last_synced_at":"2026-05-06T18:32:46.041Z","repository":{"id":57382930,"uuid":"114065184","full_name":"rrwen/twitter2mongodb-cli","owner":"rrwen","description":"Command line tool for extracting Twitter data to MongoDB databases","archived":false,"fork":false,"pushed_at":"2018-01-22T02:18:21.000Z","size":1232,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-24T20:43:28.561Z","etag":null,"topics":["api","cli","cmd","command","data","database","get","interface","line","mdb","media","mongo","mongod","mongodb","post","social","stream","tool","tweet","twitter"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rrwen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-13T02:47:07.000Z","updated_at":"2017-12-13T07:47:05.000Z","dependencies_parsed_at":"2022-09-11T04:10:23.050Z","dependency_job_id":null,"html_url":"https://github.com/rrwen/twitter2mongodb-cli","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rrwen","download_url":"https://codeload.github.com/rrwen/twitter2mongodb-cli/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246952278,"owners_count":20859813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","cli","cmd","command","data","database","get","interface","line","mdb","media","mongo","mongod","mongodb","post","social","stream","tool","tweet","twitter"],"created_at":"2024-12-16T03:18:33.224Z","updated_at":"2026-05-06T18:32:45.996Z","avatar_url":"https://github.com/rrwen.png","language":"JavaScript","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=NQNSAHK5X46D2"],"categories":[],"sub_categories":[],"readme":"# twitter2mongodb-cli\n\nRichard Wen  \nrrwen.dev@gmail.com  \n\nCommand line tool for extracting Twitter data to MongoDB databases\n\n[![npm version](https://badge.fury.io/js/twitter2mongodb-cli.svg)](https://badge.fury.io/js/twitter2mongodb-cli)\n[![Build Status](https://travis-ci.org/rrwen/twitter2mongodb-cli.svg?branch=master)](https://travis-ci.org/rrwen/twitter2mongodb-cli)\n[![npm](https://img.shields.io/npm/dt/twitter2mongodb-cli.svg)](https://www.npmjs.com/package/twitter2mongodb-cli)\n[![GitHub license](https://img.shields.io/github/license/rrwen/twitter2mongodb-cli.svg)](https://github.com/rrwen/twitter2mongodb-cli/blob/master/LICENSE)\n[![Donarbox Donate](https://img.shields.io/badge/donate-Donarbox-yellow.svg)](https://donorbox.org/rrwen)\n[![PayPal Donate](https://img.shields.io/badge/donate-PayPal-yellow.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=NQNSAHK5X46D2)\n[![Twitter](https://img.shields.io/twitter/url/https/github.com/rrwen/twitter2mongodb-cli.svg?style=social)](https://twitter.com/intent/tweet?text=Command%20line%20tool%20for%20extracting%20Twitter%20data%20to%20MongoDB%20databases:%20https%3A%2F%2Fgithub.com%2Frrwen%2Ftwitter2mongodb-cli%20%23nodejs%20%23npm)\n\n## Install\n\n1. Install [Node.js](https://nodejs.org/en/)\n2. Install [twitter2mongodb-cli](https://www.npmjs.com/package/twitter2mongodb-cli) via `npm`\n\n```\nnpm install -g twitter2mongodb-cli\n```\n\nFor the latest developer version, see [Developer Install](#developer-install).\n\n## Usage\n\nGet help:\n\n```\ntwitter2mongodb --help\n```\n\nOpen documentation in web browser:\n\n```\ntwitter2mongodb doc twitter2mongodb\ntwitter2mongodb doc twitter\ntwitter2mongodb doc mongodb\n```\n\nSee [twitter2mongodb](https://www.npmjs.com/package/twitter2mongodb) for programmatic usage.\n\n### Environment File\n\nAn environment file `.env` is used to store [Twitter API credentials](https://apps.twitter.com/) and [MongoDB details](https://docs.mongodb.com/manual/reference/connection-string/).  \n  \n**Step 1.** Set the default config for the `.env` file:\n\n* Every `twitter2mongodb` command will now use the designated `.env` file\n\n```\ntwitter2mongodb config set env path/to/.env\n```\n\n**Step 2.** Set [Twitter API credentials](https://apps.twitter.com/)\n\n```\ntwitter2mongodb env set TWITTER_CONSUMER_KEY ***\ntwitter2mongodb env set TWITTER_CONSUMER_SECRET ***\ntwitter2mongodb env set TWITTER_ACCESS_TOKEN_KEY ***\ntwitter2mongodb env set TWITTER_ACCESS_TOKEN_SECRET ***\n```\n\n**Step 3.** Set [MongoDB connection](https://docs.mongodb.com/manual/reference/connection-string/)\n\n```\ntwitter2mongodb env set MONGODB_CONNECTION mongodb://localhost:27017\n```\n\n### REST API\n\nThe REST API obtains Twitter data in batches using search queries.  \n  \n**Step 1.** Setup default twitter options:\n\n1. Set Twitter REST method (one of `get`, `post`, `delete` or `stream`)\n2. Set [Twitter path](https://developer.twitter.com/en/docs/api-reference-index)\n3. Set Twitter parameters for path\n\n```\ntwitter2mongodb config set twitter.method get\ntwitter2mongodb config set twitter.path search/tweets\ntwitter2mongodb config set twitter.params \"{\\\"q\\\":\\\"twitter\\\"}\"\n```\n\n**Step 2.** Setup default MongoDB options:\n\n1. Set database to store streamed Twitter data\n2. Set collection to store streamed Twitter data\n3. Set MongoDB [query method](https://mongodb.github.io/node-mongodb-native/3.0/api/Collection) for streamed Twitter data\n4. Set [jsonata](https://www.npmjs.com/package/jsonata) filter before inserting\n\n```\ntwitter2mongodb config set mongodb.database twitter2mongodb_database\ntwitter2mongodb config set mongodb.collection twitter_data\ntwitter2mongodb config set mongodb.method insertMany\ntwitter2mongodb config set jsonata statuses\n```\n\n**Step 3.** Extract Twitter data into MongoDB collection given setup options:\n\n```\ntwitter2mongodb \u003e log.csv\n```\n\n### Stream API\n\nThe Stream API obtains Twitter data in real-time using tracking filters.\n  \n**Step 1.** Setup default twitter options:\n\n1. Set Twitter stream method\n2. Set Twitter path\n3. Set [Twitter stream parameters](https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html)\n\n```\ntwitter2mongodb config set twitter.method stream\ntwitter2mongodb config set twitter.path statuses/filter\ntwitter2mongodb config set twitter.params \"{\\\"track\\\":\\\"twitter\\\"}\"\n```\n\n**Step 2.** Setup default MongoDB options:\n\n1. Set database to store streamed Twitter data\n2. Set collection to store streamed Twitter data\n3. Set MongoDB [query method](https://mongodb.github.io/node-mongodb-native/3.0/api/Collection) for streamed Twitter data\n\n```\ntwitter2mongodb config set mongodb.database twitter2mongodb_database\ntwitter2mongodb config set mongodb.collection twitter_data\ntwitter2mongodb config set mongodb.method insertOne\n```\n\n**Step 3a.** Stream Twitter data into MongoDB collection given setup options:\n\n```\ntwitter2mongodb \u003e log.csv\n```\n\n**Step 3b.** Stream Twitter data into a MongoDB collection as a service:\n\n1. Save a [node](https://nodejs.org/api/cli.html) runnable script of the current options\n2. Install [pm2](https://www.npmjs.com/package/pm2) (`npm install pm2 -g`)\n2. Use `pm2`  to run the saved script as a service\n\n```\ntwitter2mongodb save path/to/script.js\npm2 start path/to/script.js\npm2 save\n```\n\n### Logs\n\nThe logs are in the following Comma-Separated Values (CSV) format:\n\n* `time_iso8601`: Time and date in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format\n* `status`: Status of the log\n* `message`: Relevant messages\n* `json`: [JSON](https://www.json.org/) object containing relevant debugging information\n\ntime_iso8601 | status | message | json\n--- | --- | --- | ---\n... | ... | ... | ...\n\n## Contributions\n\n1. Reports for issues and suggestions can be made using the [issue submission](https://github.com/rrwen/twitter2mongodb-cli/issues) interface.\n2. Code contributions are submitted via [pull requests](https://github.com/rrwen/twitter2mongodb-cli/pulls)\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for more details.\n\n## Developer Notes\n\n### Developer Install\n\nInstall the latest developer version with `npm` from github:\n\n```\nnpm install git+https://github.com/rrwen/twitter2mongodb-cli\n```\n  \nInstall from `git` cloned source:\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Clone into current path\n3. Install via `npm`\n\n```\ngit clone https://github.com/rrwen/twitter2mongodb-cli\ncd twitter2mongodb-cli\nnpm install\n```\n\n### Tests\n\n1. Clone into current path `git clone https://github.com/rrwen/twitter2mongodb-cli`\n2. Enter into folder `cd twitter2mongodb-cli`\n3. Ensure [devDependencies](https://docs.npmjs.com/files/package.json#devdependencies) are installed and available\n4. Run tests with a `.env` file (see [tests/README.md](tests/README.md))\n5. Results are saved to [tests/log](tests/log) with each file corresponding to a version tested\n\n```\nnpm install\nnpm test\n```\n\n### Upload to Github\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Inside the `twitter2mongodb-cli` folder, add all files and commit changes\n3. Push to github\n\n```\ngit add .\ngit commit -a -m \"Generic update\"\ngit push\n```\n\n### Upload to npm\n\n1. Update the version in `package.json`\n2. Run tests and check for OK status\n3. Login to npm\n4. Publish to npm\n\n```\nnpm test\nnpm login\nnpm publish\n```\n\n### Implementation\n\nThe module [twitter2mongodb-cli](https://www.npmjs.com/package/twitter2mongodb-cli) uses the following [npm](https://www.npmjs.com/) packages for its implementation:\n\nnpm | Purpose\n--- | ---\n[path](https://nodejs.org/api/path.html) | Handle file and directory paths\n[fs](https://nodejs.org/api/fs.html) | Read and write config file\n[envfile](https://www.npmjs.com/package/envfile) | Parse and write env files\n[dotenv](https://www.npmjs.com/package/dotenv) | Load environmental variables from a file\n[yargs](https://www.npmjs.com/package/yargs) | Command line builder and parser\n[yargs-command-config](https://www.npmjs.com/package/yargs-command-config) | Command for managing config files\n[yargs-command-env](https://www.npmjs.com/package/yargs-command-env) | Command for managing env files\n[twitter2mongodb](https://www.npmjs.com/package/twitter2mongodb) | Extracts Twitter data to MongoDB\n[opn](https://www.npmjs.com/package/opn) | Open online browser documentation\n[mongodb](https://www.npmjs.com/package/mongodb) | Send queries to MongoDB database\n[parse-mongo-url](https://www.npmjs.com/package/parse-mongo-url) | Parse MongoDB urls\n\n```\n  path     \u003c-- Handle file and dir paths\n   |\n   fs      \u003c-- Read and write config file\n   |\n envfile   \u003c-- parse and write env file\n   |\n dotenv    \u003c-- load env file\n   |\n yargs\n   |--- yargs-command-config   \u003c-- manage config\n   |--- yargs-command-env      \u003c-- manage env\n   |--- twitter2mongodb        \u003c-- default command\n   |--- opn                    \u003c-- doc\n   |--- mongodb                \u003c-- query\n   |--- parse-mongo-url        \u003c-- parse MongoDB url for info\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2mongodb-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frrwen%2Ftwitter2mongodb-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2mongodb-cli/lists"}