{"id":23061808,"url":"https://github.com/rrwen/twitter2mongodb","last_synced_at":"2026-05-06T19:31:36.974Z","repository":{"id":57156910,"uuid":"113707973","full_name":"rrwen/twitter2mongodb","owner":"rrwen","description":"Module for extracting Twitter data to MongoDB databases","archived":false,"fork":false,"pushed_at":"2018-01-22T02:26:07.000Z","size":1274,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-21T22:38:04.815Z","etag":null,"topics":["api","data","database","geo","get","location","mdb","media","mongo","mongod","mongodb","oauth","post","rest","sample","social","stream","token","tweet","twitter"],"latest_commit_sha":null,"homepage":"https://rrwen.github.io/twitter2mongodb","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rrwen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-12-09T23:14:10.000Z","updated_at":"2022-08-21T10:50:05.000Z","dependencies_parsed_at":"2022-09-03T16:51:06.177Z","dependency_job_id":null,"html_url":"https://github.com/rrwen/twitter2mongodb","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2mongodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rrwen","download_url":"https://codeload.github.com/rrwen/twitter2mongodb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246678950,"owners_count":20816399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","data","database","geo","get","location","mdb","media","mongo","mongod","mongodb","oauth","post","rest","sample","social","stream","token","tweet","twitter"],"created_at":"2024-12-16T03:18:32.760Z","updated_at":"2026-05-06T19:31:36.934Z","avatar_url":"https://github.com/rrwen.png","language":"JavaScript","funding_links":["https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=NQNSAHK5X46D2"],"categories":[],"sub_categories":[],"readme":"# twitter2mongodb\n\nRichard Wen  \nrrwen.dev@gmail.com  \n\n* [Documentation](https://rrwen.github.io/twitter2mongodb)\n\nModule for extracting Twitter data to MongoDB databases\n\n[![npm version](https://badge.fury.io/js/twitter2mongodb.svg)](https://badge.fury.io/js/twitter2mongodb)\n[![Build Status](https://travis-ci.org/rrwen/twitter2mongodb.svg?branch=master)](https://travis-ci.org/rrwen/twitter2mongodb)\n[![Coverage Status](https://coveralls.io/repos/github/rrwen/twitter2mongodb/badge.svg?branch=master)](https://coveralls.io/github/rrwen/twitter2mongodb?branch=master)\n[![npm](https://img.shields.io/npm/dt/twitter2mongodb.svg)](https://www.npmjs.com/package/twitter2mongodb)\n[![GitHub license](https://img.shields.io/github/license/rrwen/twitter2mongodb.svg)](https://github.com/rrwen/twitter2mongodb/blob/master/LICENSE)\n[![Donarbox Donate](https://img.shields.io/badge/donate-Donarbox-yellow.svg)](https://donorbox.org/rrwen)\n[![PayPal Donate](https://img.shields.io/badge/donate-PayPal-yellow.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick\u0026hosted_button_id=NQNSAHK5X46D2)\n[![Twitter](https://img.shields.io/twitter/url/https/github.com/rrwen/twitter2mongodb.svg?style=social)](https://twitter.com/intent/tweet?text=Module%20for%20extracting%20Twitter%20data%20to%20MongoDB%20databases:%20https%3A%2F%2Fgithub.com%2Frrwen%2Ftwitter2mongodb%20%23nodejs%20%23npm)\n\n## Install\n\n1. Install [Node.js](https://nodejs.org/en/)\n2. Install [twitter2mongodb](https://www.npmjs.com/package/twitter2mongodb) via `npm`\n3. **Recommended**: Install [dotenv](https://www.npmjs.com/package/dotenv)\n\n```\nnpm install --save twitter2mongodb\nnpm install --save dotenv\n```\n\nFor the latest developer version, see [Developer Install](#developer-install).\n\n## Usage\n\nIt is recommended to use a `.env` file at the root of your project directory with the following contents:\n\n* Twitter keys can be obtained from https://apps.twitter.com/\n* `TWITTER_CONSUMER_KEY`: Consumer key (API Key)\n* `TWITTER_CONSUMER_SECRET`: Consumer secret (API secret)\n* ` TWITTER_ACCESS_TOKEN_KEY`: Access token\n* `TWITTER_ACCESS_TOKEN_SECRET`: Access token secret\n* `MONGODB_CONNECTION`: MongoDB [connection string](https://docs.mongodb.com/manual/reference/connection-string/)\n* `MONGODB_DATABASE`: MongoDB database name\n* `MONGODB_COLLECTION`: MongoDB collection name\n\n```\nTWITTER_CONSUMER_KEY=***\nTWITTER_CONSUMER_SECRET=***\nTWITTER_ACCESS_TOKEN_KEY=***\nTWITTER_ACCESS_TOKEN_SECRET=***\nMONGODB_CONNECTION=mongodb://localhost:27017\nMONGODB_DATABASE=test\nMONGODB_COLLECTION=twitter_data\n```\n\nThe `.env` file above can be loaded using [dotenv](https://www.npmjs.com/package/dotenv):\n\n```javascript\nrequire('dotenv').config();\n```\n\nSee [Documentation](https://rrwen.github.io/twitter2mongodb) for more details.\n\n### REST API\n\n1. Load `.env` file variables\n2. Load `twitter2mongodb`\n3. Create `options` object\n4. Optionally define Twitter API keys and MongoDB connection\n5. Search keyword `twitter` from `GET search/tweets`\n6. Apply a `jsonata` filter for statuses key only\n7. Execute `twitter2mongodb` with the REST API options\n\n```javascript\nrequire('dotenv').config();\n\nvar twitter2mongodb = require('twitter2mongodb');\n\n// (options) Initialize options object\nvar options = {\n\ttwitter: {},\n\tmongodb: {}\n};\n\n// (options_twitter_rest) Search for keyword 'twitter' in path 'GET search/tweets'\noptions.twitter.method = 'get'; // get, post, or stream\noptions.twitter.path = 'search/tweets'; // api path\noptions.twitter.params = {q: 'twitter'}; // query tweets\n\n// (options_jsonata) Filter for statuses array using jsonata\noptions.jsonata = 'statuses';\n\n// (options_mongodb) MongoDB options\noptions.mongodb.method = 'insertMany'; // insert many objects due to array\n\n// (twitter2mongodb_rest) Query tweets using REST API into MongoDB collection\ntwitter2mongodb(options)\n\t.then(data =\u003e {\n\t\tconsole.log(data);\n\t}).catch(err =\u003e {\n\t\tconsole.error(err.message);\n\t});\n```\n\n### Stream API\n\n1. Load `.env` file variables\n2. Load `twitter2mongodb`\n3. Create `options` object\n4. Optionally define Twitter API keys and MongoDB connection\n5. Track keyword `twitter` from `POST statuses/filter`\n6. Log the `tweets` when they are received\n7. Execute `twitter2return` with the Stream API options\n\n```javascript\nrequire('dotenv').config();\n\nvar twitter2mongodb = require('twitter2mongodb');\n\n// (options) Initialize options object\nvar options = {\n\ttwitter: {},\n\tmongodb: {}\n};\n\n// (options_twitter_connection) Track keyword 'twitter' in path 'POST statuses/filter'\noptions.twitter.method = 'stream'; // get, post, or stream\noptions.twitter.path = 'statuses/filter'; // api path\noptions.twitter.params = {track: 'twitter'}; // query tweets\n\n// (options_mongodb) MongoDB options\noptions.mongodb.method = 'insertOne';\noptions.mongodb.check = function(tweets) {return(true)}; // only insert if returns true\n\n// (twitter2mongodb_stream) Stream tweets into MongoDB collection\nvar stream = twitter2mongodb(options);\nstream.on('error', function(error) {\n\tconsole.error(error.message);\n});\n```\n\n## Contributions\n\n1. Reports for issues and suggestions can be made using the [issue submission](https://github.com/rrwen/twitter2mongodb/issues) interface.\n2. Code contributions are submitted via [pull requests](https://github.com/rrwen/twitter2mongodb/pulls)\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for more details.\n\n## Developer Notes\n\n### Developer Install\n\nInstall the latest developer version with `npm` from github:\n\n```\nnpm install git+https://github.com/rrwen/twitter2mongodb\n```\n  \nInstall from `git` cloned source:\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Clone into current path\n3. Install via `npm`\n\n```\ngit clone https://github.com/rrwen/twitter2mongodb\ncd twitter2mongodb\nnpm install\n```\n\n### Tests\n\n1. Clone into current path `git clone https://github.com/rrwen/twitter2mongodb`\n2. Enter into folder `cd twitter2mongodb`\n3. Ensure [devDependencies](https://docs.npmjs.com/files/package.json#devdependencies) are installed and available\n4. Run tests with a `.env` file (see [tests/README.md](tests/README.md))\n5. Results are saved to [tests/log](tests/log) with each file corresponding to a version tested\n\n```\nnpm install\nnpm test\n```\n\n### Documentation\n\nUse [documentationjs](https://www.npmjs.com/package/documentation) to generate html documentation in the `docs` folder:\n\n```\nnpm run docs\n```\n\nSee [JSDoc style](http://usejsdoc.org/) for formatting syntax.\n\n### Upload to Github\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Inside the `twitter2mongodb` folder, add all files and commit changes\n3. Push to github\n\n```\ngit add .\ngit commit -a -m \"Generic update\"\ngit push\n```\n\n### Upload to npm\n\n1. Update the version in `package.json`\n2. Run tests and check for OK status (see [tests/README.md](tests/README.md))\n3. Generate documentation\n4. Login to npm\n5. Publish to npm\n\n```\nnpm test\nnpm run docs\nnpm login\nnpm publish\n```\n\n### Implementation\n\nThe module [twitter2mongodb](https://www.npmjs.com/package/twitter2mongodb) uses the following [npm](https://www.npmjs.com/) packages for its implementation:\n\nnpm | Purpose\n--- | ---\n[twitter2return](https://www.npmjs.com/package/twitter2return) | Connections to the Twitter API REST and Streaming Application Programming Interfaces (APIs) using [twitter](https://www.npmjs.com/package/twitter), and Filters with [jsonata](https://www.npmjs.com/package/jsonata) before inserting into MongoDB\n[mongodb](https://www.npmjs.com/package/mongodb) | Insert Twitter data to MongoDB collections\n\n```\ntwitter2return   \u003c-- Extract Twitter data from API and Filter JSON data\n      |\n   mongodb       \u003c-- Insert filtered Twitter data into MongoDB collection\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frrwen%2Ftwitter2mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2mongodb/lists"}