{"id":23061792,"url":"https://github.com/rrwen/twitter2pg-cli","last_synced_at":"2026-04-12T18:45:19.237Z","repository":{"id":91307977,"uuid":"111503932","full_name":"rrwen/twitter2pg-cli","owner":"rrwen","description":"Command line tool for extracting Twitter data to PostgreSQL databases ","archived":false,"fork":false,"pushed_at":"2017-12-14T04:19:17.000Z","size":1236,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-11T06:54:43.655Z","etag":null,"topics":["api","cli","cmd","command","data","database","geo","interface","line","location","media","pg","postgres","postgresql","rest","social","stream","tool","tweet","twitter"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rrwen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-21T05:32:57.000Z","updated_at":"2023-02-14T02:53:52.000Z","dependencies_parsed_at":null,"dependency_job_id":"87def52f-314c-4af6-a5bf-ec99474d2029","html_url":"https://github.com/rrwen/twitter2pg-cli","commit_stats":{"total_commits":41,"total_committers":2,"mean_commits":20.5,"dds":"0.14634146341463417","last_synced_commit":"cc0552d0bfd74ce06387acecd214f1bc61106b10"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2pg-cli","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2pg-cli/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2pg-cli/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rrwen%2Ftwitter2pg-cli/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rrwen","download_url":"https://codeload.github.com/rrwen/twitter2pg-cli/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246952278,"owners_count":20859813,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","cli","cmd","command","data","database","geo","interface","line","location","media","pg","postgres","postgresql","rest","social","stream","tool","tweet","twitter"],"created_at":"2024-12-16T03:18:30.629Z","updated_at":"2026-04-12T18:45:14.152Z","avatar_url":"https://github.com/rrwen.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# twitter2pg-cli\n\nRichard Wen  \nrrwen.dev@gmail.com  \n  \nCommand line tool for extracting Twitter data to PostgreSQL databases\n\n[![npm version](https://badge.fury.io/js/twitter2pg-cli.svg)](https://badge.fury.io/js/twitter2pg-cli)\n[![Build Status](https://travis-ci.org/rrwen/twitter2pg-cli.svg?branch=master)](https://travis-ci.org/rrwen/twitter2pg-cli)\n[![npm](https://img.shields.io/npm/dt/twitter2pg-cli.svg)](https://www.npmjs.com/package/twitter2pg-cli)\n[![GitHub license](https://img.shields.io/github/license/rrwen/twitter2pg-cli.svg)](https://github.com/rrwen/twitter2pg-cli/blob/master/LICENSE)\n[![Twitter](https://img.shields.io/twitter/url/https/github.com/rrwen/twitter2pg-cli.svg?style=social)](https://twitter.com/intent/tweet?text=Command%20line%20tool%20for%20extracting%20Twitter%20data%20to%20PostgreSQL%20databases:%20https%3A%2F%2Fgithub.com%2Frrwen%2Ftwitter2pg-cli%20%23nodejs%20%23npm)\n\n## Install\n\n1. Install [Node.js](https://nodejs.org/en/)\n2. Install [twitter2pg-cli](https://www.npmjs.com/package/twitter2pg-cli) via `npm`\n\n```\nnpm install -g twitter2pg-cli\n```\n\nFor the latest developer version, see [Developer Install](#developer-install).\n\n## Usage\n\nGet help:\n\n```\ntwitter2pg --help\n```\n\nOpen documentation in web browser:\n\n```\ntwitter2pg doc twitter2pg\ntwitter2pg doc twitter\ntwitter2pg doc pg\n```\n\nSee [twitter2pg](https://www.npmjs.com/package/twitter2pg) for programmatic usage.\n\n### Environment File\n\nCreate a template `.env` file for Twitter and PostgreSQL details:\n\n* Edit this file with your [Twitter API credentials](https://apps.twitter.com/) and [PostgreSQL details](https://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html)\n\n```\ntwitter2pg file path/to/.env\n```\n\nSet default for the `.env` file:\n\n* Every `twitter2pg` command will now use the designated `.env` file\n\n```\ntwitter2pg set file path/to/.env\n```\n\n### PostgreSQL Query\n\nSend a query to a PostgreSQL database after defining and setting the default [Environment File](#environment-file).  \n  \nThe usage examples require a table named `twitter_data` which can be created with the command below:\n\n```\ntwitter2pg query \"CREATE TABLE twitter_data(tweets jsonb);\"\n```\n\nrow | tweets\n--- | ---\n1 | {...}\n2 | {...}\n3 | {...}\n... | ...\n\n### REST API\n\nSetup default twitter options:\n\n1. Set Twitter REST method (one of `get`, `post`, `delete` or `stream`)\n2. Set [Twitter path](https://developer.twitter.com/en/docs/api-reference-index)\n3. Set Twitter parameters for path\n\n```\ntwitter2pg set twitter.method get\ntwitter2pg set twitter.path search/tweets\ntwitter2pg set twitter.params \"{\\\"q\\\":\\\"twitter\\\"}\"\n```\n\nSetup default PostgreSQL options:\n\n1. Set table to store received Twitter data\n2. Set column to store received Twitter data\n3. Set [insert query](https://www.postgresql.org/docs/current/static/sql-insert.html) for received Twitter data\n4. Set [jsonata](https://www.npmjs.com/package/jsonata) filter before inserting\n\n```\ntwitter2pg set pg.table twitter_data\ntwitter2pg set pg.column tweets\ntwitter2pg set pg.query \"INSERT INTO $options.pg.table($options.pg.column) SELECT * FROM json_array_elements($1);\"\ntwitter2pg set jsonata statuses\n```\n\nExtract Twitter data into PostgreSQL table given setup options:\n\n```\ntwitter2pg \u003e log.csv\n```\n\n### Stream API\n\nSetup default twitter options:\n\n1. Set Twitter stream method\n2. Set Twitter path\n3. Set [Twitter stream parameters](https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter.html)\n\n```\ntwitter2pg set twitter.method stream\ntwitter2pg set twitter.path statuses/filter\ntwitter2pg set twitter.params \"{\\\"track\\\":\\\"twitter\\\"}\"\n```\n\nSetup default PostgreSQL options:\n\n1. Set table to store streamed Twitter data\n2. Set column to store streamed Twitter data\n3. Set [insert query](https://www.postgresql.org/docs/current/static/sql-insert.html) for streamed Twitter data\n4. Set [jsonata](https://www.npmjs.com/package/jsonata) filter before inserting\n\n```\ntwitter2pg set pg.table twitter_data\ntwitter2pg set pg.column tweets\ntwitter2pg set pg.query \"INSERT INTO $options.pg.table($options.pg.column) VALUES($1);\"\ntwitter2pg set jsonata statuses\n```\n\nStream Twitter data into PostgreSQL table given setup options:\n\n```\ntwitter2pg \u003e log.csv\n```\n\nStream Twitter data into a PostgreSQL table as a service:\n\n1. Save a [node](https://nodejs.org/api/cli.html) runnable script of the current options\n2. Install [pm2](https://www.npmjs.com/package/pm2) (`npm install pm2 -g`)\n2. Use `pm2`  to run the saved script as a service\n\n```\ntwitter2pg save path/to/script.js\npm2 start path/to/script.js\npm2 save\n```\n\n### Logs\n\nThe logs are in the following Comma-Separated Values (CSV) format:\n\n* `time_iso8601`: Time and date in [ISO 8601](https://www.iso.org/iso-8601-date-and-time-format.html) format\n* `status`: Status of the log\n* `message`: Relevant messages\n* `json`: [JSON](https://www.json.org/) object containing relevant debugging information\n\ntime_iso8601 | status | message | json\n--- | --- | --- | ---\n... | ... | ... | ...\n\n## Contributions\n\n### Report Contributions\n\nReports for issues and suggestions can be made using the [issue submission](https://github.com/rrwen/twitter2pg-cli/issues) interface.\n\nWhen possible, ensure that your submission is:\n\n* **Descriptive**: has informative title, explanations, and screenshots\n* **Specific**: has details of environment (such as operating system and hardware) and software used\n* **Reproducible**: has steps, code, and examples to reproduce the issue\n\n### Code Contributions\n\nCode contributions are submitted via [pull requests](https://help.github.com/articles/about-pull-requests/):\n\n1. Ensure that you pass the [Tests](#tests)\n2. Create a new [pull request](https://github.com/rrwen/twitter2pg-cli/pulls)\n3. Provide an explanation of the changes\n\nA template of the code contribution explanation is provided below:\n\n```\n## Purpose\n\nThe purpose can mention goals that include fixes to bugs, addition of features, and other improvements, etc.\n\n## Description\n\nThe description is a short summary of the changes made such as improved speeds or features, and implementation details.\n\n## Changes\n\nThe changes are a list of general edits made to the files and their respective components.\n* `file_path1`:\n    * `function_module_etc`: changed loop to map\n    * `function_module_etc`: changed variable value\n* `file_path2`:\n    * `function_module_etc`: changed loop to map\n    * `function_module_etc`: changed variable value\n\n## Notes\n\nThe notes provide any additional text that do not fit into the above sections.\n```\n\nFor more information, see [Developer Install](#developer-install) and [Implementation](#implementation).\n\n## Developer Notes\n\n### Developer Install\n\nInstall the latest developer version with `npm` from github:\n\n```\nnpm install -g git+https://github.com/rrwen/twitter2pg-cli\n```\n  \nInstall from `git` cloned source:\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Clone into current path\n3. Install via `npm`\n\n```\ngit clone https://github.com/rrwen/twitter2pg-cli\ncd twitter2pg-cli\nnpm -g install\n```\n\n### Tests\n\n1. Clone into current path `git clone https://github.com/rrwen/twitter2pg-cli`\n2. Enter into folder `cd twitter2pg-cli`\n3. Ensure [devDependencies](https://docs.npmjs.com/files/package.json#devdependencies) are installed and available\n4. Run tests with a `.env` file (see [tests/README.md](tests/README.md))\n5. Results are saved to [tests/log](tests/log) with each file corresponding to a version tested\n\n```\nnpm install\nnpm test\n```\n\n### Upload to Github\n\n1. Ensure [git](https://git-scm.com/) is installed\n2. Inside the `twitter2pg-cli` folder, add all files and commit changes\n3. Push to github\n\n```\ngit add .\ngit commit -a -m \"Generic update\"\ngit push\n```\n\n### Upload to npm\n\n1. Update the version in `package.json`\n2. Run tests and check for OK status\n3. Login to npm\n4. Publish to npm\n\n```\nnpm test\nnpm login\nnpm publish\n```\n\n### Implementation\n\nThe module [twitter2pg](https://www.npmjs.com/package/twitter2pg) uses the following [npm](https://www.npmjs.com/) packages for its implementation:\n\nnpm | Purpose\n--- | ---\n[yargs](https://www.npmjs.com/package/yargs) | Command line builder and parser\n[twitter2pg](https://www.npmjs.com/package/twitter2pg) | Extracts Twitter data to PostgreSQL\n[dotenv](https://www.npmjs.com/package/dotenv) | Load environmental variables from a file\n[opn](https://www.npmjs.com/package/opn) | Open online browser documentation\n[pg](https://www.npmjs.com/package/pg) | Send queries to PostgreSQL database\n\n```\nyargs\n   |--- twitter2pg   \u003c-- default command\n   |--- dotenv       \u003c-- file\n   |--- opn          \u003c-- doc\n   |--- pg           \u003c-- query\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2pg-cli","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frrwen%2Ftwitter2pg-cli","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frrwen%2Ftwitter2pg-cli/lists"}