{"id":17645017,"url":"https://github.com/andefined/twitterfarm","last_synced_at":"2025-07-26T03:09:36.908Z","repository":{"id":57547607,"uuid":"90997899","full_name":"andefined/twitterfarm","owner":"andefined","description":"Quickly collect data from Twitter Streaming API","archived":false,"fork":false,"pushed_at":"2017-12-19T15:30:46.000Z","size":6259,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-07T03:46:13.521Z","etag":null,"topics":["cli","elasticsearch","go","golang","twitter"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andefined.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-05-11T15:56:45.000Z","updated_at":"2023-08-17T22:19:35.000Z","dependencies_parsed_at":"2022-09-26T18:40:58.142Z","dependency_job_id":null,"html_url":"https://github.com/andefined/twitterfarm","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/andefined/twitterfarm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andefined%2Ftwitterfarm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andefined%2Ftwitterfarm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andefined%2Ftwitterfarm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andefined%2Ftwitterfarm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andefined","download_url":"https://codeload.github.com/andefined/twitterfarm/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andefined%2Ftwitterfarm/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267110003,"owners_count":24037631,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","elasticsearch","go","golang","twitter"],"created_at":"2024-10-23T10:44:33.392Z","updated_at":"2025-07-26T03:09:36.881Z","avatar_url":"https://github.com/andefined.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/andefined/twitterfarm.svg?branch=master)](https://travis-ci.org/andefined/twitterfarm)\n[![Go Report Card](https://goreportcard.com/badge/github.com/andefined/twitterfarm)](https://goreportcard.com/report/github.com/andefined/twitterfarm)\n\n# twitterfarm\ntwitterfarm is a Twitter CLI tool written in [Go](https://golang.org/). The goal is to collect and store data from Twitter Streaming API into an Elasticsearch index fast and easy. Before you begin you must have Elasticsearch up \u0026 running and Twitter Application keys/secrets.\n\n- [Installing Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/5.x/install-elasticsearch.html)\n- [Twitter Applications](https://apps.twitter.com/)\n\n\n## Installation\nYou can download the binaries from the [releases](/releases) section, or you can install it with Go.\n\n```bash\ngo install github.com/andefined/twitterfarm\n```\n\n## How to use\n```\nNAME:\n   twitterfarm - Collect data from Twitter\n\nUSAGE:\n   twitterfarm [global options] command [command options] [arguments...]\n\nVERSION:\n   $VERSION\n\nCOMMANDS:\n     init     Initialize twitterfarm. Will create a folder under $HOME/.twitterfarm\n     create   Create a new project\n     list     List all projects\n     test     Test project configuration\n     rm       Remove a project\n     start    Start a project\n     stop     Stop a project\n     restart  Restart a project\n     exec     Execute a project\n     help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --help, -h     show help\n   --version, -v  print the version\n```\n\n## Initialize twitterfarm\nTo start using twitterfarm you need to create a working directory. The command simple creates a folder under `$HOME/.twitterfarm` where we store the configuration files for every project and needs to be run only once.\n\n```bash\ntwitterfarm init\n```\n\n## Create a project\nYou can create a project either by using the `--config` flag to load your custom [configuration](config/default.yml) or by providing individual flags. If successfully created it will return the project **ID**.\n\n```bash\ntwitterfarm create --config config/default.yml\n```\n\nOR\n\n```bash\ntwitterfarm create \\\n    --name \"us2016\" \\\n    --track \"trump,the giant douche,hillary,turd sandwich\" \\\n    --elasticsearch-host \"http://elastic:changeme@localhost:9200\" \\\n    --elasticsearch-index \"twitterfarm_trump_hillary\" \\\n    --consumer-key $TWITTER_CONSUMER_KEY \\\n    --consumer-secret $TWITTER_CONSUMER_SECRET \\\n    --access-token $TWITTER_ACCESS_TOKEN \\\n    --access-token-secret $TWITTER_ACCESS_TOKEN_SECRET\n```\n\n#### Sample configuration file\n\n```yaml\n# Project Name\nname: us2016\n\n# Streaming Specific (https://dev.twitter.com/streaming/overview/request-parameters)\ntrack: trump,the giant douche,hillary,turd sandwich\nfilter-level: none\nlanguage: en\nlocation: -126.56,22.88,-65.04,49.98\nfollow: \"\"\nstreaming-type: filter\nstall-warnings: false\n\n# Twitter Application Specific (https://apps.twitter.com/)\n# Consumer Key\nconsumer-key: TWITTER_CONSUMER_KEY\n# Consumer Secret\nconsumer-secret: TWITTER_CONSUMER_SECRET\n# Access Token\naccess-token: TWITTER_ACCESS_TOKEN\n # Access Token Secret\naccess-token-secret: TWITTER_ACCESS_TOKEN_SECRET\n\n# Elasticsearch Specific (https://www.elastic.co/)\n# Elasticsearch Host (ex. http://elastic:changeme@host-a:9200, http://elastic:changeme@host-b:9200)\nelasticsearch-host: http://elastic:changeme@localhost:9200\n# Elasticsearch Index\nelasticsearch-index: twitterfarm_trump_hillary\n```\n\n## List your projects\nWill return **ID**, **PID**, **STATUS**, **NAME** and **TRACK**.\n\n```bash\ntwitterfarm list\n\nID         | PID   | STATUS           | NAME             | TRACK\n           -       -                  -                  -\n215BC106C5 | 0     | not initialized  | us2016           | trump,the giant douch...\n247FBA4667 | 0     | not initialized  | us2016           | trump,the giant douch...\n```\n\nOR if you want to return only the **ID**.\n\n```bash\ntwitterfarm list -q\n\n215BC106C5\n247FBA4667\n```\n\n## Test a project\nTest command is very useful for testing the connection with Twitter Streaming API and your Elasticsearch. By providing the `--create-index` flag it will create the corresponding Elasticsearch Index (fi not exists).\n\n```bash\ntwitterfarm test $PROJECT_ID\n\nID           | TWITTER API  | ELASTIC HOST | ELASTIC INDEX\n             -              -              -\n28342C7FF7   | true         | false        | false\n```\n\n## Start a project\n```bash\ntwitterfarm start $PROJECT_ID\n```\n\n## Restart a project\n```bash\ntwitterfarm restart $PROJECT_ID\n```\n\n## Stop a project\n```bash\ntwitterfarm stop $PROJECT_ID\n```\n\n## Remove a project\nThe command will try to stop the project `proc.Kill()` and then will remove the configuration file from `$HOME/.twitterfarm/{{$PROJECT_ID}}.yml`. It will NOT remove any data from your elasticsearch index.\n\n```bash\ntwitterfarm rm $PROJECT_ID\n```\n\n## TO DO\n\n- [ ] Global Logger \u0026 Log Command\n- [ ] Twitter Streaming API Request Parameters\n- [ ] Twitter Streaming API User/Site\n- [ ] Elasticsearch Tweet Mappings\n- [ ] Before Save Pipe Script\n\n## Contributing\n1. Fork it\n2. Create your feature branch: `git checkout -b feature-name`\n3. Commit your changes: `git commit -am feature-name`\n4. Push to the branch: `git push origin feature-name`\n5. Submit a pull request\n\n## Credits\n- [urfave/cli](https://github.com/urfave/cli)\n- [olivere/elastic.v5](https://github.com/olivere/elastic/tree/v5.0.38)\n- [dghubble/go-twitter](https://github.com/dghubble/go-twitter)\n- [dghubble/oauth1](https://github.com/dghubble/oauth1)\n- [mitchellh/go-homedir](https://github.com/mitchellh/go-homedir)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandefined%2Ftwitterfarm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandefined%2Ftwitterfarm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandefined%2Ftwitterfarm/lists"}