{"id":13746830,"url":"https://github.com/elastic/elasticsearch-river-twitter","last_synced_at":"2025-04-06T06:12:40.071Z","repository":{"id":65978225,"uuid":"2917150","full_name":"elastic/elasticsearch-river-twitter","owner":"elastic","description":"Twitter River Plugin for elasticsearch (STOPPED)","archived":false,"fork":false,"pushed_at":"2024-08-08T07:14:19.000Z","size":383,"stargazers_count":204,"open_issues_count":0,"forks_count":71,"subscribers_count":359,"default_branch":"master","last_synced_at":"2025-03-30T05:06:40.607Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-12-05T14:47:05.000Z","updated_at":"2024-10-15T02:32:42.000Z","dependencies_parsed_at":"2024-01-13T01:39:59.987Z","dependency_job_id":"2cf18679-e64d-480e-91ec-b8ae4897da1c","html_url":"https://github.com/elastic/elasticsearch-river-twitter","commit_stats":{"total_commits":148,"total_committers":20,"mean_commits":7.4,"dds":"0.31081081081081086","last_synced_commit":"877ba075aae0100cf05f39d86d0dad6475996489"},"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-twitter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-twitter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-twitter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Felasticsearch-river-twitter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/elasticsearch-river-twitter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247441059,"owners_count":20939239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T06:01:02.499Z","updated_at":"2025-04-06T06:12:40.052Z","avatar_url":"https://github.com/elastic.png","language":"Java","funding_links":[],"categories":["Java"],"sub_categories":[],"readme":"**Important**: This project has been stopped since elasticsearch 2.0.\n\n----\n\nTwitter River Plugin for Elasticsearch\n==================================\n\nThe Twitter river indexes the public [twitter stream](http://dev.twitter.com/pages/streaming_api), aka the hose,\nand makes it searchable.\n\n**Rivers are [deprecated](https://www.elastic.co/blog/deprecating_rivers) and will be removed in the future.**\nHave a look at [logstash twitter input](http://www.elastic.co/guide/en/logstash/current/plugins-inputs-twitter.html).\n\nIn order to install the plugin, run: \n\n```sh\nbin/plugin install elasticsearch/elasticsearch-river-twitter/2.6.0\n```\n\nAfter installing the plugin you need to restart elasticsearch.\n\nYou need to install a version matching your Elasticsearch version:\n\n|       Elasticsearch    |Twitter River Plugin|                                                            Docs                                                                   |\n|------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------|\n|    master              | Build from source | See below                                                                                                                          |\n|    es-1.x              | Build from source  | [2.7.0-SNAPSHOT](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/es-1.x/#version-270-snapshot-for-elasticsearch-1x)|\n|    es-1.6              |     2.6.0         | [2.6.0](https://github.com/elastic/elasticsearch-river-twitter/tree/v2.6.0/#version-260-for-elasticsearch-16)                  |\n|    es-1.5              |     2.5.0         | [2.5.0](https://github.com/elastic/elasticsearch-river-twitter/tree/v2.5.0/#version-250-for-elasticsearch-15)                  |\n|    es-1.4              |     2.4.2         | [2.4.2](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/v2.4.2/#version-242-for-elasticsearch-14)                  |\n|    es-1.3              |     2.3.0         | [2.3.0](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/v2.3.0/#version-230-for-elasticsearch-13)                  |\n|    es-1.2              |     2.2.0         | [2.2.0](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/v2.2.0/#twitter-river-plugin-for-elasticsearch)          |\n|    es-1.0              |     2.0.0         | [2.0.0](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/v2.0.0/#twitter-river-plugin-for-elasticsearch)          |\n|    es-0.90             |     1.5.0         | [1.5.0](https://github.com/elasticsearch/elasticsearch-river-twitter/tree/v1.5.0/#twitter-river-plugin-for-elasticsearch)          |\n\nTo build a `SNAPSHOT` version, you need to build it with Maven:\n\n```bash\nmvn clean install\nplugin --install river-twitter \\ \n       --url file:target/releases/elasticsearch-river-twitter-X.X.X-SNAPSHOT.zip\n```\n\nPrerequisites\n-------------\n\nYou need to get an OAuth token in order to use Twitter river.\nPlease follow [Twitter documentation](https://dev.twitter.com/docs/auth/tokens-devtwittercom), basically:\n\n* Login to: https://dev.twitter.com/apps/\n* Create a new Twitter application (let's say elasticsearch): https://dev.twitter.com/apps/new\nYou don't need a callback URL.\n* When done, click on `Create my access token`.\n* Open `OAuth tool` tab and note `Consumer key`, `Consumer secret`, `Access token` and `Access token secret`.\n\n\nCreate river\n------------\n\nCreating the twitter river can be done using:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"oauth\" : {\n            \"consumer_key\" : \"*** YOUR Consumer key HERE ***\",\n            \"consumer_secret\" : \"*** YOUR Consumer secret HERE ***\",\n            \"access_token\" : \"*** YOUR Access token HERE ***\",\n            \"access_token_secret\" : \"*** YOUR Access token secret HERE ***\"\n        }\n    },\n    \"index\" : {\n        \"index\" : \"my_twitter_river\",\n        \"type\" : \"status\",\n        \"bulk_size\" : 100,\n        \"flush_interval\" : \"5s\",\n        \"retry_after\" : \"10s\"\n    }\n}\n```\n\nThe above lists all the options controlling the creation of a twitter river.\n\nIf you don't define `index.index`, it will use your river name (`my_twitter_river`) as the default index name.\nIf you don't define `index.type`, default `status` type will be used. \n\nNote that you can define any or all of your oauth settings in `elasticsearch.yml` file on each node by prefixing \nsetting with `river.twitter.`:\n\n```\nriver.twitter.oauth.consumer_key: \"*** YOUR Consumer key HERE ***\"\nriver.twitter.oauth.consumer_secret: \"*** YOUR Consumer secret HERE ***\"\nriver.twitter.oauth.access_token: \"*** YOUR Access token HERE ***\"\nriver.twitter.oauth.access_token_secret: \"*** YOUR Access token secret HERE ***\"\n```\n\nIn that case, you can create the river using:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\"\n}\n```\n\nYou can also overload any of `elasticsearch.yml` setting. A good practice could be to have `consumer_key` and \n`consumer_secret` in `elasticsearch.yml` and provide to the river `access_token` and `access_token_secret` properties.\n\nBy default, the twitter river will read a small random of all public statuses using\n[sample API](https://dev.twitter.com/docs/api/1.1/get/statuses/sample).\n\nBut, you can define statuses type you want to read:\n\n* [sample](https://dev.twitter.com/docs/api/1.1/get/statuses/sample): the default one\n* [filter](https://dev.twitter.com/docs/api/1.1/post/statuses/filter): track for text, users and locations.\nSee [Filtered Stream](#filtered-stream)\n* [user](https://dev.twitter.com/docs/streaming-apis/streams/user): listen to tweets in the authenticated user's timeline.\nSee [User Stream](#user-stream)\n* [firehose](https://dev.twitter.com/docs/api/1.1/get/statuses/firehose): all public statuses (restricted access)\n\nFor example:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"type\" : \"firehose\"\n    }\n}\n```\n\nNote that if you define a filter (see [next section](#filtered-stream)), type will be automatically set to `filter`.\n\nTweets will be indexed once a `bulk_size` of them have been accumulated (default to `100`)\nor every `flush_interval` period (default to `5s`).\n\nFiltered Stream\n===============\n\nFiltered stream can also be supported (as per the twitter stream API). Filter stream can be configured to\nsupport `tracks`, `follow`, `locations` and `language`. `user_lists` is a shortcut to follow all members of a public \ntwitter list identified by the user id and the list slug (last part of uri when open a list in your browser). The \nconfiguration is the same as the twitter API (a single comma separated string value, or using json arrays). \nHere is an example:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"filter\" : {\n            \"tracks\" : \"test,something,please\",\n            \"follow\" : \"111,222,333\",\n            \"user_lists\" : \"ownerScreenName1/slug1,ownerScreenName2/slug2\",\n            \"locations\" : \"-122.75,36.8,-121.75,37.8,-74,40,-73,41\",\n            \"language\" : \"fr,en\"\n        }\n    }\n}\n```\n\nNote that locations use geoJSON order (longitude, latitude).\n\nNote that if you want to use language filtering you need also to define at least one of `tracks`,\n`follow` or `locations` filter.\nSupported languages identifiers are [BCP 47](http://tools.ietf.org/html/bcp47). You can filter\nwhatever language defined in [Twitter Advanced Search](https://twitter.com/search-advanced).\n\nHere is an array based configuration example:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"filter\" : {\n            \"tracks\" : [\"test\", \"something\"],\n            \"follow\" : [111, 222, 333],\n            \"locations\" : [ [-122.75,36.8], [-121.75,37.8], [-74,40], [-73,41]],\n            \"language\" : [ \"fr\", \"en\" ]\n        }\n    }\n}\n```\n\nUser Stream\n===========\n\nUser stream can also be supported (as per the twitter stream API). This stream return tweets on the authenticated user's\ntimeline. Here is a basic configuration example:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"type\" : \"user\"\n    }\n}\n```\n\nIndexing RAW Twitter stream\n===========================\n\nBy default, elasticsearch twitter river will convert tweets to an equivalent representation\nin elasticsearch. If you want to index RAW twitter JSON content without any transformation,\nyou can set `raw` to `true`:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"raw\" : true\n    }\n}\n```\n\nNote that you should think of creating a mapping first for your tweets. See Twitter documentation on\n[raw Tweet format](https://dev.twitter.com/docs/platform-objects/tweets):\n\n```\nPUT my_twitter_river/status/_mapping\n{\n    \"status\" : {\n        \"properties\" : {\n            \"text\" : {\"type\" : \"string\", \"analyzer\" : \"standard\"}\n        }\n    }\n}\n```\n\nIgnoring Retweets\n=================\n\nIf you don't want to index retweets (aka RT), just set `ignore_retweet` to `true` (default to `false`):\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"ignore_retweet\" : true\n    }\n}\n```\n\nIncrease the schedule time to reconnect the river\n=================================================\n\nIt can happen that the river fails, thus closing the current connection to the Streaming API. Then, a new connection is scheduled by the river after 10s by default.\nIf you want to manage this time, simply use the `retry_after` option, as in:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"index\" : {\n        \"retry_after\" : \"30s\"\n    }\n}\n```\n\nGeo location points as array\n============================\n\nBy default, elasticsearch twitter river index `location` field using the *lat lon as properties* format.\nYou can set `geo_as_array` to `true` if you prefer having `location` indexed as an array `[lon, lat]`.\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"geo_as_array\" : true\n    }\n}\n```\n\nRemove the river\n================\n\nIf you need to stop the Twitter river, you have to remove it:\n\n```\nDELETE _river/my_twitter_river/\n```\n\nUsing a proxy\n=============\n\nYou can define a proxy if you are using one:\n\n```\nPUT _river/my_twitter_river/_meta\n{\n    \"type\" : \"twitter\",\n    \"twitter\" : {\n        \"proxy\" : {\n            \"host\": \"host\",\n            \"port\": \"port\",\n            \"user\": \"proxy_user_if_any\",\n            \"password\": \"proxy_password_if_any\"\n        }\n    }\n}\n```\n\nYou can also define proxy settings in `elasticsearch.yml`file on each node by prefixing setting with `river.twitter.`:\n                                                         \n```yaml\nriver.twitter.proxy.host: \"host\"\nriver.twitter.proxy.port: \"port\"\nriver.twitter.proxy.user: \"proxy_user_if_any\"\nriver.twitter.proxy.password: \"proxy_password_if_any\"\n```\n\nSample document\n===============\n\nHere is how a document could look like when using this river (without `raw` option):\n\n```js\n{\n   \"text\":\"This is a text\",\n   \"created_at\":\"2015-01-26T15:22:35.000Z\",\n   \"source\":\"\u003ca href=\\\"http://www.twitter.com\\\" rel=\\\"nofollow\\\"\u003eTwitter for Windows Phone\u003c/a\u003e\",\n   \"truncated\":false,\n   \"language\":\"en\",\n   \"mention\":[\n\n   ],\n   \"retweet_count\":0,\n   \"hashtag\":[\n\n   ],\n   \"location\":[\n      78.418407,\n      17.431913\n   ],\n   \"place\":{\n      \"id\":\"243cc16f6417a167\",\n      \"name\":\"Hyderabad\",\n      \"type\":\"city\",\n      \"full_name\":\"Hyderabad, Andhra Pradesh\",\n      \"street_address\":null,\n      \"country\":\"India\",\n      \"country_code\":\"IN\",\n      \"url\":\"https://api.twitter.com/1.1/geo/id/243cc16f6417a167.json\"\n   },\n   \"link\":[\n\n   ],\n   \"user\":{\n      \"id\":1111111111,\n      \"name\":\"User Name\",\n      \"screen_name\":\"twitter_handle\",\n      \"location\":\"A full text location description\",\n      \"description\":\"A description\",\n      \"profile_image_url\":\"http://pbs.twimg.com/profile_images/1111111111/QATJ00Yp_normal.jpeg\",\n      \"profile_image_url_https\":\"https://pbs.twimg.com/profile_images/1111111111/QATJ00Yp_normal.jpeg\"\n   }\n}\n```\n\nTests\n=====\n\nIntegrations tests in this plugin require working Twitter account and therefore disabled by default. \nYou need to create your credentials as explained in [Prerequisites](#prerequisites).\n\nTo enable tests prepare a config file `elasticsearch.yml` with the following content:\n\n```\nriver:\n  twitter:\n      oauth:\n         consumer_key: \"your_consumer_key\"\n         consumer_secret: \"your_consumer_secret\"\n         access_token: \"your_access_token\"\n         access_token_secret: \"your_access_token_secret\"\n```\n\nReplace all occurrences of `your_consumer_key`, `your_consumer_secret`, `your_access_token` and \n`your_access_token_secret` with your settings.\n\nTo run test:\n\n```sh\nmvn -Dtests.twitter=true -Dtests.config=/path/to/config/file/elasticsearch.yml clean test\n```\n\nNote that if you want to test User Stream, you need to define write rights for your twitter \napplication.\n\nLicense\n-------\n\n    This software is licensed under the Apache 2 license, quoted below.\n\n    Copyright 2009-2014 Elasticsearch \u003chttp://www.elasticsearch.org\u003e\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-twitter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Felasticsearch-river-twitter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Felasticsearch-river-twitter/lists"}