{"id":13442410,"url":"https://github.com/elastic/stream2es","last_synced_at":"2025-09-29T23:31:02.195Z","repository":{"id":6678927,"uuid":"7923887","full_name":"elastic/stream2es","owner":"elastic","description":"Stream data into ES (Wikipedia, Twitter, stdin, or other ESes)","archived":true,"fork":false,"pushed_at":"2018-10-29T20:46:44.000Z","size":312,"stargazers_count":355,"open_issues_count":26,"forks_count":62,"subscribers_count":366,"default_branch":"master","last_synced_at":"2024-09-22T23:04:42.555Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elastic.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-30T21:08:38.000Z","updated_at":"2024-03-31T14:10:44.000Z","dependencies_parsed_at":"2022-08-28T04:00:27.023Z","dependency_job_id":null,"html_url":"https://github.com/elastic/stream2es","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fstream2es","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fstream2es/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fstream2es/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elastic%2Fstream2es/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elastic","download_url":"https://codeload.github.com/elastic/stream2es/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219874706,"owners_count":16554610,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:01:45.399Z","updated_at":"2025-09-29T23:30:56.868Z","avatar_url":"https://github.com/elastic.png","language":"Clojure","funding_links":[],"categories":["Clojure"],"sub_categories":[],"readme":"# stream2es\n\nStandalone utility to stream different inputs into Elasticsearch.\n\n## Read This First\n\n*If you've just wandered here, first check out [Logstash](http://github.com/elasticsearch/logstash).  It's a much more general tool, and one of our featured products.  If for some reason it doesn't do something that's important to you, create an issue there.  stream2es is a dev tool that originated before the author knew much about Logstash.  That said, there are some important differences that are specific to Elasticsearch.  stream2es supports bulks by byte-length (`--bulk-bytes`) instead of doc count, which is crucial with docs of varying size.  It also supports exporting raw bulks via `--tee-bulk` to a hashed dir on the filesystem, and you can make the incoming stream finite with `--max-docs`.*\n\n## Install\n\nYou'll need Java 8+.  Run `java -version` to make sure.\n\n### Unix\n\nDownload `stream2es` and make it executable:\n\n```\n% curl -O download.elasticsearch.org/stream2es/stream2es; chmod +x stream2es\n```\n\n### Windows\n\n```\n\u003e curl -O download.elasticsearch.org/stream2es/stream2es\n\u003e java -jar stream2es help\n```\n\n\n\n# Usage\n\n## stdin\n\nBy default, `stream2es` reads JSON documents from stdin.\n\n```\n% echo '{\"f\":1}' | stream2es\n2014-10-08T12:29:56.318-0500 INFO  00:00.116 8.6d/s 0.4K/s (0.0mb) indexed 1 streamed 1 errors 0\n%\n```\n\nIf you want more logging, set `--log debug`.  If you don't want any output, set `--log warn`.\n\n## Wikipedia\n\nIndex the latest Wikipedia article dump.\n\n    % stream2es wiki --target http://localhost:9200/tmp --log debug\n    create index http://localhost:9200/tmp\n    stream wiki from http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2\n    ^Cstreamed 1158 docs 1082 bytes xfer 15906901 errors 0\n\nIf you're at a café or want to use a local copy of the dump, supply `--source`:\n\n    % ./stream2es wiki --max-docs 5 --source /d/data/enwiki-20121201-pages-articles.xml.bz2\n\nNote that if you live-stream the WMF-hosted dump, it will cut off after a while. Grab a torrent and index it locally if you need more than a few thousand docs.\n\n## Generator\n\n`stream2es` can fuzz data for you.  It can create blank documents, or documents with integer fields, or documents with string fields if you supply a dictionary.\n\nBlank documents are easy:\n\n```\nstream2es generator\n```\n\nInts need to know how big you want them.  This template would give you a single field with values between `0` and `127`, inclusive.\n\n```\nstream2es generator --fields f1:int:128\n```\n\nTo add a string, we need to add a template for it, and a file of newline-separated lines of text.  Given a field template of `NAME:str:N`, `stream2es` will select `N` random words from the dictionary for each field.\n\n```\n# zsh\n% stream2es generator --fields f1:int:128,f2:str:2 --dictionary \u003c(/bin/echo -e \"foo\\nbar\\nbaz\")\n#### same as:\n% stream2es generator --fields f1:int:128,f2:str:2 --dictionary /dev/stdin --max-docs 5 \u003c\u003cEOF\nfoo\nbar\nbaz\nEOF\n% curl -s localhost:9200/foo/_search\\?format=yaml | fgrep -A2 _source\n    _source:\n      f1: 28\n      f2: \"foo baz\"\n--\n    _source:\n      f1: 88\n      f2: \"baz foo\"\n--\n    _source:\n      f1: 26\n      f2: \"baz baz\"\n--\n    _source:\n      f1: 68\n      f2: \"bar baz\"\n--\n    _source:\n      f1: 64\n      f2: \"foo foo\"\n%\n```\n\nFortunately, most *nix systems come with `/usr/share/dict/words` (Ubuntu package `wamerican-small`, for example), which is a great choice if you just need some (English) text.  Install other langs if you prefer.\n\n\n## Elasticsearch\n\n*Note: ES 2.3 added a [reindex API](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/docs-reindex.html) that completely obviates this feature of stream2es.  Also, Logstash 1.5.0 has an [Elasticsearch input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html).*\n\nIf you use the `es` stream, you can copy indices from one Elasticsearch to another.  Example:\n\n    % stream2es es \\\n         --source http://foo.local:9200/wiki \\\n         --target http://bar.local:9200/wiki2\n\nThis is a convenient way to reindex data if you need to change the number of shards or update your mapping.\n\n## Twitter\n\nIn order to stream Twitter, you have to create an app and authorize it.\n\n### Create app\n\nVisit (https://dev.twitter.com/apps/new) and create an app.  Call it `stream2es`.  Note the `Consumer key` and `Consumer secret`.\n\n### Authorize app\n\nNow run `stream2es twitter --authorize --key CONSUMER_KEY --secret CONSUMER_SECRET` and complete the dialog.\n\n### Run with new creds\n\nYou should now be able to stream twitter with simply `stream2es twitter`.  stream2es will grab the most recent cached credentials from `~/.authinfo.stream2es`.\n\n### Tracking keywords\n\nBy default, stream2es will stream random sample of all public tweets, however\nyou can configure stream2es to _track_ specific keywords as follows:\n\n    \tstream2es twitter --track \"Linux%%New York%%March Madness\"\n\n# Options\n\n    % stream2es --help\n    Copyright 2013 Elasticsearch\n\n    Usage: stream2es [CMD] [OPTS]\n\n    ..........\n\n\nYou can change index settings by supplying `--settings`:\n\n    % echo '{\"name\":\"alfredo\"}' | ./stream2es stdin --settings '\n    {\n        \"settings\": {\n            \"refresh_interval\": \"2m\"\n        }\n    }'\n\n# Contributing\n\nstream2es is written in Clojure.  You'll need leiningen 2.0+ to build.\n\n    % lein bin\n    % target/stream2es\n\nYou'll also need this little git alias if you want to do `make`:\n\n```\n[alias]\nver = \"!git log --pretty=format:'%ai %h' -1 | perl -pe 's,(\\\\d\\\\d\\\\d\\\\d)-(\\\\d\\\\d)-(\\\\d\\\\d) (\\\\d\\\\d):(\\\\d\\\\d):(\\\\d\\\\d) [^ ]+ ([a-z0-9]+),\\\\1\\\\2\\\\3\\\\7,'\"\n```\n\n# License\n\nThis software is licensed under the Apache 2 license, quoted below.\n\n    Copyright 2009-2013 Elasticsearch \u003chttp://www.elasticsearch.org\u003e\n\n    Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\n    use this file except in compliance with the License. You may obtain a copy of\n    the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n    License for the specific language governing permissions and limitations under\n    the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fstream2es","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felastic%2Fstream2es","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felastic%2Fstream2es/lists"}