https://github.com/nymag/elasticsearch-streamer

Command-line interface for exporting and importing Elastic docs
https://github.com/nymag/elasticsearch-streamer

Last synced: 4 months ago
JSON representation

Command-line interface for exporting and importing Elastic docs

• Host: GitHub
• URL: https://github.com/nymag/elasticsearch-streamer
• Owner: nymag
• Created: 2018-02-28T18:48:58.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2023-10-13T22:18:30.000Z (about 1 year ago)
• Last Synced: 2024-08-07T03:30:41.714Z (4 months ago)
• Language: JavaScript
• Size: 57.6 KB
• Stars: 1
• Watchers: 33
• Forks: 0
• Open Issues: 2
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# elasticsearch streamer

A programmatic and command-line utility for streaming docs from Elasticsearch indices and putting docs into indices in bulk.

## Installation

`npm install -g elasticsearch-streamer`

## Usage

Save all docs in Index A to a file:

    ess get localhost:9200/a > b.txt

Put all docs in a file into Index C:

    

    ess put localhost:9200/c < b.txt

Moves docs from Index A to Index B:

    

    ess get localhost:9200/a | ess put localhost:9200/b

Move docs from host1's index A to host2's index A:

    ess get host1:9200/a | ess put host2:9200

Move docs from host1's indices into the same indices at host2:

    ess get host1:9200 | ess put host2:9200

Stream all docs where `foo` is `bar`:

    ess get localhost:9200/a --search '{"body": {"query": {"bool": {"filter": {"term": {"foo": "bar"}}}}}}'

## Commands

### get

Streams documents out of the specified Elastic index endpoint as JSON.

Options:

* `--search`: Client search options as JSON, e.g. `{"body": {"query": {"bool": {"filter": {"term": {"foo": "bar"}}}}}}`. Passed directly to `client.search`; see [ElasticSearch's Javascript API docs](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#api-search). `scroll` defaults to `10s` and `size` to `100`.

* `--timeout`: Set the timeout for the Elastic client (in ms).

### put

Put documents (as JSON strings) from stdin into the specified Elastic index endpoint, and stream bulk results objects (JSON strings).

Options:

* `--batch`: size of bulk batches. Default: `100`.

* `--timeout`: Set the timeout for the Elastic client (in ms).

## Tips

Use with [jq](https://stedolan.github.io/jq/) to parse and filter JSON.