https://github.com/mikeyburkman/elasticsearch-sender

Easier use of Elasticsearch
https://github.com/mikeyburkman/elasticsearch-sender

Last synced: about 5 hours ago
JSON representation

Easier use of Elasticsearch

• Host: GitHub
• URL: https://github.com/mikeyburkman/elasticsearch-sender
• Owner: MikeyBurkman
• License: mit
• Created: 2018-01-25T14:40:56.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2018-06-04T15:48:21.000Z (over 6 years ago)
• Last Synced: 2024-10-10T17:53:42.533Z (about 1 month ago)
• Language: JavaScript
• Size: 10.7 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Elasticsearch Sender

### Why?

Sending bulk data to Elasticsearch is common, but is not easy to do with the official ES client.

In particular, creating new indexes for time-based data is not trivial. In ES, it is very inefficient

to delete records by a query, and it is [recommended in the official docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/_deleting_documents.html) that you delete

by index instead. This of course requires you to keep creating and writing data to the correct index. This

library will handle that for you.

### Usage

```js

const esSend = require('elasticsearch-sender').buildSender({

    host: 'myElasticsearchHost.com',

    indexName: 'my-index',

    indexType: 'monthly',

    indexShape: {

        name: { type: 'keyword', index: true },

        numProcessed: { type: 'integer' },

        created: { type: 'date', index: true}

    },

    recordType: 'myRecordType'

});

...

esSend([{

    name: 'foo service',

    numProcessed: 42,

    created: new Date().toISOString()

}, {

    name: 'bar service',

    numProcessed: 2,

    created: new Date().toISOString() 

}])

.then(() => console.log('Items sent to Elasticsearch'));

```

### buildSender(opts)

Builds a sender function that will automatically handle bulk processing and indexing

##### Opts

- `host` Required. The Elasticsearch hostname

- `indexName` Required. Will be used when creating an index. Must be all lowercase.

- `indexShape` Required. An object describing indexes for the records being sent to ES. This is used for the `properties` field when creating the index in ES.

- `indexType` Required. Must be one of `'monthly'`, `'daily'`, or `'single'`. Controls the granularity of indexes. For instance, if `'daily'`, then a record created on Jan 25 2018 will be sent to the index named `'${indexPrefix}-2018-01-25'`. (This index will be automatically created as needed.) `'monthly'` will only have the year and month in the index name. `'single'` means only a single index called `${indexPrefix}` will be ever be created and used.

- `recordType` Required. The `_type` value for each record in Elasticsearch.

- `esLogLevel` Optional. The log level for the Elasticsearch client. Defaults to `'info'`

This returns a function that takes in an array of items and sends them all to Elasticsearch.

## Disclaimer

This library is designed for my personal common use cases, namely, sending medium amounts of small pieces of data to a hosted ES server, allowing for me to run metrics queries easily. It does not try to get into more advanced use cases.