https://github.com/codelibs/elasticsearch-indexing-proxy
This plugin intercepts indexing requests and writes them to files
https://github.com/codelibs/elasticsearch-indexing-proxy
elasticsearch-plugin
Last synced: 8 months ago
JSON representation
This plugin intercepts indexing requests and writes them to files
- Host: GitHub
- URL: https://github.com/codelibs/elasticsearch-indexing-proxy
- Owner: codelibs
- License: apache-2.0
- Created: 2017-08-16T05:09:41.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2021-12-14T20:32:22.000Z (about 4 years ago)
- Last Synced: 2025-03-30T18:04:33.819Z (9 months ago)
- Topics: elasticsearch-plugin
- Language: Java
- Homepage:
- Size: 137 KB
- Stars: 6
- Watchers: 7
- Forks: 3
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Elasticsearch Indexing Proxy Plugin
===================================
## Overview
This plugin provides a proxy feature for target indices.
For target indices, this plugin intercepts indexing requests and writes them to files.
Moreover, Document Sender provided by this plugin reads the files and sends them to any indices.
## Version
[Versions in Maven Repository](http://central.maven.org/maven2/org/codelibs/elasticsearch-indexing-proxy/)
### Issues/Questions
Please file an [issue](https://github.com/codelibs/elasticsearch-indexing-proxy/issues "issue").
## Installation
Not released yet.
$ $ES_HOME/bin/elasticsearch-plugin install org.codelibs:elasticsearch-indexing-proxy:5.5.0
## Getting Started
### Configuration
This plugin has the following settings.
| Setting | Type | Default | Description |
|:-------------------------------------|:-------:|:------------:|:------------------------------|
| idxproxy.data.path | string | es.data.path | directory to store data file |
| idxproxy.target.indices | string | none | target indices |
| idxproxy.data.file\_size | size | 100m | file size |
| idxproxy.sender.interval | time | 30s | interval to check file |
| idxproxy.sender.retry\_count | int | 10 | retry count for error file |
| idxproxy.sender.request.retry\_count | int | 3 | retry count for error request |
| idxproxy.sender.skip.error\_file | boolean | true | skip error file |
For example, put settings as below:
```
idxproxy.data.path: /opt/elasticsearch/idxproxy-data
idxproxy.target.indices: sample
```
The above settings intercepts indexing(index,update,delete) requests and stores them to /opt/elasticsearch/idxproxy-data/[0-9]+.dat.
### Create Index and Alias
Create sample1 index and sample alias:
```
curl -XPUT 'localhost:9200/sample1?pretty' -H 'Content-Type: application/json' -d'
{
"settings" : {
"index" : {
"number_of_shards" : 3,
"number_of_replicas" : 0
}
}
}
'
curl -XPOST 'localhost:9200/_aliases?pretty' -H 'Content-Type: application/json' -d'
{
"actions" : [
{ "add" : { "index" : "sample1", "alias" : "sample" } }
]
}
'
```
### Send Document
Send 1 document to sample alias:
```
curl -XPOST 'localhost:9200/sample/doc/1?pretty' -H 'Content-Type: application/json' -d'
{
"msg" : "test 1"
}
'
```
Check sample index:
```
curl 'localhost:9200/sample/_search?q=*:*&pretty'
```
If this plugin works correctly, hits.total is 0.
.tmp file is a working files in idxproxy.data.path directory.
```
$ ls idxproxy-data/
0000000000000000001.tmp
```
### Flush Data file
Send refresh request:
```
curl -XPOST 'localhost:9200/_refresh?pretty'
```
.dat file and new .tmp file are created.
```
ls idxproxy-data/
0000000000000000001.dat 0000000000000000002.tmp
```
### Start Document Sender
Start Document Sender for sample1 index:
```
curl -XPOST 'localhost:9200/sample1/_idxproxy/process?pretty'
```
Check sample index:
```
curl 'localhost:9200/sample/_search?q=*:*&pretty'
```
You will find a sent document.
### Stop Document Sender
Stop Document Sender for sample1 index:
```
curl -XDELETE 'localhost:9200/sample1/_idxproxy/process?pretty'
```