Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/xros/jsonpyes
The tool which imports raw JSON to ElasticSearch in one line of commands
https://github.com/xros/jsonpyes
algorithm cross-platform elasticsearch json-data multi-threading python
Last synced: 3 months ago
JSON representation
The tool which imports raw JSON to ElasticSearch in one line of commands
- Host: GitHub
- URL: https://github.com/xros/jsonpyes
- Owner: xros
- License: other
- Created: 2015-11-27T05:42:14.000Z (about 9 years ago)
- Default Branch: master
- Last Pushed: 2019-07-04T17:53:05.000Z (over 5 years ago)
- Last Synced: 2024-10-10T15:12:55.643Z (4 months ago)
- Topics: algorithm, cross-platform, elasticsearch, json-data, multi-threading, python
- Language: Python
- Homepage:
- Size: 979 KB
- Stars: 67
- Watchers: 7
- Forks: 21
- Open Issues: 17
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
json-py-es
==========
[](https://pepy.tech/project/jsonpyes)
[](https://travis-ci.org/xros/jsonpyes)
[](https://github.com/xros/jsonpyes/releases)
[](https://github.com/xros/jsonpyes/blob/master/LICENSE)
Alexander Liu
* To import raw JSON data files to ElasticSearch in one line of commands
Very fast -- 4 to 10 times faster when processing big data.
### Installation
```pip install jsonpyes```
> **Notice**: Before using `pip` to install `jsonpyes`, firstly you need to install `python-pip` on your system. ( Supports Python2.7, 3,4, 3.5, 3.6 )
jsonpyes
--------
#### Instructions:
There are 3 proccesses of importing raw JSON data to ElasticSearch
1. Only validating raw JSON data
2. Without validating ,just import data to ElasticSearch
3. After validating successfully, then import data to ElasticSearch
A valid JSON file here refers to a JSON file stacked with many lines of data
file valid_data.json and its content
{"key1": "valueA", "key2": {"sub_key1": "value2A", "sub_key2": ["Good", "Morning"]}}
{"key1": "valueB", "key2": {"sub_key1": "value2B", "sub_key2": ["Good", "Afternoon"]}}
...
{"key1": "valueC", "key2": {"sub_key1": "value2C", "sub_key2": ["Good", "Evening"]}}
Functions included
------------------
##### 1. Validating JSON format data
```jsonpyes --data raw_data.json --check```
If the json data file is valid:
If the json data file is invalid:
##### 2. Only importing without validating
```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2```
Notice: If the raw JSON data file is invalid, `jsonpyes` will not import it.
Or enable multi-threads ```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex2 --type mytype2 --thread 8```
```jsonpyes``` supports multi-threads when importing data to elasticsearch
##### Multi-threads comparison
1. No multi-threads
2. With 8 threads and `jsonpyes` cuts files into pieces, then destributes to workers fairly
> As you can see these two containers have same docs loaded, if we use **_--thread 8_** it could be several times faster, usually 5 to 10 times faster.
> That really depends on your computer/server resources.
This was tested on a 4GB RAM / 2.4Ghz intel i5 Linux x64 laptop system.
And it works.
##### 3. Both validating and importing
```jsonpyes --data raw_data.json --bulk http://localhost:9200 --import --index myindex1 --type mytype1 --check```
And it works.
Reference
---------
* Algorithm handwritting
##### Happy hacking!