https://github.com/snuids/elastic_helper
A simple helper library to convert elastic search to panda dataframe in both directions.
https://github.com/snuids/elastic_helper
elasticsearch pandas-dataframe
Last synced: about 1 month ago
JSON representation
A simple helper library to convert elastic search to panda dataframe in both directions.
- Host: GitHub
- URL: https://github.com/snuids/elastic_helper
- Owner: snuids
- Created: 2019-06-18T12:35:38.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2025-04-25T09:11:40.000Z (about 2 months ago)
- Last Synced: 2025-05-09T19:54:07.535Z (about 1 month ago)
- Topics: elasticsearch, pandas-dataframe
- Language: Python
- Homepage:
- Size: 77.1 KB
- Stars: 7
- Watchers: 2
- Forks: 5
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# elastic_helper
Two simple functions
* One used to convert an elastic search collection into a dataframe. See the code for the various parameters.
* One used to convert a dataframe into an elastic search collection
## Installation
```sh
pip install elastic-helper
```
the pypi page (https://pypi.org/project/elastic-helper/):
## Example Elastic To Pandas
### Parameters
* **es** -- The elastic connection object
* **index** -- The elastic index
* **query** -- (optional) The elastic query in lucene format Example: "module: *"
* **start** -- (optional) The time range start if any
* **end** -- (optional) The time range start if any
* **timestampfield** -- (optional) The timestamp field used by the start and stop parameters
* **datecolumns** -- (optional) A collection of columns that must be converted to dates
* **scrollsize** -- (optional) The size of the scroll to use
* **size** -- (optional) The maximum number of records to retrieve
* **_source** -- (optional) The fields to retrieve
```python
from elastic_helper import es_helper
dataframe=es_helper.elastic_to_dataframe(es,index="docker_stats*"
,_source=['read', 'cpu_percent', 'name']
,datecolumns=["read"]
,timestampfield="read"
,start=datetime.now()-timedelta(hours=1)
,end=datetime.now())
```
## Example Pandas To Elastic
* Use an **_index** column in the dataframe to specify the target index
* Use an **_id** column in the dataframe to specify the id
```python
from elastic_helper import es_helper
es_helper.dataframe_to_elastic(es,my_df)
```