https://github.com/mlof/elasticuploader
https://github.com/mlof/elasticuploader
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/mlof/elasticuploader
- Owner: mlof
- Created: 2023-02-28T10:19:02.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-03-01T21:39:39.000Z (over 3 years ago)
- Last Synced: 2025-03-01T14:41:33.732Z (over 1 year ago)
- Language: C#
- Size: 28.3 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Elastic Uploader
A simple tool to upload a csv file to an Elasticsearch index. Because screw having to write a logstash config for every csv file.
If index name is not specified, the file name will be used.
Authentication can be done with either a cloud id and api key or a elastic uri and user/password.
When in doubt, run with -h to get help.
## Example usage
```bat
elastic-upload -f C:\temp\test.csv -e "http://localhost:9200" -u elastic -p changeme -b 1000
elastic-upload -f C:\temp\test.csv -c cloudid:cloudid -k apikey -b 1000
```
## Performance tuning
The tool uses a bulk upload to send the data to Elasticsearch. The default batch size is 1000. This can be changed with the -b parameter. It awaits all bulk actions to be completed before sending the next batch. I don't want to set your server on fire.
Official documentation says that the Bulk API wil comfortably handle 5-15 mb per bulk. I don't know how wide your data is, so you might want to adjust the batch size accordingly.
Dynamic bulk sizing does sound interesting, so it might be added in the future.