https://github.com/codecentric/elasticsearch-fileimport

Import JSON files directly as documents into Elasticsearch by using the ES transport protocol (not HTTP/REST)
https://github.com/codecentric/elasticsearch-fileimport

Last synced: 9 months ago
JSON representation

Import JSON files directly as documents into Elasticsearch by using the ES transport protocol (not HTTP/REST)

• Host: GitHub
• URL: https://github.com/codecentric/elasticsearch-fileimport
• Owner: codecentric
• Created: 2015-06-20T09:32:25.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2015-06-25T09:44:00.000Z (over 10 years ago)
• Last Synced: 2023-03-22T13:44:41.094Z (almost 3 years ago)
• Language: Java
• Homepage:
• Size: 125 KB
• Stars: 5
• Watchers: 160
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
#elasticsearch-fileimport

Import JSON files directly as documents into Elasticsearch by using the ES transport protocol (not HTTP/REST).

The import is done via bulk requests.

##Prerequisites

* Java 8

* Tested with Elasticsearch 1.6 (should also work with earlier versions)

##Usage

    mvn package

    java -jar target/elasticsearch-fileimport-0.6-SNAPSHOT-jar-with-dependencies.jar file_import_settings.yml

To execute all unittests, also those who runs longer execute

    mvn clean package -DargLine="-Dfileimport.tests.big"

##Configuration

The configuration is done in YAML (or JSON). Look at the following example to see what's configurable:

    fileimport:

        # either transport or node

        mode: node

        

        # Folder with data

        # Folder is scanned recursively

        root: import/data

        

        # Fileextension of files to import, default is json

        #fileext: json

        

        # If set to true then import files line by line (one json doc per line)

        # If set to false (which is the default) treat one file as one document

        #linebyline: true   

        

        # Index name

        index: test

        

        # Type name

        type: test

        

        # Bulk settings, see https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/bulk.html

        #flush_interval: 5s

        max_bulk_actions: 1000

        max_concurrent_bulk_requests: 1

        max_volume_per_bulk_request: 10mb

        

        # needed when mode: transport

        transport:

            addresses:

                - localhost:9300

                - es.comany.com:9300

                

    # needed when mode: node

    cluster:

        name: elasticsearch

        

    

    # list other elasticsearch settings here if needed

    #discovery.zen.ping.multicast.enabled: false

    #discovery.zen.ping.unicast.hosts: host:9300

    #...

##Before you run it

* Install and start your Elasticsearch cluster

* Create your index (with your desired settings)

* Create your type mapping