Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/neelkanthk/esloader
Instantly index your data from multiple sources into Elasticsearch.
https://github.com/neelkanthk/esloader
elasticsearch elasticsearch-csv elasticsearch-json elasticsearch-xml php7
Last synced: 27 days ago
JSON representation
Instantly index your data from multiple sources into Elasticsearch.
- Host: GitHub
- URL: https://github.com/neelkanthk/esloader
- Owner: neelkanthk
- Created: 2019-08-05T17:46:59.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2019-08-13T19:03:25.000Z (about 5 years ago)
- Last Synced: 2023-07-15T13:06:31.086Z (over 1 year ago)
- Topics: elasticsearch, elasticsearch-csv, elasticsearch-json, elasticsearch-xml, php7
- Language: PHP
- Homepage:
- Size: 107 KB
- Stars: 5
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
EsLoader is a lighweight PHP package for indexing data from multiple sources into Elasticsearch index.
**Table of Contents**
### Features
- Simple and Easy 3 step integration.
- Supports CSV, XML and JSON files.
- Support for MongoDb and MySQL coming soon in next release.
- Indexes data quickly by leveraging the bulk indexing feature of Elasticsearch.
- Supports integration of AWS Elasticsearch Service.
- Fully configurable - Define your own index name, custom mapping and settings and even set the size of each batch request.
**NOTE 1: This package is developed and tested on PHP 7.x and Elasticsearch 6.x.**
**NOTE 2: This package uses `_doc` for the `_type` meta field of Elasticsearch document. This setting is not configurable. To know more read : [Removal of mapping types.](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html "Removal of mapping types.")**
### Installation
`$ composer require neelkanthk/esloader`
### Usage
```php
require __DIR__ . '/vendor/autoload.php';
use Neelkanthk\EsLoader\Core\EsLoader;
//1. Specify the path of file to be indexed.
$filePath = __DIR__ . "/data.csv";
//2. Load array of configurations.
$config = [
"index" => "esloader",
"doc_id_key" => NULL,
"connection" => "local",
"local" => [
'host' => "localhost",
'port' => "9200"
],
"aws" => [
'host' => "",
'region' => "",
'access_key' => "",
'secret_key' => ""
],
"mappings" => [
"_doc" => [
'_source' => [
'enabled' => true
],
"properties" => [
"id" => ['type' => 'keyword'],
"first_name" => ['type' => 'text'],
"last_name" => ['type' => 'text'],
"email" => ['type' => 'keyword'],
"gender" => ['type' => 'keyword'],
"points" => ['type' => 'integer']
]
],
],
"settings" => [
'number_of_shards' => 2,
'number_of_replicas' => 0
],
"batch_size" => 100
];
//3. Pass the file and configuration to the `EsLoader::load` method.
EsLoader::load($filePath, $config);
```
### Configuration
```php
"esloader",
"doc_id_key" => NULL,
"connection" => "local",
"local" => [
'host' => "localhost",
'port' => "9200"
],
"aws" => [
'host' => "",
'region' => "",
'access_key' => "",
'secret_key' => ""
],
"mappings" => [
"_doc" => [
'_source' => [
'enabled' => true
],
"properties" => [
"id" => ['type' => 'keyword'],
"first_name" => ['type' => 'text'],
"last_name" => ['type' => 'text'],
"email" => ['type' => 'keyword'],
"gender" => ['type' => 'keyword'],
"points" => ['type' => 'integer']
]
],
],
"settings" => [
'number_of_shards' => 2,
'number_of_replicas' => 0
],
"batch_size" => 100
];
```
### Demo
The demo code is placed inside the **testdata** directory. The testdata contains sample csv, json and xml files having 1000 records each.
The configuration for the demo can be modified in **config.php** file.
To run the demo you need to have an elasticsearch cluster running. Once you have that you just need to run `php example.php` to run the demo.