{"id":23076382,"url":"https://github.com/ofarukcaki/duckimport","last_synced_at":"2025-10-11T11:12:56.260Z","repository":{"id":35143142,"uuid":"212167486","full_name":"ofarukcaki/duckimport","owner":"ofarukcaki","description":"Elasticsearch Data Import Tool","archived":false,"fork":false,"pushed_at":"2023-02-19T19:51:25.000Z","size":139,"stargazers_count":13,"open_issues_count":4,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-14T09:30:14.727Z","etag":null,"topics":["big","csv","elasticsearch","file","import","index"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ofarukcaki.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-10-01T18:16:42.000Z","updated_at":"2023-02-24T02:47:57.000Z","dependencies_parsed_at":"2023-11-27T16:02:50.696Z","dependency_job_id":"8949be68-22d1-43e9-bda1-7fdc56264ac7","html_url":"https://github.com/ofarukcaki/duckimport","commit_stats":{"total_commits":31,"total_committers":2,"mean_commits":15.5,"dds":"0.032258064516129004","last_synced_commit":"784b0293dadafcbd92d8735f394b3881babfeef7"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofarukcaki%2Fduckimport","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofarukcaki%2Fduckimport/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofarukcaki%2Fduckimport/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ofarukcaki%2Fduckimport/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ofarukcaki","download_url":"https://codeload.github.com/ofarukcaki/duckimport/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229937595,"owners_count":18147640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big","csv","elasticsearch","file","import","index"],"created_at":"2024-12-16T09:31:27.989Z","updated_at":"2025-10-11T11:12:51.239Z","avatar_url":"https://github.com/ofarukcaki.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![npm](https://img.shields.io/npm/v/duckimport.svg)](https://www.npmjs.com/package/duckimport)\n[![npm](https://img.shields.io/node/v/duckimport.svg)](https://www.npmjs.com/package/duckimport)\n[![npm](https://img.shields.io/npm/dt/duckimport.svg)](https://www.npmjs.com/package/duckimport)\n[![GitHub license](https://img.shields.io/github/license/ofarukcaki/duckimport.svg)](https://github.com/ofarukcaki/duckimport/blob/master/LICENSE)\n\n\n![duckimport](https://raw.githubusercontent.com/ofarukcaki/duckimport/master/assets/logo-text.png?token=AFZ3AYHQOYIIK3NASHVVVEK5TXQSY)\n\n# Elasticsearch cli Data Importer \nsee [faq](#faq)\n## Features\n- Can Index(import) very **large** files.\n- Runs on any platform (Windows, Mac, Linux)\n- Easy to use\n- Ability to provide a command-line encoded config. No need to create a local config file\n\n## Install\nInstall [Nodejs](https://nodejs.org) if you haven't already.    \n**Node.js version 11.10** or higher is required. \n\nThen install the package globally:  \n `npm i -g duckimport`  \nor  \n`yarn global add duckimport`      \n\n## Demo\n![Gif](https://raw.githubusercontent.com/ofarukcaki/duckimport/master/assets/demo.gif)\n\n## Usage\n\n`duckimport \u003ccommand\u003e`  \nYou can see available options with `duckimport --help`  \n```\nUsage: duckimport [options]\n\nOptions:\n  -c, --config \u003cpath\u003e          config file path\n  -i, --inline \u003cconfigString\u003e  base64 encoded config object\n  -h, --help                   output usage information\n\nExamples:\n  $ duckimport -c ./config.json\n  $ duckimport -i NDJjNGVx........GZzZGY=\n```     \n\n## Examples\n- `duckimport -c ./config.json`   \n- `duckimport -i ewogICAgIm.....KfQ==`\n\n### You will need a proper json config in order to run **duckimport**    \nAn example config file:\n```javascript\n{\n    \"client\": {\n        \"node\": \"http://localhost:9200\"\n    },\n    \"file\": \"bigFile.csv\",\n    \"separator\": \",\",\n    \"columns\": [\n        \"firstname\",\n        \"lastname\"\n    ],\n    \"lines\": 10000,\n    \"createNewIndex\": true,\n    \"index\": {\n        \"index\": \"peopleIndex\",\n        \"body\": {\n            \"settings\": {\n                \"number_of_replicas\": 0,\n                \"auto_expand_replicas\": false\n            },\n            \"mappings\": {\n                \"properties\": {\n                    \"firstname\": {\n                        \"type\": \"keyword\"\n                    },\n                    \"lastname\": {\n                        \"type\": \"keyword\"\n                    }\n                }\n            }\n        }\n    }\n}\n```\n\n\n## `config`\n\n You can pass a config file using `duckimport -c \u003cconfig file path\u003e`    \n or\n `duckimport -i \u003cbase64 encoded config object\u003e`\n- ### `client`\n    - **Type:** *Object*\n    - Elasticsearch client configuration. [Reference](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/client-configuration.html)\n- ### `file`\n    - **Type:** *String*\n    - The file you want to import(aka. indexing) into Elasticsearch. Can be .txt .csv .tsv etc.\n- ### `separator`\n    - **Type:** *String*\n    - The separator between your data's columns\n    - Exp: \";\",  \":\",  \",\" etc.\n- ### `columns`\n    - **Type:** *Array*\n    - Array of column headers or field names. If there is a non-specified headers they won't be imported.\n- ### `lines`\n    - **Type:** *Number*\n    - Number of lines included in every chunk sent to the Elasticsearch\n- ### `createNewIndex`\n    - **Type:** *Boolean*\n- ### `index`\n    - **Type:** *Object*\n    - Index configuration. [Reference](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html#_indices_create)\n    - **index** field is represents your Index name and it's mandatory. You don't need to include a body field or so if you set **createNewIndex: false**      \n\n\n## F.A.Q\n### How does it work?\n**duckimport** process your files in any size line by line and send them into Elasticsearch as chunks. Thanks to [nexline](https://github.com/sharpart555/nexline)\n### What is `lps` mean?\n**L**ines **P**er **S**econds. Represents how many lines of your file is processing in a second,\n### How can I use without a config file?\nYou can use inline base64 encoded config string using `-i` flag. All you need is prepare your config object(json or js object and encode it using base64. duckimport will decode the encoded string and process it.\n\n\n## Milestones\n- [ ] duckimport GUI - 100+ Github stars \n---  \n\u003csup\u003eDuck icon made by [Freepik](https://www.freepik.com/home) from [http://www.flaticon.com/](www.flaticon.com)\u003c/sup\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofarukcaki%2Fduckimport","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fofarukcaki%2Fduckimport","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fofarukcaki%2Fduckimport/lists"}