{"id":28959422,"url":"https://github.com/arthubhub/elk_index_exporter","last_synced_at":"2026-05-07T13:47:29.682Z","repository":{"id":300752665,"uuid":"1007043226","full_name":"arthubhub/ELK_Index_exporter","owner":"arthubhub","description":"This repository contains a script to export a bunch of indexes from elk stack into json","archived":false,"fork":false,"pushed_at":"2025-06-23T11:31:33.000Z","size":5,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-23T12:33:59.735Z","etag":null,"topics":["elasticsearch","json"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arthubhub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-23T11:27:56.000Z","updated_at":"2025-06-23T11:32:19.000Z","dependencies_parsed_at":"2025-06-23T12:46:30.555Z","dependency_job_id":null,"html_url":"https://github.com/arthubhub/ELK_Index_exporter","commit_stats":null,"previous_names":["arthubhub/elk_index_exporter"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/arthubhub/ELK_Index_exporter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthubhub%2FELK_Index_exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthubhub%2FELK_Index_exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthubhub%2FELK_Index_exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthubhub%2FELK_Index_exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arthubhub","download_url":"https://codeload.github.com/arthubhub/ELK_Index_exporter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arthubhub%2FELK_Index_exporter/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261577977,"owners_count":23179766,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","json"],"created_at":"2025-06-24T00:01:47.493Z","updated_at":"2026-05-07T13:47:29.638Z","avatar_url":"https://github.com/arthubhub.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ELK_Index_exporter\nYou are looking for a way to **export elk logs in json** ?\n-\u003e Use the ELK_Index_exporter\n\n## Requirements\n- jq\n- curl\n- gzip\n\n## Process\n- The program checks the required packages\n- It creates an output dir\n- It reads the indexes as input\n- For each index : it reads the whole index into a json temp file, scrolling along the file with GET requests\n- After downloading each index it compresses it, and remove the temp file\n\n## Usage\n\n### 1° Configuration\n\n- In the export_all.sh, setup your parameters (dont push your creds ;]) :\n  ```bash\n  #!/bin/bash\n\n  ELASTICSEARCH_HOST=\"127.0.0.1:9200\" # -\u003e \"192.168.1.1:9200\"\n  USERNAME=\"elastic\" # -\u003e \"elastic\"\n  PASSWORD=\"mySup3rP4ssw0rd\" # -\u003e \"yourpass\"\n  INDEX_LIST_FILE=\"index_list.txt\"\n  OUTPUT_DIR=\"./exports\"\n  BATCH_SIZE=5000\n  SCROLL_TIMEOUT=\"5m\"\n  ```\n- List your indexes to export (I will not explain how to find them on kibana or with a curl command, do it yourself ^^)\n  ```bash\n  #Here is an example with indexes created from a datastream :\n  .ds-logs-generic-default-2025.06.13-000001\n  .ds-logs-netflow-default-2025.06.13-000001\n  .ds-logs-router_netflow-default-2025.06.13-000001\n  .ds-logs-router_syslog-default-2025.06.13-000001\n  .ds-logs-radius-default-2025.06.13-000001\n  .ds-filebeat-8.13.0-2025.06.13-000001  \n  ```\n### 2° Launch your script \n- Try first without indexes if you want to check the connection !\n- To run, simply do : `chrmod +x ./export_all.sh`\n- Then : `./export_all.sh`\n- If you want to save output : `./export_all.sh \u003e\u003e app_logs.txt`\n\n### 3° Example of usage\n\n- Running a test with a list of ~13M logs\n- `./export_all.sh \u003e\u003e index_list.txt\"`\n- `tail -f index_list.txt`\n- Result : `cat script_logs.txt | grep -v \"Batch : 5000 documents | Total exporté :\"`\n    ```text\n    [2025-06-23 13:44:04] Début de l'export des index Elasticsearch\n    [2025-06-23 13:44:04] Vérification des prérequis...\n    [SUCCESS] Prérequis vérifiés\n    [2025-06-23 13:44:04] Répertoire de sortie créé : ./exports\n    [2025-06-23 13:44:04] ----------------------------------------\n    [2025-06-23 13:44:04] Début de l'export de l'index : .ds-logs-generic-default-2025.06.13-000001\n    [2025-06-23 13:44:04] Nombre total de documents : 11858\n    [2025-06-23 13:44:06] Premier batch : 5000 documents exportés\n    [2025-06-23 13:44:07] Batch : 1858 documents | Total exporté : 11858/11858\n    [2025-06-23 13:44:08] Batch : 0 documents | Total exporté : 11858/11858\n    [SUCCESS] Index .ds-logs-generic-default-2025.06.13-000001 exporté : 11858 documents dans ./exports/.ds-logs-generic-default-2025.06.13-000001.json\n    [SUCCESS] Fichier compressé : ./exports/.ds-logs-generic-default-2025.06.13-000001.json.gz\n    [2025-06-23 13:44:08] ----------------------------------------\n    [2025-06-23 13:44:08] Début de l'export de l'index : .ds-logs-netflow-default-2025.06.13-000001\n    [2025-06-23 13:44:08] Nombre total de documents : 1502414\n    [2025-06-23 13:44:09] Premier batch : 5000 documents exportés\n    [2025-06-23 13:52:21] Batch : 2414 documents | Total exporté : 1502414/1502414\n    [2025-06-23 13:52:21] Batch : 0 documents | Total exporté : 1502414/1502414\n    [SUCCESS] Index .ds-logs-netflow-default-2025.06.13-000001 exporté : 1502414 documents dans ./exports/.ds-logs-netflow-default-2025.06.13-000001.json\n    [SUCCESS] Fichier compressé : ./exports/.ds-logs-netflow-default-2025.06.13-000001.json.gz\n    [2025-06-23 13:53:19] ----------------------------------------\n    [2025-06-23 13:53:19] Début de l'export de l'index : .ds-logs-router_netflow-default-2025.06.13-000001\n    [2025-06-23 13:53:19] Nombre total de documents : 5461188\n    [2025-06-23 13:53:20] Premier batch : 5000 documents exportés\n    [2025-06-23 14:24:59] Batch : 1188 documents | Total exporté : 5461188/5461188\n    [2025-06-23 14:24:59] Batch : 0 documents | Total exporté : 5461188/5461188\n    [SUCCESS] Index .ds-logs-router_netflow-default-2025.06.13-000001 exporté : 5461188 documents dans ./exports/.ds-logs-router_netflow-default-2025.06.13-000001.json\n    [SUCCESS] Fichier compressé : ./exports/.ds-logs-router_netflow-default-2025.06.13-000001.json.gz\n    [2025-06-23 14:27:14] ----------------------------------------\n    [2025-06-23 14:27:15] Début de l'export de l'index : .ds-logs-router_syslog-default-2025.06.13-000001\n    [2025-06-23 14:27:15] Nombre total de documents : 6955737\n    [2025-06-23 14:27:16] Premier batch : 5000 documents exportés\n    [2025-06-23 15:06:18] Batch : 737 documents | Total exporté : 6955737/6955737\n    [2025-06-23 15:06:18] Batch : 0 documents | Total exporté : 6955737/6955737\n    [SUCCESS] Index .ds-logs-router_syslog-default-2025.06.13-000001 exporté : 6955737 documents dans ./exports/.ds-logs-router_syslog-default-2025.06.13-000001.json\n    [SUCCESS] Fichier compressé : ./exports/.ds-logs-router_syslog-default-2025.06.13-000001.json.gz\n    [2025-06-23 15:09:02] ----------------------------------------\n    [2025-06-23 15:09:02] Début de l'export de l'index : .ds-logs-radius-default-2025.06.13-000001\n    [2025-06-23 15:09:02] Nombre total de documents : 529\n    [2025-06-23 15:09:02] Premier batch : 529 documents exportés\n    [2025-06-23 15:09:02] Batch : 0 documents | Total exporté : 529/529\n    [SUCCESS] Index .ds-logs-radius-default-2025.06.13-000001 exporté : 529 documents dans ./exports/.ds-logs-radius-default-2025.06.13-000001.json\n    [SUCCESS] Fichier compressé : ./exports/.ds-logs-radius-default-2025.06.13-000001.json.gz\n    [2025-06-23 15:09:02] ----------------------------------------\n    [2025-06-23 15:09:02] Début de l'export de l'index : .ds-filebeat-8.13.0-2025.06.13-000001\n    [2025-06-23 15:09:02] Nombre total de documents : 20\n    null\n    [2025-06-23 15:09:02] ========================================\n    [2025-06-23 15:09:02] Export terminé !\n    [2025-06-23 15:09:02] Total des index : 6\n    [SUCCESS] Exports réussis : 5\n    [2025-06-23 15:09:02] Fichiers de sortie dans : ./exports\n    [2025-06-23 15:09:02] Taille totale des exports : 446M\n    ```\n\n### 4° Statistics\n- Count of indexes : 6\n- Errors : 1\n- Logs : ~13M\n- Output :\n  ```\n  ls -la exports\n  total 456116\n  drwxr-xr-x 2 arthubhub arthubhub      4096 Jun 23 15:09 .\n  drwxr-xr-x 4 arthubhub arthubhub      4096 Jun 23 13:44 ..\n  -rw-r--r-- 1 arthubhub arthubhub    346657 Jun 23 13:44 .ds-logs-generic-default-2025.06.13-000001.json.gz\n  -rw-r--r-- 1 arthubhub arthubhub  47820077 Jun 23 13:52 .ds-logs-netflow-default-2025.06.13-000001.json.gz\n  -rw-r--r-- 1 arthubhub arthubhub     16784 Jun 23 15:09 .ds-logs-radius-default-2025.06.13-000001.json.gz\n  -rw-r--r-- 1 arthubhub arthubhub 183610222 Jun 23 14:24 .ds-logs-router_netflow-default-2025.06.13-000001.json.gz\n  -rw-r--r-- 1 arthubhub arthubhub 235246670 Jun 23 15:06 .ds-logs-router_syslog-default-2025.06.13-000001.json.gz\n  ```\n- Duration :  `15:09:02 - 13:44:04 = 01:24:58`\n- Speed : `155 000` logs per minute\n  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthubhub%2Felk_index_exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farthubhub%2Felk_index_exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farthubhub%2Felk_index_exporter/lists"}