{"id":13595537,"url":"https://github.com/whitfin/limber","last_synced_at":"2025-04-15T21:24:59.567Z","repository":{"id":62442735,"uuid":"186912370","full_name":"whitfin/limber","owner":"whitfin","description":"A simple (but quick) tool for backing up Elasticsearch documents","archived":false,"fork":false,"pushed_at":"2023-06-09T18:54:46.000Z","size":33,"stargazers_count":51,"open_issues_count":2,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-09T16:48:53.897Z","etag":null,"topics":["backup-utility","command-line","datastore","elasticsearch","performance"],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/whitfin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-05-15T22:41:45.000Z","updated_at":"2025-02-07T10:37:17.000Z","dependencies_parsed_at":"2024-01-16T22:19:47.789Z","dependency_job_id":"ef661a0e-b23e-48bf-8090-00abd255a970","html_url":"https://github.com/whitfin/limber","commit_stats":{"total_commits":29,"total_committers":1,"mean_commits":29.0,"dds":0.0,"last_synced_commit":"c9e3f9dd2b4a476aa5cf4be8c52b3f705f58cc5d"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Flimber","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Flimber/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Flimber/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/whitfin%2Flimber/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/whitfin","download_url":"https://codeload.github.com/whitfin/limber/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249155584,"owners_count":21221633,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backup-utility","command-line","datastore","elasticsearch","performance"],"created_at":"2024-08-01T16:01:51.981Z","updated_at":"2025-04-15T21:24:59.547Z","avatar_url":"https://github.com/whitfin.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Limber\n[![Build Status](https://img.shields.io/github/actions/workflow/status/whitfin/limber/ci.yml)](https://github.com/whitfin/limber/actions)\n[![Crates.io](https://img.shields.io/crates/v/limber.svg)](https://crates.io/crates/limber)\n\nA simple (but quick) tool for backing up Elasticsearch documents.\n\nLimber offers a very minimal interface to export and import documents from\nElasticsearch clusters. Other tools do exist for this purpose, but they're\nfairly slow and thus unusable on larger clusters. This tool is intended to\nbe as lightweight and fast as possible (given that it's a CLI tool). Usage\nis meant to align with Unix style pipes to allow piping directly into other\nclusters, or your own compression algorithms, etc.\n\nThe feature set is deliberately small as other tools exist for those who\nare interested in sugar rather than speed. If you would like to request a\nnew feature, please file an issue and we can discuss inclusion.\n\n### Installation\n\nLimber will be available via [Crates.io](https://crates.io/crates/limber),\nso you can install it directly with `cargo`:\n\n```shell\n$ cargo install limber\n```\n\nOnce I become more familiar with cross compilation, I'll try to attach some\npre-built binaries to the repository to make it easier for those outside of\nthe Rust ecosystem.\n\n### Usage\n\nThis section will cover (at a very high level), the most relevant commands\noffered by Limber. If you want up to date usage, please check the help menu\ngenerated via `limber -h` in your terminal session.\n\n#### Exporting Documents\n\nTo export documents from an Elasticsearch cluster, you can use the `export`\nsubcommand. This is formed in the following pattern:\n\n```shell\n$ limber export \u003csource\u003e\n```\n\nThe `source` parameter is required, and much contain the Elasticsearch URL\nof the cluster you wish to export from. It is assumed that the cluster is\nreachable from the machine you're running this tool on. If you wish to\nexport from a specific index, you can also include this on the URL in the\nform of:\n\n```text\nhttp://localhost:9200/my_index\nhttp://localhost:9200/my_first_index,my_second_index\n```\n\nExported documents are dumped directly to `stdout`, and progress will be\nreported to `stderr`. This allows you to pipe the results into whatever\ndestination you wish in a streaming fashion. As such, invocation of the\nexport command typically looks something like this:\n\n```shell\n$ limber export http://localhost:9200 | gzip -9 \u003e export.jsonl.gz\n```\n\nThe above command will compress all exported data into the `export.jsonl.gz`\nfile, whilst occasionally reporting progress to your terminal session (as\nlong as you don't also redirect `stderr`).\n\nThere are several options which can be used to customize the export, such\nas the concurrency factor, batch sizes, document filtering, etc. All of\nthese options can be found via `limber export -h`.\n\n#### Importing Documents\n\nThe process of importing documents is extremely similar to exporting them,\nexcept it operates in reverse. Just like the `export` command, this is\nformed in the following pattern:\n\n```shell\n$ limber import \u003ctarget\u003e\n```\n\nThe `target` parameter is required, and works in the same way as the `source`\nparameter to control which index you place your documents into. If you do\nnot provide an index, the documents will be placed into the same index name\nas the one they were exported from. Naturally, using multiple index names no\nlonger makes sense when indexing documents.\n\nIn similar fashion to exports, documents to import are read from `stdin` in\norder to allow you to stream from one location to another. Below is the an\nexample of importing the documents exported in the example further up:\n\n```shell\n$ gzcat export.jsonl.gz | limber import http://localhost:9200\n```\n\nYou might notice that the API here allows you to pipe from one cluster or\nindex directly into another. As an example:\n\n```shell\n$ limber export http://localhost:9200/my_first_index | \\\n    limber import http://localhost:9200/my_second_index\n```\n\nThe import command also allows for customization of concurrency factor and\nbatch sizes. For all available options, please see `limber import -h`.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitfin%2Flimber","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwhitfin%2Flimber","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwhitfin%2Flimber/lists"}