{"id":13614863,"url":"https://github.com/ihcsim/promdump","last_synced_at":"2025-10-23T02:26:18.267Z","repository":{"id":40570820,"uuid":"355683449","full_name":"ihcsim/promdump","owner":"ihcsim","description":"A tool to dump and restore Prometheus data blocks.","archived":false,"fork":false,"pushed_at":"2022-08-24T04:54:20.000Z","size":24481,"stargazers_count":137,"open_issues_count":5,"forks_count":8,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-06T02:19:18.211Z","etag":null,"topics":["kubectl-plugin","kubernetes","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ihcsim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-07T21:12:08.000Z","updated_at":"2025-07-29T13:28:38.000Z","dependencies_parsed_at":"2022-08-09T23:21:53.804Z","dependency_job_id":null,"html_url":"https://github.com/ihcsim/promdump","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/ihcsim/promdump","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihcsim%2Fpromdump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihcsim%2Fpromdump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihcsim%2Fpromdump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihcsim%2Fpromdump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ihcsim","download_url":"https://codeload.github.com/ihcsim/promdump/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ihcsim%2Fpromdump/sbom","scorecard":{"id":483053,"data":{"date":"2025-07-28","repo":{"name":"github.com/ihcsim/promdump","commit":"be920d15c8739eb2f93059415cfcad0b6e21a1ca"},"scorecard":{"version":"v5.2.1-41-g40576783","commit":"40576783fda6698350fcbbeaea760ff827433034"},"score":2.3,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/22 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/pipeline.yaml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#token-permissions"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#packaging"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#fuzzing"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:15: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:26: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:30: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:48: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:49: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:63: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:67: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:73: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/pipeline.yaml:90: update your workflow using https://app.stepsecurity.io/secureworkflow/ihcsim/promdump/pipeline.yaml/main?enable=pin","Warn: containerImage not pinned by hash: Dockerfile:1: pin your Docker image by updating debian:bookworm-20220622 to debian:bookworm-20220622@sha256:81ad6aae9ba6d6b674ef02352851c98e5845a8e91a2f78f9c8bc02915f4882ee","Warn: downloadThenRun not pinned by hash: .github/workflows/pipeline.yaml:18","Info:   0 out of   6 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of   1 containerImage dependencies pinned","Info:   0 out of   1 downloadThenRun dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#pinned-dependencies"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v0.2.5 not signed: https://api.github.com/repos/ihcsim/promdump/releases/75180996","Warn: release artifact v0.2.4 not signed: https://api.github.com/repos/ihcsim/promdump/releases/65776867","Warn: release artifact v0.2.3 not signed: https://api.github.com/repos/ihcsim/promdump/releases/42393473","Warn: release artifact v0.2.2 not signed: https://api.github.com/repos/ihcsim/promdump/releases/42301027","Warn: release artifact v0.2.1 not signed: https://api.github.com/repos/ihcsim/promdump/releases/42244846","Warn: release artifact v0.2.5 does not have provenance: https://api.github.com/repos/ihcsim/promdump/releases/75180996","Warn: release artifact v0.2.4 does not have provenance: https://api.github.com/repos/ihcsim/promdump/releases/65776867","Warn: release artifact v0.2.3 does not have provenance: https://api.github.com/repos/ihcsim/promdump/releases/42393473","Warn: release artifact v0.2.2 does not have provenance: https://api.github.com/repos/ihcsim/promdump/releases/42301027","Warn: release artifact v0.2.1 does not have provenance: https://api.github.com/repos/ihcsim/promdump/releases/42244846"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 12 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":0,"reason":"27 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GO-2022-0619 / GHSA-r48q-9g5r-8q2h","Warn: Project is vulnerable to: GO-2021-0053 / GHSA-c3h9-896r-86jm","Warn: Project is vulnerable to: GO-2022-0322 / GHSA-cg3q-j54f-5p7p","Warn: Project is vulnerable to: GO-2021-0227 / GHSA-3vm4-22fp-5rfm","Warn: Project is vulnerable to: GO-2022-0968 / GHSA-gwc9-m7rh-j2ww","Warn: Project is vulnerable to: GO-2021-0356 / GHSA-8c26-wmh5-6g9v","Warn: Project is vulnerable to: GO-2024-2961","Warn: Project is vulnerable to: GO-2023-2402 / GHSA-45x7-px36-x8w8","Warn: Project is vulnerable to: GO-2024-3321 / GHSA-v778-237x-gjrc","Warn: Project is vulnerable to: GO-2025-3487 / GHSA-hcg3-q754-cr77","Warn: Project is vulnerable to: GO-2021-0238 / GHSA-83g2-8m93-v3w7","Warn: Project is vulnerable to: GO-2022-0288","Warn: Project is vulnerable to: GO-2022-0969 / GHSA-69cg-p879-7622","Warn: Project is vulnerable to: GO-2022-1144 / GHSA-xrjj-mj9h-534m","Warn: Project is vulnerable to: GO-2023-1571 / GHSA-vvpx-j8f3-3w6h","Warn: Project is vulnerable to: GO-2023-1988 / GHSA-2wrh-6pvc-2jm9","Warn: Project is vulnerable to: GO-2023-2102 / GHSA-4374-p667-p6c8","Warn: Project is vulnerable to: GHSA-qppj-fm5r-hxr3","Warn: Project is vulnerable to: GO-2024-2687 / GHSA-4v7x-pqxf-cx7m","Warn: Project is vulnerable to: GO-2024-3333","Warn: Project is vulnerable to: GO-2025-3503 / GHSA-qxp5-gwg8-xv66","Warn: Project is vulnerable to: GO-2025-3595 / GHSA-vvgc-356p-c3xw","Warn: Project is vulnerable to: GO-2025-3488 / GHSA-6v2p-p543-phr9","Warn: Project is vulnerable to: GO-2022-0493 / GHSA-p782-xgp4-8hr8","Warn: Project is vulnerable to: GO-2021-0113 / GHSA-ppp9-7jff-5vj2","Warn: Project is vulnerable to: GO-2022-1059 / GHSA-69ch-w2m2-3vjp","Warn: Project is vulnerable to: GO-2024-2611 / GHSA-8r3f-844c-mc37"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-19T17:05:32.063Z","repository_id":40570820,"created_at":"2025-08-19T17:05:32.064Z","updated_at":"2025-08-19T17:05:32.064Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278548520,"owners_count":26004912,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-06T02:00:05.630Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["kubectl-plugin","kubernetes","prometheus"],"created_at":"2024-08-01T20:01:06.518Z","updated_at":"2025-10-06T13:45:48.712Z","avatar_url":"https://github.com/ihcsim.png","language":"Go","funding_links":[],"categories":["Go","Observability"],"sub_categories":[],"readme":"# promdump\n\n![pipeline](https://github.com/ihcsim/promdump/actions/workflows/pipeline.yaml/badge.svg)\n\npromdump dumps the head and persistent blocks of Prometheus. It supports\nfiltering the persistent blocks by time range.\n\n* [Why This Tool](#why-this-tool)\n* [How It Works](#how-it-works)\n* [Getting Started](#getting-started)\n* [FAQ](#faq)\n* [Limitations](#limitations)\n* [Development](#development)\n* [License](#license)\n\n## Why This Tool\n\nWhen debugging Kubernetes clusters, I often find it helpful to get access to the\nin-cluster Prometheus metrics. Since it is unlikely that the users will grant me\ndirect access to their Prometheus instance, I have to ask them to export the\ndata. To reduce the amount of back-and-forth with the users (due to missing\nmetrics, incorrect labels etc.), it makes sense to ask the users to _\"get me\neverything around the time of the incident\"_.\n\nThe most common way to achieve this is to use commands like `kubectl exec` and\n`kubectl cp` to compress and dump Prometheus' entire data directory. On\nnon-trivial clusters, the resulting compressed file can be very large. To\nimport the data into a local test instance, I will need at least the same amount\nof disk space.\n\npromdump is a tool that can be used to dump Prometheus data blocks. It is\ndifferent from the `promtool tsdb dump` command in such a way that its output\ncan be re-used in another Prometheus instance. See this\n[issue](https://github.com/prometheus/prometheus/issues/8281) for a discussion\non the limitation on the output of `promtool tsdb dump`. And unlike the\nPromethues TSDB `snapshot` API, promdump doesn't require Prometheus to be\nstarted with the `--web.enable-admin-api` option. Instead of dumping the entire\nTSDB, promdump offers the flexibility to filter persistent blocks by time range.\n\n## How It Works\n\nThe promdump kubectl plugin uploads the compressed, embeded promdump to the\ntargeted Prometheus container, via the pod's `exec` subresource.\n\nWithin the Prometheus container, promdump queries the Prometheus TSDB using the\n[`tsdb`](https://pkg.go.dev/github.com/prometheus/prometheus/tsdb) package. It\nreads and streams the WAL files, head block and persistent blocks to stdout,\nwhich can be redirected to a file on your local file system. To regulate the\nsize of the dump, persistent blocks can be filtered by time range.\n\n⭐ _promdump performs read-only operations on the TSDB._\n\nWhen the data dump is completed, the promdump binary will be automatically\ndeleted from your Prometheus container.\n\nThe `restore` subcommand can then be used to copy this dump file to another\nPrometheus container. When this container is restarted, it will reconstruct its\nin-memory index and chunks using the restored on-disk memory-mapped chunks and\nWAL.\n\nThe `--debug` option can be used to output more verbose logs for each command.\n\n## Getting Started\n\nInstall promdump as a `kubectl` plugin:\n```sh\nkubectl krew update\n\nkubectl krew install promdump\n\nkubectl promdump --version\n```\n\nFor demonstration purposes, use [kind](https://kind.sigs.k8s.io/) to create two\nK8s clusters:\n```sh\nfor i in {0..1}; do \\\n  kind create cluster --name dev-0$i ;\\\ndone\n```\n\nInstall Prometheus on both clusters using the community\n[Helm chart](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus):\n```sh\nfor i in {0..1}; do \\\n  helm --kube-context=kind-dev-0$i install prometheus prometheus-community/prometheus ;\\\ndone\n```\n\nDeploy a custom controller to cluster `dev-00`. This controller is annotated for\nmetrics scraping:\n```sh\nkubectl --context=kind-dev-00 apply -f https://raw.githubusercontent.com/ihcsim/controllers/master/podlister/deployment.yaml\n```\n\nPort-forward to the Prometheus pod to find the custom `demo_http_requests_total`\nmetric.\n\n📝 _Later, we will use promdump to copy the samples of this metric over to the\n`dev-01` cluster._\n\n```sh\nCONTEXT=\"kind-dev-00\"\nPOD_NAME=$(kubectl --context \"${CONTEXT}\" get pods --namespace default -l \"app=prometheus,component=server\" -o jsonpath=\"{.items[0].metadata.name}\")\nkubectl --context=\"${CONTEXT}\" port-forward \"${POD_NAME}\" 9090\n```\n\n![Demo controller metrics](img/demo_http_requests_total_dev_00.png)\n\n📝 _In subsequent commands, the `-c` and `-d` options can be used to change\nthe container name and data directoy._\n\nDump the data from the first cluster:\n```sh\n# check the tsdb metadata.\n# if it's reported that there are no persistent blocks yet, then no data dump\n# will be captured, until Prometheus persists the data.\nkubectl promdump meta --context=$CONTEXT -p $POD_NAME\nHead Block Metadata\n------------------------\nMinimum time (UTC): | 2021-04-18 18:00:03\nMaximum time (UTC): | 2021-04-18 20:34:48\nNumber of series    | 18453\n\nPersistent Blocks Metadata\n----------------------------\nMinimum time (UTC):     | 2021-04-15 03:19:10\nMaximum time (UTC):     | 2021-04-18 18:00:00\nTotal number of blocks  | 9\nTotal number of samples | 92561234\nTotal number of series  | 181304\nTotal size              | 139272005\n\n# capture the data dump\nTARFILE=\"dump-`date +%s`.tar.gz\"\nkubectl promdump \\\n  --context \"${CONTEXT}\" \\\n  -p \"${POD_NAME}\" \\\n  --min-time \"2021-04-15 03:19:10\" \\\n  --max-time \"2021-04-18 20:34:48\"  \u003e \"${TARFILE}\"\n\n# view the content of the tar file. expect to see the 'chunk_heads', 'wal' and\n# persistent blocks directories.\n$ tar -tf \"${TARFILE}\"\n```\n\nRestore the data dump to the Prometheus pod on the `dev-01` cluster, where we\ndon't have the custom controller:\n```sh\nCONTEXT=\"kind-dev-01\"\nPOD_NAME=$(kubectl --context \"${CONTEXT}\" get pods --namespace default -l \"app=prometheus,component=server\" -o jsonpath=\"{.items[0].metadata.name}\")\n\n# check the tsdb metadata\nkubectl promdump meta --context \"${CONTEXT}\" -p \"${POD_NAME}\"\nHead Block Metadata\n------------------------\nMinimum time (UTC): | 2021-04-18 20:39:21\nMaximum time (UTC): | 2021-04-18 20:47:30\nNumber of series    | 20390\n\nNo persistent blocks found\n\n# restore the data dump found at ${TARFILE}\nkubectl promdump restore \\\n  --context=\"${CONTEXT}\" \\\n  -p \"${POD_NAME}\" \\\n  -t \"${TARFILE}\"\n\n# check the metadata again. it should match that of the dev-00 cluster\nkubectl promdump meta --context \"${CONTEXT}\" -p \"${POD_NAME}\"\nHead Block Metadata\n------------------------\nMinimum time (UTC): | 2021-04-18 18:00:03\nMaximum time (UTC): | 2021-04-18 20:35:48\nNumber of series    | 18453\n\nPersistent Blocks Metadata\n----------------------------\nMinimum time (UTC):     | 2021-04-15 03:19:10\nMaximum time (UTC):     | 2021-04-18 18:00:00\nTotal number of blocks  | 9\nTotal number of samples | 92561234\nTotal number of series  | 181304\nTotal size              | 139272005\n\n# confirm that the WAL, head and persistent blocks are copied to the targeted\n# Prometheus server\nkubectl --context=\"${CONTEXT}\" exec \"${POD_NAME}\" -c prometheus-server -- ls -al /data\n```\n\nRestart the Prometheus pod:\n```sh\nkubectl --context=\"${CONTEXT}\" delete po \"${POD_NAME}\"\n```\n\nPort-forward to the pod to confirm that the samples of\nthe `demo_http_requests_total` metric have been copied over:\n```sh\nkubectl --context=\"${CONTEXT}\" port-forward \"${POD_NAME}\" 9091:9090\n```\n\nMake sure that time frame of your query matches that of the restored data.\n\n![Restored metrics](img/demo_http_requests_total_dev_01.png)\n\n## FAQ\n\nQ: The `promdump meta` subcommand shows that the time range of the restored\npersistent data blocks is different from the ones I specified.\n\nA: There isn't a way to fetch partial data blocks from the TSDB. If the time\nrange you specified spans across multiple data blocks, then all of them need\nto be retrieved. The amount of excessive data retrieved is dependent on the\nspan of the data blocks.\n\nThe time range reported by the `promdump meta` subcommand should cover the one\nyou specified.\n\n----\nQ: I am not seeing the restored data\n\nA: There are a few things you can check:\n\n* When generating the dump, make sure the start and end date times are\nspecified in the UTC time zone.\n* If using the Prometheus console, make sure the time filter falls within the\ntime range of your data dump. You can confirm your restored data time range\nusing the `promdump meta` subcommand.\n* Compare the TSDB metadata of the target Prometheus with the source Prometheus\nto see if their time range match, using the `promdump meta` subcommand.\nThe head block metadata may deviate slightly depending on how old your data dump\nis.\n* Use the `kubectl exec` command to run commands likes `ls -al \u003cdata_dir\u003e`\nand `cat \u003cdata_dir\u003e/\u003cdata_block\u003e/meta.json` to confirm the data range of a\nparticular data block.\n* Try restarting the target Prometheus pod after the restoration to let\nPrometheus replay the restored WALs. The restored data must be persisted to\nsurvive a restart.\n* Check Prometheus logs to see if there are any errors due to corrupted data\nblocks, and report any [issues](https://github.com/ihcsim/promdump/issues/new).\n* Run the `promdump restore` subcommand with the `--debug` flag to see\nif it provides more hints.\n\n----\nQ: The `promdump meta` and `promdump restore` subcommands are failing with this\nerror:\n```sh\nfound unsequential head chunk files\n```\n\nA: This happens when there are out-of-sequence files in the `chunk_heads` folder\nof the source Prometheus instance.\n\nThe `promdump` command can still be used to generate the dump `.tar.gz` file\nbecause it doesn't parse the folder content, using the `tsdb` API.  It simply\nadds the the entire `chunk_heads` folder to the dump `.tar.gz` file.\n\nE.g, a dump file with 2 out-of-sequence head files may look like this:\n```sh\n$ tar -tf dump.tar.gz\n./\n./chunks_head/\n./chunks_head/000027 # out-of-sequence\n./chunks_head/000029 # out-of-sequence\n./chunks_head/000033\n./chunks_head/000034\n./01F5ETH5T4MKTXJ1PEHQ71758P/\n./01F5ETH5T4MKTXJ1PEHQ71758P/index\n./01F5ETH5T4MKTXJ1PEHQ71758P/chunks/\n./01F5ETH5T4MKTXJ1PEHQ71758P/chunks/000001\n./01F5ETH5T4MKTXJ1PEHQ71758P/meta.json\n./01F5ETH5T4MKTXJ1PEHQ71758P/tombstones\n...\n```\n\nAny attempts to restore this dump file will crash the target Prometheus with the\nabove error, complaining that files `000027` and `000028` are out-of-sequence.\n\nTo fix this dump file, we will have to manually delete those offending files:\n```sh\nmkdir temp\n\ntar -xvfz dump.tar.gz -C temp\n\nrm temp/chunks_head/000027 chunks_head/000029\n\ntar -C temp -czvf restored.tar.gz .\n```\n\nNow you can restore the `restored.tar.gz` file to your target Prometheus with:\n```\nkubectl promdump restore -p $POD_NAME -t restored.tar.gz\n```\n\nNote that deleting those head files may cause some head data to be lost.\n\n----\n## Limitations\n\npromdump is still in its experimental phase. SREs can use it to copy data blocks\nfrom one Prometheus instance to another development instance, while debugging\ncluster issues.\n\nBefore restoring the data dump, promdump will erase the content of the data\nfolder in the target Prometheus instance, to avoid corrupting the data blocks\ndue to conflicting segment error such as:\n\n```sh\nopening storage failed: get segment range: segments are not sequential\n```\n\nRestoring a data dump containing out-of-sequence head blocks will crash the\ntarget Prometheus. See [FAQ](#faq) on how to fix the data dump.\n\npromdump not suitable for production backup/restore operation.\n\nLike `kubectl cp`, promdump requires the `tar` binary to be installed in the\nPrometheus container.\n\n## Development\n\nTo run linters and unit test:\n```sh\nmake lint test\n```\n\nTo produce local builds:\n```sh\n# the promdump core\nmake core\n\n# the kubectl CLI plugin\nmake cli\n```\n\nTo test the `kubectl` plugin locally, the plugin manifest at\n`plugins/promdump.yaml` must be updated with the new checksums. Then run:\n```sh\nkubectl krew install --manifest=plugins/promdump.yaml --archive=target/releases/\u003crelease\u003e/plugins/kubectl-promdump-linux-amd64-\u003crelease\u003e.tar.gz\n```\n\n\nAll the binaries can be found in the local `target/bin` folder.\n\nTo install Prometheus via Helm:\n```sh\nmake hack/prometheus\n```\n\nTo create a release candidate:\n\n```sh\nmake release VERSION=\u003crc-name\u003e\n```\n\nTo do a release:\n```sh\ngit tag -a v$version\n\nmake release\n```\nAll the release artifacts can be found in the local `target/releases` folder.\nNote that the GitHub Actions pipeline uses the same make release targets.\n\n## License\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use\nthese files except in compliance with the License. You may obtain a copy of the\nLicense at:\n\n```\nhttp://www.apache.org/licenses/LICENSE-2.0\n```\n\nUnless required by applicable law or agreed to in writing, software distributed\nunder the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR\nCONDITIONS OF ANY KIND, either express or implied. See the License for the\nspecific language governing permissions and limitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fihcsim%2Fpromdump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fihcsim%2Fpromdump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fihcsim%2Fpromdump/lists"}