{"id":29670422,"url":"https://github.com/percona/pmm-dump","last_synced_at":"2025-07-22T19:36:13.915Z","repository":{"id":41954396,"uuid":"382074375","full_name":"percona/pmm-dump","owner":"percona","description":"Export metrics or QAN data from or import them to PMM","archived":false,"fork":false,"pushed_at":"2025-06-18T14:08:51.000Z","size":4932,"stargazers_count":6,"open_issues_count":13,"forks_count":10,"subscribers_count":17,"default_branch":"main","last_synced_at":"2025-06-18T14:25:01.745Z","etag":null,"topics":["export","metrics","pmm","qan"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/percona.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":"support-files/README.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-07-01T15:15:37.000Z","updated_at":"2025-06-18T14:07:25.000Z","dependencies_parsed_at":"2023-02-08T19:16:12.259Z","dependency_job_id":"d381ab10-8e13-4686-945e-d9b121a7d407","html_url":"https://github.com/percona/pmm-dump","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/percona/pmm-dump","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpmm-dump","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpmm-dump/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpmm-dump/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpmm-dump/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/percona","download_url":"https://codeload.github.com/percona/pmm-dump/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpmm-dump/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266561555,"owners_count":23948632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["export","metrics","pmm","qan"],"created_at":"2025-07-22T19:36:13.363Z","updated_at":"2025-07-22T19:36:13.907Z","avatar_url":"https://github.com/percona.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PMM Dump (pmm-import-export-tool)\n\nPMM Dump is a tool that allows to transfer metrics and QAN data from one PMM Server instance to another. It helps Percona Services engineers troubleshoot issues.\n\n## How to build?\n\nYou will need to have Go 1.21+ installed.\n\nIn the root directory: `make build`\n\n## Using PMM Dump\n\nThe transfer process is split into two main parts: export and import.\n\nIn order to run either export or import, you have to specify PMM URL with credentials at least:\n```\n./pmm-dump export --pmm-url \"http://USER:PASS@HOST\"\n./pmm-dump import --pmm-url \"http://USER:PASS@HOST\" --dump-path FILENAME.tar.gz\n```\nAlso, you can use credentials flags or envars:\n```\n./pmm-dump export --pmm-url \"http://HOST\" --pmm-user USER --pmm-pass PASS\nPMM_USER=USER PMM_PASS=PASS ./pmm-dump import --pmm-url \"http://HOST\" --dump-path FILENAME.tar.gz\n```\n\nHere are main commands/flags:\n\n\n| Command   | Flag                 | Description                                                                                               | Example                                                                                                    |\n|-----------|----------------------|-----------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------------------------------------------- |\n| any       | pmm-url              | URL of PMM instance. Envar: `PMM_URL`                                                                     | `http://admin:admin@localhost`                                                                             |\n| any       | pmm-host             | Host of PMM instance(with scheme). Envar: `PMM_HOST`                                                      | `http://localhost`                                                                                         |\n| any       | pmm-port             | Port of PMM instance. Envar: `PMM_PORT`                                                                   | `80`                                                                                                       |\n| any       | pmm-user             | PMM credentials user. Envar: `PMM_USER`                                                                   | -                                                                                                          |\n| any       | pmm-pass             | PMM credentials password. Envar: `PMM_PASS`                                                               | -                                                                                                          |\n| any       | pmm-token            | PMM API token. Envar: `PMM_TOKEN`                                                                         |                                                                                                            |\n| any       | pmm-cookie           | PMM auth cookie value. Envar: `PMM_COOKIE`                                                                 |                                                                                                            |\n| any       | dump-core            | Process core metrics                                                                                      | -                                                                                                          |\n| any       | dump-qan             | Process QAN metrics                                                                                       | -                                                                                                          |\n| any       | workers              | Set the number of import/export workers                                                                   | `4`                                                                                                        |\n| export    | start-ts             | Start date-time to limit timeframe (in [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) format)            | `2006-01-02T15:04:05Z` (please note that you can't use offset for UTC time)\u003cbr\u003e`2006-01-02T15:04:05-07:00` |\n| export    | end-ts               | End date-time to limit timeframe (in [RFC3339](https://www.ietf.org/rfc/rfc3339.txt) format)              | `2006-01-02T15:04:05Z` (please note that you can't use offset for UTC time)\u003cbr\u003e`2006-01-02T15:04:05-07:00` |\n| export    | ignore-load          | Disable checking for load values                                                                          | -                                                                                                          |\n| export    | max-load             | Max value of a metric to postpone export                                                                  | `CPU=50,RAM=50,MYRAM=10`                                                                                   |\n| export    | critical-load        | Max value of a metric to stop export                                                                      | `CPU=70,RAM=70,MYRAM=30`                                                                                   |\n| export    | stdout               | Redirect output to STDOUT                                                                                 | -                                                                                                          |\n| export    | vm-native-data       | Use VictoriaMetrics' native export format. Reduces dump size, but can be incompatible between PMM versions | -                                                                                                          |\n| import    | vm-content-limit     | Limit the chunk content size for VictoriaMetrics (in bytes). Doesn't work with native format              | `1024`                                                                                                     |\n| any       | dump-path, d         | Path to dump file                                                                                         | `/tmp/pmm-dumps/pmm-dump-1624342596.tar.gz`                                                                |\n| any       | verbose, v           | Enable verbose (debug) mode                                                                               | -                                                                                                          |\n| any       | allow-insecure-certs | For self-signed certificates                                                                              | -                                                                                                          |\n| show-meta | -                    | Shows dump meta in human readable format                                                                  | -                                                                                                          |\n| show-meta | no-prettify          | Shows raw dump meta                                                                                       | -                                                                                                          |\n| version   | -                    | Shows binary version                                                                                      | -                                                                                                          |\n\n\nFor filtering you could use the following commands (will be improved in the future):\n\n| Command | Flag        | Description                       | Example                      |\n| ------- | ----------- | --------------------------------- | ---------------------------- |\n| export  | ts-selector | Timeseries selector (for VM only) | `{service_name=\"mongo\"}`     |\n| export  | where       | WHERE statement (for CH only)     | `service_name='mongo'`       |\n| export  | dashboard   | Dashboard name (for VM only)      | `MongoDB Instances Overview` |\n| export  | instance    | Filter by service name            | `mongo`                      |\n\nYou could filter by instance using service name or id. For example, we have registered the following mongodb instance:\n\n```\n\u003e pmm-admin add mongodb --username=pmm_mongodb --password=password mongo mongodb:27017\nMongoDB Service added.\nService ID  : 6d7fbaa0-6b21-4c3f-a4a7-4be1e4f58b11\nService name: mongo\n```\n\nSo the value of `ts-selector` would be: `{service_name=\"mongo\"}` or `{service_id=\"6d7fbaa0-6b21-4c3f-a4a7-4be1e4f58b11\"}`.\nThe same for `where` QAN filter: `service_name='mongo'` or `service_id='6d7fbaa0-6b21-4c3f-a4a7-4be1e4f58b11'`.\nNote: On version 2 value of `ts-selector` would be: \n`{service_name=\"mongo\"}` or `{service_id=\"/service_id/6d7fbaa0-6b21-4c3f-a4a7-4be1e4f58b11\"}` \nand QAN filter: \n`service_name='mongo'` or `service_id='/service_id/6d7fbaa0-6b21-4c3f-a4a7-4be1e4f58b11'`.\nAlso, you can use `instance` option which filters QAN and core metrics by service name\n\n```\n\u003e ./pmm-dump export --pmm-url=\"http://admin:admin@localhost:8282\" --ts-selector=`{service_name=\"mongo\"}` --dump-qan --where=`service_name='mongo'`\n```\nis same as\n```\n\u003e ./pmm-dump export --pmm-url=\"http://admin:admin@localhost:8282\" --instance=\"mongo\" --dump-qan\n```\n\nTo filter by multiple dashboards, you can use `dashboard` flag multiple times:\n```\n\u003e ./pmm-dump export --pmm-url=\"http://admin:admin@localhost:8282\" --dashboard='MongoDB Instances Overview' --dashboard='MySQL Instances Overview'`\n```\n\nIn some cases you would need to override default configuration for VM/CH processing:\n\n| Command | Flag                 | Description                                         | Example                                        |\n| ------- | -------------------- | --------------------------------------------------- | ---------------------------------------------- |\n| any     | victoria-metrics-url | URL of Victoria Metrics                             | `http://admin:admin@localhost:8282/prometheus` |\n| any     | click-house-url      | URL of Click House                                  | `http://localhost:9000?database=pmm`           |\n| export  | chunk-time-range     | Time range to be fit into a single chunk (VM only)  | `45s`, `5m`, `1h`                              |\n| export  | chunk-rows           | Amount of rows to fit into a single chunk (CH only) | `1000`                                         |\n\n### Using in pipelines\nYou can redirect output to STDOUT with --stdout option. It's useful to redirect output to another pmm-dump in a pipeline:\n```\n\u003e ./pmm-dump export --pmm-url=\"http://admin:admin@localhost:8282\" --dump-qan --stdout | ./pmm-dump import --pmm-url=\"http://admin:admin@localhost:8282\" --dump-qan \n```\n\n### Stop or postpone during export\nYou can set threshold values to stop or postpone pmm-dump during export using `max-load` and `critical-load` options.\n\nThe syntax for these options is following:\n\n```\n\u003cthreshold\u003e=\u003cpercent_value\u003e\n```\n\nYou can provide multiple threshold values separated by commas. For example:\n\n``` \n--max-load='CPU=100,RAM=30'\n```\nAvailable thresholds:\n- `CPU` - CPU load of PMM instance in percents (0-100)\n- `RAM` - RAM load of PMM instance in percents (0-100)\n- `MYRAM` - RAM load of instance which uses pmm-dump in percents (0-100)\n\n## About the dump file\n\nDump file is a `tar` archive compressed via `gzip`. Here is the shape of dump file:\n\n* `dump.tar.gz/meta.json` - contains metadata about the dump (JSON object)\n* `dump.tar.gz/vm/` - contains Victoria Metrics data chunks split by timeframe (in native VM format)\n* `dump.tar.gz/ch/` - contains ClickHouse data chunks split by rows count (in TSV format)\n\n\n## Using Makefile for local development environment\n\nThere is a Makefile that contains commands to build and test pmm-dump locally. It uses docker-compose to set up PMM Server, PMM Client and MongoDB.\n\nYou will need to have Go 1.21+ and Docker installed.\n\n| Rule                | Description                  |\n| ------------------- | ---------------------------- |\n| make                | Shortcut for fast test       |\n| make build          | Builds pmm-dump binary       |\n| make up             | Sets up docker containers    |\n| mongo-reg           | Registers MongoDB in PMM     |\n| mongo-insert        | Executes MongoDB insert      |\n| make down           | Shuts down docker containers |\n| make re             | Shortcut for `down up`       |\n| make export-all     | Runs export from local PMM   |\n| make run-tests      | Runs all tests               |\n| make run-e2e-tests  | Runs all e2e tests           |\n| make run-e2e-tests-v2| Runs all e2e tests for version 2|\n| make run-unit-tests | Runs all unit tests          |\n\nRead `Makefile` for more.\n\n## Running End-to-End Tests\n\nFor detailed instructions on executing end-to-end tests, refer to [Executing e2e tests](./internal/test/README.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercona%2Fpmm-dump","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpercona%2Fpmm-dump","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercona%2Fpmm-dump/lists"}