{"id":29670418,"url":"https://github.com/percona/percona-link-mongodb","last_synced_at":"2025-07-22T19:36:12.447Z","repository":{"id":294478203,"uuid":"897885695","full_name":"percona/percona-link-mongodb","owner":"percona","description":"Cluster-to-Cluster MongoDB replication","archived":false,"fork":false,"pushed_at":"2025-07-08T07:09:23.000Z","size":521,"stargazers_count":11,"open_issues_count":1,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-07-08T07:50:35.195Z","etag":null,"topics":["clone","database","replication","sync"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/percona.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-12-03T12:13:35.000Z","updated_at":"2025-07-08T07:09:28.000Z","dependencies_parsed_at":"2025-05-20T16:37:03.322Z","dependency_job_id":"3d086f9c-12c2-4749-9904-2c6d6db8bd11","html_url":"https://github.com/percona/percona-link-mongodb","commit_stats":null,"previous_names":["percona/percona-mongolink","percona/percona-link-mongodb"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/percona/percona-link-mongodb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpercona-link-mongodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpercona-link-mongodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpercona-link-mongodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpercona-link-mongodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/percona","download_url":"https://codeload.github.com/percona/percona-link-mongodb/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/percona%2Fpercona-link-mongodb/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266561555,"owners_count":23948632,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clone","database","replication","sync"],"created_at":"2025-07-22T19:36:11.294Z","updated_at":"2025-07-22T19:36:12.429Z","avatar_url":"https://github.com/percona.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Percona Link for MongoDB\n\nPercona Link for MongoDB is a tool for replicating data from a source MongoDB cluster to a target MongoDB cluster. It supports cloning data, replicating changes, and managing collections and indexes.\n\n## Features\n\n- **Clone**: Instantly transfer existing data from a source MongoDB to a target MongoDB.\n- **Real-Time Replication**: Tail the oplog to keep your target cluster up to date.\n- **Namespace Filtering**: Specify which databases and collections to include or exclude.\n- **Automatic Index Management**: Ensure necessary indexes are created on the target.\n- **HTTP API**: Start, finalize, pause, resume, and check replication status via REST endpoints.\n\n## Setup\n\n### Prerequisites\n\n- Go 1.24 or later\n- MongoDB 6.0 or later\n- Python 3.13 or later (for testing)\n- Poetry (for managing Python dependencies)\n\n### Installation\n\n1. Clone the repository:\n\n    ```sh\n    git clone https://github.com/percona/percona-link-mongodb.git\n    cd percona-link-mongodb\n    ```\n\n2. Build the project using the Makefile:\n\n    ```sh\n    make build\n    ```\n\n    Alternatively, you can install PLM from the cloned repo using `go install`:\n\n    ```sh\n    go install .\n    ```\n\n    \u003e This will install `plm` into your `GOBIN` directory. If `GOBIN` is included in your `PATH`, you can run Percona Link for MongoDB by typing `plm` in your terminal.\n\n3. Run the server:\n\n    ```sh\n    bin/plm --source \u003csource-mongodb-uri\u003e --target \u003ctarget-mongodb-uri\u003e\n    ```\n\n    Alternatively, you can use environment variables:\n\n    ```sh\n    export PLM_SOURCE_URI=\u003csource-mongodb-uri\u003e\n    export PLM_TARGET_URI=\u003ctarget-mongodb-uri\u003e\n    bin/plm\n    ```\n\n## Usage\n\n### Starting the Replication\n\nTo start the replication process, you can either use the command-line interface or send a POST request to the `/start` endpoint with the desired options:\n\n#### Using Command-Line Interface\n\n```sh\nbin/plm start\n```\n\n#### Using HTTP API\n\n```sh\ncurl -X POST http://localhost:2242/start -d '{\n    \"includeNamespaces\": [\"db1.collection1\", \"db2.collection2\"],\n    \"excludeNamespaces\": [\"db3.collection3\", \"db4.*\"]\n}'\n```\n\n### Finalizing the Replication\n\nTo finalize the replication process, you can either use the command-line interface or send a POST request to the `/finalize` endpoint:\n\n#### Using Command-Line Interface\n\n```sh\nbin/plm finalize\n```\n\n#### Using HTTP API\n\n```sh\ncurl -X POST http://localhost:2242/finalize\n```\n\n### Pausing the Replication\n\nTo pause the replication process, you can either use the command-line interface or send a POST request to the `/pause` endpoint:\n\n#### Using Command-Line Interface\n\n```sh\nbin/plm pause\n```\n\n#### Using HTTP API\n\n```sh\ncurl -X POST http://localhost:2242/pause\n```\n\n### Resuming the Replication\n\nTo resume the replication process, you can either use the command-line interface or send a POST request to the `/resume` endpoint:\n\n#### Using Command-Line Interface\n\n```sh\nbin/plm resume\n```\n\n#### Using HTTP API\n\n```sh\ncurl -X POST http://localhost:2242/resume\n```\n\n### Checking the Status\n\nTo check the current status of the replication process, you can either use the command-line interface or send a GET request to the `/status` endpoint:\n\n#### Using Command-Line Interface\n\n```sh\nbin/plm status\n```\n\n#### Using HTTP API\n\n```sh\ncurl http://localhost:2242/status\n```\n\n## PLM Options\n\nWhen starting the PLM server, you can use the following options:\n\n- `--port`: The port on which the server will listen (default: 2242)\n- `--source`: The MongoDB connection string for the source cluster\n- `--target`: The MongoDB connection string for the target cluster\n- `--log-level`: The log level (default: \"info\")\n- `--log-json`: Output log in JSON format with disabled color\n- `--no-color`: Disable log ASCI color\n\nExample:\n\n```sh\nbin/plm \\\n    --source \u003csource-mongodb-uri\u003e \\\n    --target \u003ctarget-mongodb-uri\u003e \\\n    --port 2242 \\\n    --log-level debug \\\n    --log-json\n```\n\n## Log JSON Fields\n\nWhen using the `--log-json` option, the logs will be output in JSON format with the following fields:\n\n- `time`: Unix time when the log entry was created.\n- `level`: Log level (e.g., \"debug\", \"info\", \"warn\", \"error\").\n- `message`: Log message, if any.\n- `error`: Error message, if any.\n- `s`: Scope of the log entry.\n- `ns`: Namespace (database.collection format).\n- `elapsed_secs`: The duration in seconds for the specific operation to complete.\n\nExample:\n\n```json\n{ \"level\": \"info\",\n  \"s\": \"clone\",\n  \"ns\": \"db_1.coll_1\",\n  \"elapsed_secs\": 0,\n  \"time\": \"2025-02-23 11:26:03.758\",\n  \"message\": \"Cloned db_1.coll_1\" }\n\n{ \"level\": \"info\",\n  \"s\": \"plm\",\n  \"elapsed_secs\": 0,\n  \"time\": \"2025-02-23 11:26:03.857\",\n  \"message\": \"Change replication stopped at 1740335163.1740335163 source cluster time\" }\n```\n\n## HTTP API\n\n### POST /start\n\nStarts the replication process.\n\n#### Request Body\n\n- `includeNamespaces` (optional): List of namespaces to include in the replication.\n- `excludeNamespaces` (optional): List of namespaces to exclude from the replication.\n\nExample:\n\n```json\n{\n    \"includeNamespaces\": [\"dbName.*\", \"anotherDB.collName1\", \"anotherDB.collName2\"],\n    \"excludeNamespaces\": [\"dbName.collName\"]\n}\n```\n\n#### Response\n\n- `ok`: Boolean indicating if the operation was successful.\n- `error` (optional): Error message if the operation failed.\n\nExample:\n\n```json\n{ \"ok\": true }\n```\n\n### POST /finalize\n\nFinalizes the replication process.\n\n#### Response\n\n- `ok`: Boolean indicating if the operation was successful.\n- `error` (optional): Error message if the operation failed.\n\nExample:\n\n```json\n{ \"ok\": true }\n```\n\n### POST /pause\n\nPauses the replication process.\n\n#### Response\n\n- `ok`: Boolean indicating if the operation was successful.\n- `error` (optional): Error message if the operation failed.\n\nExample:\n\n```json\n{ \"ok\": true }\n```\n\n### POST /resume\n\nResumes the replication process.\n\n#### Request Body\n\n- `fromFailure` (optional): Allows PLM to resume from failed state\n\nExample:\n\n```json\n{\n    \"fromFailure\": true\n}\n```\n\n#### Response\n\n- `ok`: Boolean indicating if the operation was successful.\n- `error` (optional): Error message if the operation failed.\n\nExample:\n\n```json\n{ \"ok\": true }\n```\n\n### GET /status\n\nThe /status endpoint provides the current state of the PLM replication process, including its progress, lag, and event processing details.\n\n#### Response\n\n- `ok`: indicates if the operation was successful.\n- `state`: the current state of the replication.\n- `info`: provides additional information about the current state.\n- `error` (optional): the error message if the operation failed.\n\n- `lagTime`: the current lag time in logical seconds between source and target clusters.\n- `eventsProcessed`: the number of events processed.\n- `lastReplicatedOpTime`: the last replicated operation time.\n\n- `initialSync.completed`: indicates if the initial sync is completed.\n- `initialSync.lagTime`: the lag time in logical seconds until the initial sync completed.\n\n- `initialSync.cloneCompleted`: indicates if the cloning process is completed.\n- `initialSync.estimatedCloneSize`: the estimated total size of the clone.\n- `initialSync.clonedSize`: the size of the data that has been cloned.\n\nExample:\n\n```json\n{\n    \"ok\": true,\n    \"state\": \"running\",\n    \"info\": \"Initial Sync\",\n\n    \"lagTime\": 22,\n    \"eventsProcessed\": 5000,\n    \"lastReplicatedOpTime\": \"1740335200.5\",\n\n    \"initialSync\": {\n        \"completed\": false,\n        \"lagTime\": 5,\n\n        \"cloneCompleted\": false,\n        \"estimatedCloneSize\": 5000000000,\n        \"clonedSize\": 2500000000\n    }\n}\n```\n\n## Testing\n\n### Prerequisites\n\n- Install Poetry:\n\n    ```sh\n    curl -sSL https://install.python-poetry.org | python3 -\n    ```\n\n- Install the required Python packages:\n\n    ```sh\n    poetry install\n    ```\n\n### Build for Testing\n\nTo build the project for testing, use the following command:\n\n```sh\nmake test-build\n```\n\n### Running Tests\n\nTo run the tests, use the following command:\n\n```sh\npoetry run pytest \\\n    --source-uri \u003csource-mongodb-uri\u003e \\\n    --target-uri \u003ctarget-mongodb-uri\u003e \\\n    --plm_url http://localhost:2242 \\\n    --plm-bin bin/plm_test\n```\n\nAlternatively, you can use environment variables:\n\n```sh\nexport TEST_SOURCE_URI=\u003csource-mongodb-uri\u003e\nexport TEST_TARGET_URI=\u003ctarget-mongodb-uri\u003e\nexport TEST_PLM_URL=http://localhost:2242\nexport TEST_PLM_BIN=bin/plm_test\npoetry run pytest\n```\n\n\u003e The `--plm-bin` flag or `TEST_PLM_BIN` environment variable specifies the path to the PLM binary. This allows the test suite to manage the PLM process, ensuring it starts and stops as needed during the tests. If neither the flag nor the environment variable is provided, you must run PLM externally before running the tests.\n\n## Contributing\n\nContributions are welcome. Please open a [JIRA](https://perconadev.atlassian.net/jira/software/c/projects/PLM/issues) issue describing the proposed change, then submit a pull request on GitHub.\n\n## License\n\nThis project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercona%2Fpercona-link-mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpercona%2Fpercona-link-mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpercona%2Fpercona-link-mongodb/lists"}