{"id":18521714,"url":"https://github.com/transferwise/pipelinewise-tap-mongodb","last_synced_at":"2026-01-14T08:43:46.995Z","repository":{"id":37976229,"uuid":"260185072","full_name":"transferwise/pipelinewise-tap-mongodb","owner":"transferwise","description":"Singer.io Tap for MongoDB - PipelineWise compatible","archived":true,"fork":false,"pushed_at":"2024-09-02T14:21:15.000Z","size":111,"stargazers_count":5,"open_issues_count":13,"forks_count":24,"subscribers_count":63,"default_branch":"master","last_synced_at":"2025-09-22T01:56:34.332Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/transferwise.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-30T10:46:46.000Z","updated_at":"2024-09-19T23:18:00.000Z","dependencies_parsed_at":"2024-06-18T16:30:57.541Z","dependency_job_id":"7696a27f-ccbb-4ee5-8856-772e70946575","html_url":"https://github.com/transferwise/pipelinewise-tap-mongodb","commit_stats":{"total_commits":46,"total_committers":6,"mean_commits":7.666666666666667,"dds":0.6739130434782609,"last_synced_commit":"6dbc3ec64a36209e1c317fe49ed2ff7eea515d43"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/transferwise/pipelinewise-tap-mongodb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-mongodb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-mongodb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-mongodb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-mongodb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/transferwise","download_url":"https://codeload.github.com/transferwise/pipelinewise-tap-mongodb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/transferwise%2Fpipelinewise-tap-mongodb/sbom","scorecard":{"id":896915,"data":{"date":"2025-08-11","repo":{"name":"github.com/transferwise/pipelinewise-tap-mongodb","commit":"9745009f125347cb7464a377e97cd546161ddbe9"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":5.4,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yaml:1","Warn: no topLevel permission defined: .github/workflows/pythonpublish.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":10,"reason":"all changesets reviewed","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Maintained","score":0,"reason":"project is archived","details":["Warn: Repository is archived."],"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Info: Possibly incomplete results: error parsing shell code: \"foo(\" must be followed by ): bin/setup_local_db.sh:31","Info: Possibly incomplete results: error parsing shell code: \"foo(\" must be followed by ): bin/setup_local_db.sh:34","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yaml:21: update your workflow using https://app.stepsecurity.io/secureworkflow/transferwise/pipelinewise-tap-mongodb/ci.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yaml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/transferwise/pipelinewise-tap-mongodb/ci.yaml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpublish.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/transferwise/pipelinewise-tap-mongodb/pythonpublish.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/pythonpublish.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/transferwise/pipelinewise-tap-mongodb/pythonpublish.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/pythonpublish.yml:29: update your workflow using https://app.stepsecurity.io/secureworkflow/transferwise/pipelinewise-tap-mongodb/pythonpublish.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/pythonpublish.yml:23","Warn: pipCommand not pinned by hash: .github/workflows/pythonpublish.yml:24","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   2 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: GNU Affero General Public License v3.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Security-Policy","score":10,"reason":"security policy file detected","details":["Info: security policy file detected: github.com/transferwise/.github/SECURITY.md:1","Info: Found linked content: github.com/transferwise/.github/SECURITY.md:1","Info: Found disclosure, vulnerability, and/or timelines in security policy: github.com/transferwise/.github/SECURITY.md:1","Info: Found text in security policy: github.com/transferwise/.github/SECURITY.md:1"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Branch-Protection","score":4,"reason":"branch protection is not maximal on development and all release branches","details":["Info: 'allow deletion' disabled on branch 'master'","Info: 'force pushes' disabled on branch 'master'","Warn: 'branch protection settings apply to administrators' is disabled on branch 'master'","Info: 'stale review dismissal' is required to merge on branch 'master'","Warn: required approving review count is 1 on branch 'master'","Info: codeowner review is required on branch 'master'","Warn: 'last push approval' is disabled on branch 'master'","Warn: 'up-to-date branches' is disabled on branch 'master'","Info: status check found to merge onto on branch 'master'","Info: PRs are required in order to make changes on branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-24T14:04:54.471Z","repository_id":37976229,"created_at":"2025-08-24T14:04:54.471Z","updated_at":"2025-08-24T14:04:54.471Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28414692,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T08:38:59.149Z","status":"ssl_error","status_checked_at":"2026-01-14T08:38:43.588Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T17:27:10.615Z","updated_at":"2026-01-14T08:43:46.978Z","avatar_url":"https://github.com/transferwise.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Notice\nTo better serve Wise business and customer needs, the PipelineWise codebase needs to shrink.\nWe have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo.\nThe last version before this decision is [v0.64.1](https://github.com/transferwise/pipelinewise/tree/v0.64.1)\n\nWe thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily\n\n# pipelinewise-tap-mongodb\n\nThis is a [Singer](https://singer.io) tap that produces JSON-formatted data following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md) from a MongoDB source.\n\n## Set up local dev environment:\n\n```shell script\nmake setup\n```\n\n## Activate virtual environment\n\n```shell script\n. venv/bin/activate\n```\n\n## Set up Config file\n\nCreate json file called `config.json`, with the following contents:\n```json\n{\n  \"password\": \"\u003cpassword\u003e\",\n  \"user\": \"\u003cusername\u003e\",\n  \"host\": \"\u003chost ip address\u003e\",\n  \"auth_database\": \"\u003cdatabase name to authenticate on\u003e\",\n  \"database\": \"\u003cdatabase name to sync from\u003e\"\n}\n```\nThe following parameters are optional for your config file:\n\n| Name | Type | Default value| Description |\n| -----|------|--------|------------ |\n| `srv` | Boolean | false | uses a `mongodb+srv` protocol to connect. Disables the usage of `port` argument if set to `True` |\n| `port` | Integer | false | Connection port. Required if a non-srv connection is being used.  |\n| `replica_set` | string | null | name of replica set |\n| `ssl` | Boolean | false | can be set to true to connect using ssl |\n| `verify_mode` | Boolean | true | Default SSL verify mode |\n| `include_schemas_in_destination_stream_name` | Boolean |false  | forces the stream names to take the form `\u003cdatabase_name\u003e-\u003ccollection_name\u003e` instead of `\u003ccollection_name\u003e`|\n| `update_buffer_size` | int | 1 | [LOG_BASED] The size of the buffer that holds detected update operations in memory, the buffer is flushed once the size is reached |\n| `await_time_ms` | int | 1000 | [LOG_BASED] The maximum amount of time in milliseconds the loge_base method waits for new data changes before exiting. |\n\nAll of the above attributes are required by the tap to connect to your mongo instance.\nhere is a [sample configuration file](./sample_config.json).\n\n## Run in discovery mode\nRun the following command and redirect the output into the catalog file\n```shell script\ntap-mongodb --config ~/config.json --discover \u003e ~/catalog.json\n```\n\nYour catalog file should now look like this:\n```json\n{\n  \"streams\": [\n    {\n      \"table_name\": \"\u003ctable name\u003e\",\n      \"tap_stream_id\": \"\u003ctap_stream_id\u003e\",\n      \"metadata\": [\n        {\n          \"breadcrumb\": [],\n          \"metadata\": {\n            \"row-count\":\u003cint\u003e,\n            \"is-view\": \u003cbool\u003e,\n            \"database-name\": \"\u003cdatabase name\u003e\",\n            \"table-key-properties\": [\n              \"_id\"\n            ],\n            \"valid-replication-keys\": [\n              \"_id\"\n            ]\n          }\n        }\n      ],\n      \"stream\": \"\u003cstream name\u003e\",\n      \"schema\": {\n        \"type\": \"object\"\n      }\n    }\n  ]\n}\n```\n\n## Edit Catalog file\n### Using valid json, edit the config.json file\nTo select a stream, enter the following to the stream's metadata:\n```json\n\"selected\": true,\n\"replication-method\": \"\u003creplication method\u003e\",\n```\n\n`\u003creplication-method\u003e` must be either `FULL_TABLE`, `INCREMENTAL` or `LOG_BASED`, if it's `INCREMENTAL`, make sure to add a `\"replication-key\"`.\n\n\nFor example, if you were to edit the example stream to select the stream as well as add a projection, config.json should look this:\n```json\n{\n  \"streams\": [\n    {\n      \"table_name\": \"\u003ctable name\u003e\",\n      \"tap_stream_id\": \"\u003ctap_stream_id\u003e\",\n      \"metadata\": [\n        {\n          \"breadcrumb\": [],\n          \"metadata\": {\n            \"row-count\": \u003cint\u003e,\n            \"is-view\": \u003cbool\u003e,\n            \"database-name\": \"\u003cdatabase name\u003e\",\n            \"table-key-properties\": [\n              \"_id\"\n            ],\n            \"valid-replication-keys\": [\n              \"_id\"\n            ],\n            \"selected\": true,\n            \"replication-method\": \"\u003creplication method\u003e\"\n          }\n        }\n      ],\n      \"stream\": \"\u003cstream name\u003e\",\n      \"schema\": {\n        \"type\": \"object\"\n      }\n    }\n  ]\n}\n\n```\n## Run in sync mode:\n```shell script\ntap-mongodb --config ~/config.json --catalog ~/catalog.json\n```\n\nThe tap will write bookmarks to stdout which can be captured and passed as an optional `--state state.json` parameter to the tap for the next sync.\n\n## Logging configuration\nThe tap uses a predefined logging config if none is provided, however, you can set your own config by setting the environment variable `LOGGING_CONFIG_FILE` as the path to the logging config.\nA sample config is available [here](./sample_logging.conf).\n\n---\n\nCopyright \u0026copy; 2020 TransferWise\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-mongodb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-mongodb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftransferwise%2Fpipelinewise-tap-mongodb/lists"}