{"id":13509884,"url":"https://github.com/compose/transporter","last_synced_at":"2025-09-29T00:32:25.869Z","repository":{"id":21892706,"uuid":"25216563","full_name":"compose/transporter","owner":"compose","description":"Sync data between persistence engines, like ETL only not stodgy","archived":true,"fork":false,"pushed_at":"2023-10-17T15:44:41.000Z","size":24009,"stargazers_count":1448,"open_issues_count":1,"forks_count":215,"subscribers_count":56,"default_branch":"master","last_synced_at":"2025-01-11T10:43:19.666Z","etag":null,"topics":["elasticsearch","etl","go","mongodb","mysql","postgresql","rabbitmq","rethinkdb"],"latest_commit_sha":null,"homepage":"https://github.com/compose/transporter/issues/523","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/compose.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-10-14T16:48:05.000Z","updated_at":"2025-01-03T21:44:15.000Z","dependencies_parsed_at":"2024-06-18T18:36:33.264Z","dependency_job_id":"b5667357-954d-4c5f-a694-b92d9c6f62aa","html_url":"https://github.com/compose/transporter","commit_stats":{"total_commits":353,"total_committers":34,"mean_commits":"10.382352941176471","dds":0.6657223796033994,"last_synced_commit":"add7994ff74d7ebaa6bd3694baa2ee672b22a6bb"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compose%2Ftransporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compose%2Ftransporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compose%2Ftransporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/compose%2Ftransporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/compose","download_url":"https://codeload.github.com/compose/transporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234575225,"owners_count":18854925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","etl","go","mongodb","mysql","postgresql","rabbitmq","rethinkdb"],"created_at":"2024-08-01T02:01:15.956Z","updated_at":"2025-09-29T00:32:18.944Z","avatar_url":"https://github.com/compose.png","language":"Go","funding_links":[],"categories":["Go","go","postgresql","Repositories"],"sub_categories":[],"readme":"[![Go Report Card](https://goreportcard.com/badge/github.com/compose/transporter)](https://goreportcard.com/report/github.com/compose/transporter) [![codecov](https://codecov.io/gh/compose/transporter/branch/master/graph/badge.svg)](https://codecov.io/gh/compose/transporter) [![Docker Repository on Quay](https://quay.io/repository/compose/transporter/status \"Docker Repository on Quay\")](https://quay.io/repository/compose/transporter) [![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/compose-transporter/Lobby)\n\nCompose Transporter helps with database transformations from one store to another.  It can also sync from one to another or several stores.\n\nTransporter\n===========\n\nThis has been archived. See this [issue](https://github.com/compose/transporter/issues/523).\n\nWarning About v1.0.0\n------------\n\nThis version officially only supports the mongodb and postgresql adaptors. Support for other DBs will be added later on. Other adaptors may or may not work. You're encouraged to still use v0.5.2 for non mongo/postgres migrations.\n\nAbout\n-----\n\nTransporter allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow.\n\nAdaptors may be able to track changes as they happen in source data. This \"tail\" capability allows a Transporter to stay running and keep the sinks in sync.\n\n***BETA Feature***\n\nAs of release `v0.4.0`, transporter contains support for being able to resume operations\nafter being stopped. The feature is disabled by default and can be enabled with the following:\n\n```\nsource = mongodb({\"uri\": \"mongo://localhost:27017/source_db\"})\nsink = mongodb({\"uri\": \"mongo://localhost:27017/sink_db\"})\nt.Config({\"xlog_dir\":\"/data/transporter\"})\n  .Source(\"source\", source)\n  .Save(\"sink\", sink)\n```\n\nWhen using the above pipeline, all messages will be appended to a commit log and\nsuccessful processing of a message is handled via consumer/sink offset tracking.\n\nBelow is a list of each adaptor and its support of the feature:\n\n```\n+---------------+-------------+----------------+\n|    adaptor    | read resume | write tracking |\n+---------------+-------------+----------------+\n| elasticsearch |     N/A     |       X        |\n|     file      |             |       X        |\n|    mongodb    |      X      |       X        |\n|  postgresql   |             |       X        |\n|     mysql     |             |       X        |\n|   rabbitmq    |      X      |                |\n|   rethinkdb   |             |       X        |\n+---------------+-------------+----------------+\n```\n\nDownloading Transporter\n-----------------------\n\nThe latest binary releases are available from the [Github Repository](https://github.com/compose/transporter/releases/latest)\n\nAdaptors\n--------\n\nEach adaptor has its own README page with details on configuration and capabilities.\n\n* [elasticsearch](./adaptor/elasticsearch)\n* [file](./adaptor/file)\n* [mongodb](./adaptor/mongodb)\n* [postgresql](./adaptor/postgres)\n* [rabbitmq](./adaptor/rabbitmq)\n* [rethinkdb](./adaptor/rethinkdb)\n* [mysql](./adaptor/mysql)\n\nNative Functions\n----------------\n\nEach native function can be used as part of a `Transform` step in the pipeline.\n\n* [goja](./function/gojajs)\n* [omit](./function/omit)\n* [opfilter](./function/opfilter)\n* [otto](./function/ottojs)\n* [pick](./function/pick)\n* [pretty](./function/pretty)\n* [rename](./function/rename)\n* [remap](./function/remap)\n* [skip](./function/skip)\n\nCommands\n--------\n\n### init\n\n```\ntransporter init [source adaptor name] [sink adaptor name]\n```\n\nGenerates a basic `pipeline.js` file in the current directory.\n\n_Example_\n```\n$ transporter init mongodb elasticsearch\n$ cat pipeline.js\nvar source = mongodb({\n  \"uri\": \"${MONGODB_URI}\"\n  // \"timeout\": \"30s\",\n  // \"tail\": false,\n  // \"ssl\": false,\n  // \"cacerts\": [\"/path/to/cert.pem\"],\n  // \"wc\": 1,\n  // \"fsync\": false,\n  // \"bulk\": false,\n  // \"collection_filters\": \"{}\",\n  // \"read_preference\": \"Primary\"\n})\n\nvar sink = elasticsearch({\n  \"uri\": \"${ELASTICSEARCH_URI}\"\n  // \"timeout\": \"10s\", // defaults to 30s\n  // \"aws_access_key\": \"ABCDEF\", // used for signing requests to AWS Elasticsearch service\n  // \"aws_access_secret\": \"ABCDEF\" // used for signing requests to AWS Elasticsearch service\n})\n\nt.Source(source).Save(sink)\n// t.Source(\"source\", source).Save(\"sink\", sink)\n// t.Source(\"source\", source, \"namespace\").Save(\"sink\", sink, \"namespace\")\n$\n```\n\nEdit the `pipeline.js` file to configure the source and sink nodes and also to set the namespace.\n\n### about\n\n`transporter about`\n\nLists all the adaptors currently available.\n\n_Example_\n```\nelasticsearch - an elasticsearch sink adaptor\nfile - an adaptor that reads / writes files\nmongodb - a mongodb adaptor that functions as both a source and a sink\npostgres - a postgres adaptor that functions as both a source and a sink\nrabbitmq - an adaptor that handles publish/subscribe messaging with RabbitMQ\nrethinkdb - a rethinkdb adaptor that functions as both a source and a sink\n```\n\nGiving the name of an adaptor produces more detail, such as the sample configuration.\n\n_Example_\n```\ntransporter about postgres\npostgres - a postgres adaptor that functions as both a source and a sink\n\n Sample configuration:\n{\n  \"uri\": \"${POSTGRESQL_URI}\"\n  // \"debug\": false,\n  // \"tail\": false,\n  // \"replication_slot\": \"slot\"\n}\n```\n\n### run\n\n```\ntransporter run [-log.level \"info\"] \u003capplication.js\u003e\n```\n\nRuns the pipeline script file which has its name given as the final parameter.\n\n### test\n\n```\ntransporter test [-log.level \"info\"] \u003capplication.js\u003e\n```\n\nEvaluates and connects the pipeline, sources and sinks. Establishes connections but does not run.\nPrints out the state of connections at the end. Useful for debugging new configurations.\n\n### xlog\n\nThe `xlog` command is useful for inspecting the current state of the commit log.\nIt contains 3 subcommands, `current`, `oldest`, and `offset`, as well as\na required flag `-xlog_dir` which should be the path to where the commit log is stored.\n\n***NOTE*** the command should only be run against the commit log when transporter\nis not actively running.\n\n```\ntransporter xlog -xlog_dir=/path/to/dir current\n12345\n```\n\nReturns the most recent offset appended to the commit log.\n\n```\ntransporter xlog -xlog_dir=/path/to/dir oldest\n0\n```\n\nReturns the oldest offset in the commit log.\n\n```\ntransporter xlog -xlog_dir=/path/to/dir show 0\noffset    : 0\ntimestamp : 2017-05-16 11:00:20 -0400 EDT\nmode      : COPY\nop        : INSERT\nkey       : MyCollection\nvalue     : {\"_id\":{\"$oid\":\"58efd14b60d271d7457b4f24\"},\"i\":0}\n```\n\nPrints out the entry stored at the provided offset.\n\n### offset\n\nThe `offset` command provides access to current state of each consumer (i.e. sink)\noffset. It contains 4 subcommands, `list`, `show`, `mark`, and `delete`, as well as\na required flag `-xlog_dir` which should be the path to where the commit log is stored.\n\n```\ntransporter offset -xlog_dir=/path/to/dir list\n+------+---------+\n| SINK | OFFSET  |\n+------+---------+\n| sink | 1103003 |\n+------+---------+\n```\n\nLists all consumers and their associated offset in `xlog_dir`.\n\n```\ntransporter offset -xlog_dir=/path/to/dir show sink\n+-------------------+---------+\n|     NAMESPACE     | OFFSET  |\n+-------------------+---------+\n| newCollection     | 1102756 |\n| testC             | 1103003 |\n| MyCollection      |  999429 |\n| anotherCollection | 1002997 |\n+-------------------+---------+\n```\n\nPrints out each namespace and its associated offset.\n\n```\ntransporter offset -xlog_dir=/path/to/dir mark sink 1\nOK\n```\n\nRewrites the namespace offset map based on the provided offset.\n\n```\ntransporter offset -xlog_dir=/path/to/dir delete sink\nOK\n```\n\nRemoves the consumer (i.e. sink) log directory.\n\n#### flags\n\n`-log.level \"info\"` - sets the logging level. This is application logging and is unrelated to the commit log. Default is info; can be debug or error.\n\nBuilding Transporter\n--------------------\n\n### Essentials\n\n```\ngo build ./cmd/transporter/...\n```\n\n### Building guides\n\n[macOS](https://github.com/compose/transporter/blob/master/READMEMACOS.md)\n[Windows](https://github.com/compose/transporter/blob/master/READMEWINDOWS.md)\n[Vagrant](https://github.com/compose/transporter/blob/master/READMEVAGRANT.md)\n\nTransporter Resources\n=====================\n\n* [Transporter Wiki](https://github.com/compose/transporter/wiki)\n* [Compose's articles](https://www.compose.io/articles/search/?s=transporter)\n\nContributing to Transporter\n===========================\n\nWant to help out with Transporter? Great! There are instructions to get you\nstarted [here](CONTRIBUTING.md).\n\nLicensing\n=========\nTransporter is licensed under the New BSD License. See LICENSE for full license text.\n\nSupport and Guarantees\n======================\nCompose does not provide support nor guarantee stability or functionality of this tool. Please take adequate caution when using Transporter to ensure that it's the right tool for the job. Transporter may not account for failure scenarios that could lead to unexpected behavior. Always take backups, always test in dev, and always feel free to submit a PR with enhancements, features, and bug fixes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompose%2Ftransporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcompose%2Ftransporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcompose%2Ftransporter/lists"}