{"id":19818735,"url":"https://github.com/sloppycoder/gtmcdc","last_synced_at":"2025-08-03T04:34:53.517Z","repository":{"id":90083696,"uuid":"211502562","full_name":"sloppycoder/gtmcdc","owner":"sloppycoder","description":"GT.M/YottaDB replication filter for doing change data capture","archived":false,"fork":false,"pushed_at":"2023-02-15T02:23:04.000Z","size":79,"stargazers_count":0,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-11T08:14:01.569Z","etag":null,"topics":["cdc","golang","kafka","mumps","yottadb"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sloppycoder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-28T13:12:43.000Z","updated_at":"2022-01-14T02:08:43.000Z","dependencies_parsed_at":"2023-03-13T18:00:00.914Z","dependency_job_id":null,"html_url":"https://github.com/sloppycoder/gtmcdc","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fgtmcdc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fgtmcdc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fgtmcdc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sloppycoder%2Fgtmcdc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sloppycoder","download_url":"https://codeload.github.com/sloppycoder/gtmcdc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241174876,"owners_count":19922401,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cdc","golang","kafka","mumps","yottadb"],"created_at":"2024-11-12T10:16:44.619Z","updated_at":"2025-02-28T15:29:21.901Z","avatar_url":"https://github.com/sloppycoder.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## GT.M/YottaDB Replication Filter\n\nA GT.M/YottaDB change event capture (CDC) mechansm by using a replication filter that convert M instructions into Kafka messages for the purpose of propagating change event to external systems.\n\nThe code here is just a skeleton. A lot more business logic needs to be implemented before it can be useful.\n\nThe scripts in [scripts/original_from_ydb_doc](scripts/original_from_ydb_doc) directory are copied from [YottaDB documentation](https://gitlab.com/YottaDB/DB/YDBDoc/tree/master/AdminOpsGuide/repl_procedures). Shout out to the great work by [YottaDB team](https://yottadb.com/)\n\n### System Requirements\n\nTested with:\n\n1. [Ubuntu 18.04](http://releases.ubuntu.com/18.04/)\n2. [Go 1.13](https://golang.org/dl/)\n3. [Confluent Platform 5.3](https://www.confluent.io/download/)\n4. [MongoDB server](https://www.mongodb.com/download-center/community)\n5. [MongoDB Kafka connector](https://www.confluent.io/hub/mongodb/kafka-connect-mongodb)\n6. [YottaDB r1.28](https://yottadb.com/product/get-started/)\n7. [FIS GT.M](https://en.wikipedia.org/wiki/GT.M)\n\n### Build\n\n```\ngo build ./cmd/cdcfilter\n\n```\n\n### Setup the environment\n\n1. Install Confluent Platform along with confluent cli. Set environment variable CONFLUENT_HOME to the installation directory.\n2. Install MongoDB kafka sink compoent from ConfluentHub.\n3. Install MongoDB server\n4. Create a user dev with password dev, by running the script below using mongo shell\n\n### Test cdcfilter using Journal file without YottaDB setup\n\nYou should probably start here.\n\nInstall and set the prerequisites.\n\n1. MongoDB 4.2 Community Edition\n2. Confluent Platform or Community edition\n3. Setup a user dev with password dev who owns the dev database, using something like below\n\n```\nuse dev\ndb.createUser(\n  {\n    user: \"dev\",\n    pwd:  \"dev\",\n    roles: [ { role: \"readWrite\", db: \"dev\" } ]\n  }\n)\n\n```\n\nThe follow the steps to setup KSQL streams and Sink in Kafka.\n\n```\n\n# install confluent kafka and confluent CLI, set the $CONFLUENT_HOME environment variable.\n# if you install using the RPM or DEB file, the kafka utilities are in your PATH, so the $CONFLUENT_HOME/bin\n# below can be omitted\n\n# startup confluent platform\n$CONFLUENT_HOME/bin/confluent local start\n\ncd scripts/kafka\n\n# create the topics and streams\n$CONFLUENT_HOME/bin/ksql\n\nksql\u003e RUN SCRIPT setup.ksql\n\n# should print bunch of stuff but no error messages.\n# ctrl-D to exit ksql prompt\n# at this point 2 KSQL streams should have been setup and along with their topics\n\n# setup the MongoDB kakfa sink\n./create_sink\n\n# at this point all things Kafka should be runing. If you have full Confluent Platform, you can check things out using the Web console.\n\n# now run the cdcfilter with journal file\ncd ../..\n./cdcfilter -i testdata/t.txt \u003e x.out\n\n# data should be in MongoDB now\nmongo -u dev -p dev dev\n\u003edb.accounts.find()\n\n# you should see 2 documents here.\n# if you don't see it, check your kafka log and cdcfilter.log to see if there're any error messages\n\n```\n\n### Test with YottaDB or GT.M replication setup\n\nInstall GT.M by running the ```apt install fis-gtm``` or [YottaDB](https://yottadb.com/product/get-started/).\nThen follow the steps below to create source database A and target database B, start the replicating processes.\n\n```\ncd scripts/ydb \n# or \ncd scripts/gtm\n\n# setup 2 databases, A and B\n./dbinit A\n./dbinit B\n\n# start replication processes\n./repl_start A B\n\n# check replication filter log\n# the log should show the filter started with no errors\ntail -f cdcfilter.log\n\n\n\n```\n\nWrite test data.\nRun the acc.m script to write data to YottaDB site A, the replication filter should receive the writes and create events in Kafka, which will be processed by MongoDB kafka connector sink component, data will be written to MongoDB database dev and collection accounts.\n\n```\n# copy test data\ncp acc.m A/.\n$gtm_dist/mumps -r acc\n\n# open mongo shell and check data db.accounts.find()\n\n# edit the test data\nvi A/acc.m\n# save data, and run the updated program\n$gtm_dist/mumps -r acc\n\n\n```\n\nTo tear down the setup and cleanup everything\n\n```\n./repl_stop\n\n# run this only if you want to delete both A and B databases\nrm -rf A B cdcfilter.log\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloppycoder%2Fgtmcdc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsloppycoder%2Fgtmcdc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsloppycoder%2Fgtmcdc/lists"}