{"id":18853693,"url":"https://github.com/rwynn/gtm","last_synced_at":"2025-04-12T19:47:24.635Z","repository":{"id":8863113,"uuid":"10574544","full_name":"rwynn/gtm","owner":"rwynn","description":"gtm (go tail mongo) is a MongoDB event listener","archived":false,"fork":false,"pushed_at":"2025-02-04T22:54:35.000Z","size":211,"stargazers_count":148,"open_issues_count":13,"forks_count":35,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-04-03T23:09:52.670Z","etag":null,"topics":["go","golang","mongodb","oplog","tail"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rwynn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-08T21:33:26.000Z","updated_at":"2025-02-04T22:51:41.000Z","dependencies_parsed_at":"2025-01-27T23:10:28.661Z","dependency_job_id":"c82fda74-6dc0-4a98-aaba-f7a92c40496a","html_url":"https://github.com/rwynn/gtm","commit_stats":null,"previous_names":[],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rwynn%2Fgtm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rwynn%2Fgtm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rwynn%2Fgtm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rwynn%2Fgtm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rwynn","download_url":"https://codeload.github.com/rwynn/gtm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625497,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","mongodb","oplog","tail"],"created_at":"2024-11-08T03:45:18.574Z","updated_at":"2025-04-12T19:47:24.608Z","avatar_url":"https://github.com/rwynn.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"gtm\n===\ngtm (go tail mongo) is a utility written in Go which tails the MongoDB oplog and \nsends create, update, delete events to your code.\nIt can be used to send emails to new users, [index documents](https://www.github.com/rwynn/monstache), \n[write time series data](https://www.github.com/rwynn/mongofluxd), or something else.\n\nThis branch is a port of the original gtm to use the new official golang driver from MongoDB.\nThe original gtm uses the community mgo driver. To use the community mgo driver use the `legacy` branch.\n\n### Requirements ###\n+ [Go](http://golang.org/doc/install)\n+ [mongodb go driver](https://github.com/mongodb/mongo-go-driver)\n+ [mongodb](http://www.mongodb.org/)\n\n### Installation ###\n\n\tgo get github.com/rwynn/gtm/v2\n\n### Setup ###\n\ngtm uses the MongoDB [oplog](https://docs.mongodb.com/manual/core/replica-set-oplog/) as an event source. \nYou will need to ensure that MongoDB is configured to produce an oplog by \n[deploying a replica set](http://docs.mongodb.org/manual/tutorial/deploy-replica-set/).\n\nIf you haven't already done so, follow the 5 step \n[procedure](https://docs.mongodb.com/manual/tutorial/deploy-replica-set/#procedure) to initiate and \nvalidate your replica set. For local testing your replica set may contain a \n[single member](https://docs.mongodb.com/manual/tutorial/convert-standalone-to-replica-set/).\n\n### Usage ###\n\n```golang\npackage main\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"go.mongodb.org/mongo-driver/bson\"\n\t\"go.mongodb.org/mongo-driver/bson/bsontype\"\n\t\"go.mongodb.org/mongo-driver/mongo\"\n\t\"go.mongodb.org/mongo-driver/mongo/options\"\n\t\"github.com/rwynn/gtm/v2\"\n\t\"reflect\"\n\t\"time\"\n)\n\nfunc main() {\n\trb := bson.NewRegistryBuilder()\n\t//rb.RegisterTypeMapEntry(bsontype.Timestamp, reflect.TypeOf(time.Time{}))\n\trb.RegisterTypeMapEntry(bsontype.DateTime, reflect.TypeOf(time.Time{}))\n\treg := rb.Build()\n\tclientOptions := options.Client()\n\tclientOptions.SetRegistry(reg)\n\tclientOptions.ApplyURI(\"mongodb://localhost:27017\")\n\tclient, err := mongo.NewClient(clientOptions)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tctxm, cancel := context.WithTimeout(context.Background(), 20*time.Second)\n\tdefer cancel()\n\terr = client.Connect(ctxm)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tdefer client.Disconnect(context.Background())\n\tctx := gtm.Start(client, \u0026gtm.Options{\n\t\tDirectReadNs: []string{\"test.test\"},\n\t\tChangeStreamNs: []string{\"test.test\"},\n\t\tMaxWaitSecs: 10,\n\t\tOpLogDisabled: true,\n\t})\n\tfor {\n\t\tselect {\n\t\tcase err := \u003c-ctx.ErrC:\n\t\t\tfmt.Printf(\"got err %+v\", err)\n\t\t\tbreak\n\t\tcase op := \u003c-ctx.OpC:\n\t\t\tfmt.Printf(\"got op %+v\", op)\n\t\t\tbreak\n\t\t}\n\t}\n}\n```\n\n### Configuration ###\n\n```golang\nfunc PipeBuilder(namespace string, changeStream bool) ([]interface{}, error) {\n\n\t// to build your pipelines for change events you will want to reference\n\t// the MongoDB reference for change events at \n\t// https://docs.mongodb.com/manual/reference/change-events/\n\n\t// you will only receive changeStream == true when you configure gtm with\n\t// ChangeStreamNS (requies MongoDB 3.6+).  You cannot build pipelines for\n\t// changes using legacy direct oplog tailing\n\n\tif namespace == \"users.users\" {\n\t\t// given a set of docs like {username: \"joe\", email: \"joe@email.com\", amount: 1}\n\t\tif changeStream {\n\t\t\treturn []interface{}{\n\t\t\t\tbson.M{\"$match\": bson.M{\"fullDocument.username\": \"joe\"}},\n\t\t\t}, nil\n\t\t} else {\n\t\t\treturn []interface{}{\n\t\t\t\tbson.M{\"$match\": bson.M{\"username\": \"joe\"}},\n\t\t\t}, nil\n\t\t}\n\t} else if namespace == \"users.status\" \u0026\u0026 changeStream {\n\t\t// return a pipeline that only receives events when a document is \n\t\t// inserted, deleted, or a specific field is changed. In this case\n\t\t// only a change to field1 is processed.  Changes to other fields\n\t\t// do not match the pipeline query and thus you won't receive the event.\n\t\treturn []interface{}{\n\t\t\tbson.M{\"$match\": bson.M{\"$or\": []interface{} {\n\t\t\t\tbson.M{\"updateDescription\": bson.M{\"$exists\": false}},\n\t\t\t\tbson.M{\"updateDescription.updatedFields.field1\": bson.M{\"$exists\": true}},\n\t\t\t}}},\n\t\t}, nil\n\t}\n\treturn nil, nil\n}\n\nfunc NewUsers(op *gtm.Op) bool {\n\treturn op.Namespace == \"users.users\" \u0026\u0026 op.IsInsert()\n}\n\n// if you want to listen only for certain events on certain collections\n// pass a filter function in options\nctx := gtm.Start(client, \u0026gtm.Options{\n\tNamespaceFilter: NewUsers, // only receive inserts in the user collection\n})\n// more options are available for tuning\nctx := gtm.Start(client, \u0026gtm.Options{\n\tNamespaceFilter      nil,           // op filter function that has access to type/ns ONLY\n\tFilter               nil,           // op filter function that has access to type/ns/data\n\tAfter:               nil,     \t    // if nil defaults to gtm.LastOpTimestamp; not yet supported for ChangeStreamNS\n\tOpLogDisabled:       false,         // true to disable tailing the MongoDB oplog\n\tOpLogDatabaseName:   nil,     \t    // defaults to \"local\"\n\tOpLogCollectionName: nil,     \t    // defaults to \"oplog.rs\"\n\tChannelSize:         0,       \t    // defaults to 20\n\tBufferSize:          25,            // defaults to 50. used to batch fetch documents on bursts of activity\n\tBufferDuration:      0,             // defaults to 750 ms. after this timeout the batch is force fetched\n\tWorkerCount:         8,             // defaults to 1. number of go routines batch fetching concurrently\n\tOrdering:            gtm.Document,  // defaults to gtm.Oplog. ordering guarantee of events on the output channel as compared to the oplog\n\tUpdateDataAsDelta:   false,         // set to true to only receive delta information in the Data field on updates (info straight from oplog)\n\tDirectReadNs:        []string{\"db.users\"}, // set to a slice of namespaces (collections or views) to read data directly from\n\tDirectReadSplitMax:  9,             // the max number of times to split a collection for concurrent reads (impacts memory consumption)\n\tPipe:                PipeBuilder,   // an optional function to build aggregation pipelines\n\tPipeAllowDisk:       false,         // true to allow MongoDB to use disk for aggregation pipeline options with large result sets\n\tLog:                 myLogger,      // pass your own logger\n\tChangeStreamNs       []string{\"db.col1\", \"db.col2\"}, // MongoDB 3.6+ only; set to a slice to namespaces to read via MongoDB change streams\n})\n```\n\n### Direct Reads ###\n\nIf, in addition to tailing the oplog, you would like to also read entire collections you can set the DirectReadNs field\nto a slice of MongoDB namespaces.  Documents from these collections will be read directly and output on the ctx.OpC channel.  \n\nYou can wait till all the collections have been fully read by using the DirectReadWg wait group on the ctx.\n\n```golang\ngo func() {\n\tctx.DirectReadWg.Wait()\n\tfmt.Println(\"direct reads are done\")\n}()\n```\n\n### Pause, Resume, Since, and Stop ###\n\nYou can pause, resume, or seek to a timestamp from the oplog. These methods effect only change events and not direct reads.\n\n```golang\ngo func() {\n\tctx.Pause()\n\ttime.Sleep(time.Duration(2) * time.Minute)\n\tctx.Resume()\n\tctx.Since(previousTimestamp)\n}()\n```\n\nYou can stop all goroutines created by `Start` or `StartMulti`. You cannot resume a context once it has been stopped. You would need to create a new one.\n\n```golang\ngo func() {\n\tctx.Stop()\n\tfmt.Println(\"all go routines are stopped\")\n}\n```\n\n### Custom Unmarshalling ###\n\nIf you'd like to unmarshall MongoDB documents into your own struct instead of the document getting\nunmarshalled to a generic map[string]interface{} you can use a custom unmarshal function:\n\n```golang\ntype MyDoc struct {\n\tId interface{} \"_id\"\n\tFoo string \"foo\"\n}\n\nfunc custom(namespace string, data []byte) (interface{}, error) {\n\t// use namespace, e.g. db.col, to map to a custom struct\n\tif namespace == \"test.test\" {\n\t\tvar doc MyDoc\n\t\tif err := bson.Unmarshal(data, \u0026doc); err == nil {\n\t\t\treturn doc, nil\n\t\t} else {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\treturn nil, errors.New(\"unsupported namespace\")\n}\n\nctx := gtm.Start(client, \u0026gtm.Options{\n\tUnmarshal: custom,\n}\n\nfor {\n\tselect {\n\tcase op:= \u003c-ctx.OpC:\n\t\tif op.Namespace == \"test.test\" {\n\t\t\tdoc := op.Doc.(MyDoc)\n\t\t\tfmt.Println(doc.Foo)\n\t\t}\n\t}\n}\n```\n\n### Workers ###\n\nYou may want to distribute event handling between a set of worker processes on different machines.\nTo do this you can leverage the **github.com/rwynn/gtm/consistent** package.  \n\nCreate a TOML document containing a list of all the event handlers.\n\n```toml\nWorkers = [ \"Tom\", \"Dick\", \"Harry\" ] \n```\n\nCreate a consistent filter to distribute the work between Tom, Dick, and Harry. A consistent filter\nneeds to acces the Data attribute of each op so it needs to be set as a Filter as opposed to a \nNamespaceFilter.\n\n```golang\nname := flag.String(\"name\", \"\", \"the name of this worker\")\nflag.Parse()\nfilter, filterErr := consistent.ConsistentHashFilterFromFile(*name, \"/path/to/toml\")\nif filterErr != nil {\n\tpanic(filterErr)\n}\n\n// there is also a method **consistent.ConsistentHashFilterFromDocument** which allows\n// you to pass a Mongo document representing the config if you would like to avoid\n// copying the same config file to multiple servers\n```\n\nPass the filter into the options when calling gtm.Tail\n\n```golang\nctx := gtm.Start(client, \u0026gtm.Options{Filter: filter})\n```\n\nIf you have your multiple filters you can use the gtm utility method ChainOpFilters\n\n```golang\nfunc ChainOpFilters(filters ...OpFilter) OpFilter\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frwynn%2Fgtm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frwynn%2Fgtm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frwynn%2Fgtm/lists"}