{"id":30561872,"url":"https://github.com/bufbuild/prototransform","last_synced_at":"2025-08-28T12:40:55.984Z","repository":{"id":65594814,"uuid":"595308064","full_name":"bufbuild/prototransform","owner":"bufbuild","description":"Client library for Buf Reflection API, for transforming Protobuf data.","archived":false,"fork":false,"pushed_at":"2025-08-27T13:21:16.000Z","size":300,"stargazers_count":48,"open_issues_count":0,"forks_count":1,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-08-27T22:24:32.023Z","etag":null,"topics":["buf-schema-registry","grpc","protobuf","protocol-buffers"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bufbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-01-30T20:22:25.000Z","updated_at":"2025-08-27T13:20:20.000Z","dependencies_parsed_at":"2023-02-16T11:45:40.948Z","dependency_job_id":"a1e8dac7-8b9d-4ab4-b425-a1d94a64c8a4","html_url":"https://github.com/bufbuild/prototransform","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/bufbuild/prototransform","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fprototransform","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fprototransform/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fprototransform/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fprototransform/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bufbuild","download_url":"https://codeload.github.com/bufbuild/prototransform/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fprototransform/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272445741,"owners_count":24936409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["buf-schema-registry","grpc","protobuf","protocol-buffers"],"created_at":"2025-08-28T12:40:54.739Z","updated_at":"2025-08-28T12:40:55.942Z","avatar_url":"https://github.com/bufbuild.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"![The Buf logo](./.github/buf-logo.svg)\n\n# Prototransform\n\n[![Build](https://github.com/bufbuild/prototransform/actions/workflows/ci.yaml/badge.svg?branch=main)][badges_ci]\n[![Report Card](https://goreportcard.com/badge/github.com/bufbuild/prototransform)][badges_goreportcard]\n[![GoDoc](https://pkg.go.dev/badge/github.com/bufbuild/prototransform.svg)][badges_godoc]\n\n### Convert protobuf message data to alternate formats\n\nUse the `prototransform` library to simplify your data transformation \u0026\ncollection. Our simple package allows the caller to convert a given message data\nblob from one format to another by referring to a type schema on the Buf Schema\nRegistry.\n\n* No need to bake in proto files\n* Supports Binary, JSON and Text formats\n* Extensible for other/custom formats\n\n## Getting started\n\n`prototransform` is designed to be flexible enough to fit quickly into your\ndevelopment environment.\n\nHere's an example of how you could use `prototransform` to transform messages\nreceived from a PubSub topic...\n\n### Transform Messages from a Topic\n\nWhilst `prototransform` has various applications, converting messages off some\nkind of message queue is a primary use-case. This can take many forms, for the\npurposes of simplicity we will look at this abstractly in a pub/sub model where\nwe want to:\n\n1. Open a subscription to a topic with the Pub/Sub service of your choice\n2. Start a `SchemaWatcher` to fetch a module from the Buf Schema Registry\n3. Receive, Transform and Acknowledge messages from the topic\n\n#### Opening a Subscription \u0026 Schema Watcher\n\n```go\nimport (\n\t\"context\"\n\t\"fmt\"\n\n\t\"github.com/bufbuild/prototransform\"\n\t\"gocloud.dev/pubsub\"\n\t_ \"gocloud.dev/pubsub/\u003cdriver\u003e\"\n)\n...\n\tsubs, err := pubsub.OpenSubscription(ctx, \"\u003cdriver-url\u003e\")\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not open topic subscription: %v\", err)\n\t}\n\tdefer subs.Shutdown(ctx)\n\t// Supply auth credentials to the BSR\n\tclient := prototransform.NewDefaultFileDescriptorSetServiceClient(\"\u003cbsr-token\u003e\")\n\t// Configure the module for schema watcher\n\tcfg := \u0026prototransform.SchemaWatcherConfig{\n\t\tSchemaPoller: prototransform.NewSchemaPoller(\n\t\t\tclient,\n\t\t\t\"buf.build/someuser/somerepo\", // BSR module\n\t\t\t\"some-tag\", // tag or draft name or leave blank for \"latest\"\n\t\t),\n\t}\n\twatcher, err := prototransform.NewSchemaWatcher(ctx, cfg)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to create schema watcher: %v\", err)\n\t}\n\tdefer watcher.Stop()\n\t// before we start processing messages, make sure the schema has been\n\t// downloaded\n\tctx, cancel := context.WithTimeout(ctx, 10*time.Second)\n\tdefer cancel()\n\tif err := watcher.AwaitReady(ctx); err != nil {\n\t\treturn fmt.Errorf(\"schema watcher never became ready: %v\", err)\n\t}\n...\n```\n\nA `SchemaWatcher` is the entrypoint of `prototransform`. This is created first\nso your code can connect to the Buf Schema Registry and fetch a schema to be used\nto transform and/or filter payloads.\n\n#### Prepare a converter\n\nA `Converter` implements the functionality to convert payloads to different formats\nand optionally filter/mutate messages during this transformation. In the following\nexample, we have initialized a `*prototransform.Converter` which expects a binary\ninput and will return JSON.\n\n```go\n...\n    converter := \u0026prototransform.Converter{\n        Resolver:       schemaWatcher,\n        InputFormat:    prototransform.BinaryInputFormat(proto.UnmarshalOptions{}),\n        OutputFormat:   prototransform.JSONOutputFormat(protojson.MarshalOptions{}),\n    }\n...\n```\n\nOut of the box, you can supply `proto`, `protojson` and `prototext` here but\nfeel free to supply your own custom formats as-well.\n\n| FORMAT                                                                                  | InputFormat                          | OutputFormat                          |\n|-----------------------------------------------------------------------------------------|--------------------------------------|---------------------------------------|\n| [JSON](https://pkg.go.dev/google.golang.org/protobuf/encoding/protojson#MarshalOptions) | `prototransform.JSONInputFormat()`   | `prototransform.JSONOutputFormat()`   |\n| [TEXT](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext#MarshalOptions) | `prototransform.TEXTInputFormat()`   | `prototransform.TEXTOutputFormat()`   |\n| [Binary](https://pkg.go.dev/google.golang.org/protobuf/proto#MarshalOptions)            | `prototransform.BinaryInputFormat()` | `prototransform.BinaryOutputFormat()` |\n\n#### Receiving and Transforming Messages\n\nNow that we have an active subscription, schema watcher, and converter, we can\nstart processing messages. A simple subscriber that transforms received messages\nlooks like this:\n\n```go\n...\n    // Loop on received messages.\n    for {\n        msg, err := subscription.Receive(ctx)\n        if err != nil {\n            log.Printf(\"Receiving message: %v\", err)\n            break\n        }\n        // Do transformation based on the message name\n        convertedMessage, err := converter.ConvertMessage(\"\u003cmessage-name\u003e\", msg.Body)\n        if err != nil {\n            log.Printf(\"Converting message: %v\", err)\n            break\n        }\n        fmt.Printf(\"Converted message: %q\\n\", convertedMessage)\n\n        msg.Ack()\n    }\n...\n```\n\nFor illustrative purposes, let's assume that the topic we have subscribed to is\n`buf.connect.demo.eliza.v1`, we have the module stored on the BSR [here](https://buf.build/bufbuild/eliza).\nWe would configure the message name as `buf.connect.demo.eliza.v1.ConverseRequest`.\n\n## Options\n\n### Cache\n\nA `SchemaWatcher` can be configured with a user-supplied cache\nimplementation, to act as a fallback when fetching schemas. The interface is of\nthe form:\n\n```go\ntype Cache interface {\n    Load(ctx context.Context, key string) ([]byte, error)\n    Save(ctx context.Context, key string, data []byte) error\n}\n```\n\nThis repo provides three implementations that you can use:\n1. [`filecache`](https://pkg.go.dev/github.com/bufbuild/prototransform/cache/filecache): Cache schemas in local files.\n2. [`rediscache`](https://pkg.go.dev/github.com/bufbuild/prototransform/cache/rediscache): Cache schemas in a shared Redis server.\n3. [`memcache`](https://pkg.go.dev/github.com/bufbuild/prototransform/cache/memcache): Cache schemas in a shared memcached server.\n\n### Filters\n\nA use-case exists where the values within the output message should differ from\nthe input given some set of defined rules. For example, Personally Identifiable\nInformation(PII) may want to be removed from a message before it is piped into a\nsink. For this reason, we have supplied Filters.\n\nHere's an example where we have defined a custom annotation to mark fields\nas `sensitive`:\n\n```protobuf\nsyntax = \"proto3\";\npackage foo.v1;\n// ...\nextend google.protobuf.FieldOptions {\n  bool sensitive = 30000;\n}\n// ...\nmessage FooMessage {\n  string name = 1 [(sensitive) = true];\n}\n```\n\nWe then use `prototransform.Redact()` to create a filter and\nsupply it to our converter via its `Filters` field:\n\n```go\n...\nisSensitive := func (in protoreflect.FieldDescriptor) bool {\n    return proto.GetExtension(in.Options(), foov1.E_Sensitive).(bool)\n}\nfilter := prototransform.Redact(isSensitive)\nconverter.Filters = prototransform.Filters{filter}\n...\n```\n\nNow, any attribute marked as \"sensitive\" will be omitted from the output\nproduced by the converter.\n\nThis package also includes a predicate named `HasDebugRedactOption` that\ncan be used to redact data for fields that have the `debug_redact` standard\noption set (this option was introduced in `protoc` v22.0).\n\n## Community\n\nFor help and discussion around Protobuf, best practices, and more, join us\non [Slack][badges_slack].\n\n## Status\n\nThis project is currently in **alpha**. The API should be considered unstable and likely to change.\n\n## Legal\n\nOffered under the [Apache 2 license][license].\n\n[badges_ci]: https://github.com/bufbuild/prototransform/actions/workflows/ci.yaml\n[badges_goreportcard]: https://goreportcard.com/report/github.com/bufbuild/prototransform\n[badges_godoc]: https://pkg.go.dev/github.com/bufbuild/prototransform\n[badges_slack]: https://join.slack.com/t/bufbuild/shared_invite/zt-f5k547ki-dW9LjSwEnl6qTzbyZtPojw\n[license]: https://github.com/bufbuild/prototransform/blob/main/LICENSE.txt\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufbuild%2Fprototransform","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbufbuild%2Fprototransform","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufbuild%2Fprototransform/lists"}