{"id":21355000,"url":"https://github.com/salrashid123/grpc_wireformat","last_synced_at":"2025-06-30T20:35:31.938Z","repository":{"id":57704525,"uuid":"497138073","full_name":"salrashid123/grpc_wireformat","owner":"salrashid123","description":"gRPC Unary requests the hard way: using protorefelect, dynamicpB and wireencoding to send messages","archived":false,"fork":false,"pushed_at":"2022-10-18T16:42:12.000Z","size":796,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-15T07:15:30.160Z","etag":null,"topics":["golang","grpc","protobuf"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salrashid123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-27T21:05:19.000Z","updated_at":"2024-03-21T18:24:36.000Z","dependencies_parsed_at":"2022-09-04T00:30:45.440Z","dependency_job_id":null,"html_url":"https://github.com/salrashid123/grpc_wireformat","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/salrashid123/grpc_wireformat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fgrpc_wireformat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fgrpc_wireformat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fgrpc_wireformat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fgrpc_wireformat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salrashid123","download_url":"https://codeload.github.com/salrashid123/grpc_wireformat/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salrashid123%2Fgrpc_wireformat/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262846745,"owners_count":23373868,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","grpc","protobuf"],"created_at":"2024-11-22T04:15:33.340Z","updated_at":"2025-06-30T20:35:31.861Z","avatar_url":"https://github.com/salrashid123.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## gRPC Unary requests the hard way: using protorefelect, dynamicpb and wire-encoding to send messages\n\n\nThis is just an academic exercise to create a probuf message and its [gRPC wireformat](https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md) by basic reflection packages go provides.\n\nAlmost always you just generate go packages for protobuf `echo.pb.go` and its corresponding transport over gRPC `echo_grpc.pb.go`.  You would then use both to 'just invoke' an API:\n\n```golang\nimport echo \"github.com/salrashid123/grpc_wireformat/grpc_services/src/echo\"\n\nc := echo.NewEchoServerClient(conn)\nr, err := c.SayHello(ctx, \u0026echo.EchoRequest{FirstName: \"sal\", LastName: \"mander\", MiddleName: \u0026echo.Middle{\n\tName: \"a\",\n}})\n```\n\n\nHowever, just as a way to see what you can do under the hood using 'first principles' we will with the following `.proto`:\n\n```protobuf\nsyntax = \"proto3\";\n\npackage echo;\noption go_package = \"github.com/salrashid123/grpc_wireformat/grpc_services/src/echo\";\n\nservice EchoServer {\n  rpc SayHello (EchoRequest) returns (EchoReply) {}\n}\n\nmessage Middle {\n  string name = 1;\n}\n\nmessage EchoRequest {\n  string first_name = 1;\n  string last_name = 2;\n  Middle middle_name = 3;\n}\n\nmessage EchoReply {\n  string message = 1;\n}\n```\n\nThen we will\n\n1. Load and Register its binary protobuf definition `echo.pb`\n2. Create an `EchoRequest` message using [protoreflect](https://pkg.go.dev/google.golang.org/protobuf/reflect/protoreflect) and [dynamicpb](https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb)\n3. Either\n\n     * a. Create Message Descriptor add fields\n  \n     * b. Create Message using anypb and protojson\n\n4. Encode that message into gRPC's Unary wireformat\n5. Send that to a gRPC server using _an ordinary `net/http` client over `http2`\n6. Recieve the wireformat response\n7. decode the wireformat to a protobuf message\n8. Convert the message to `EchoReply`\n9. print the contents of `EchoReply`\n\n\nWhat does that prove?  not much, its just academic thing i did...learning something new has its perpetual rewards..\n\nfor some background, also see\n\n* [grpc with curl](https://blog.salrashid.dev/articles/2017/grpc_curl/)\n* [Using Wireshark to decrypt TLS gRPC Client-Server protobuf messages](https://blog.salrashid.dev/articles/2021/wireshark-grpc-tls/)\n* [Envoy TAP filter for gRPC](https://blog.salrashid.dev/articles/2021/envoy_tap/)\n\n\n![images/arch.png](images/arch.png)\n\n---\n\nSo, let just see a normal gRPC client server:\n\n### Standard Client/Server\n\n```bash\n# run server\ncd grpc_services/\ngo run src/grpc_server.go --grpcport :50051\n\n# run client\ngo run src/grpc_client.go --host localhost:50051\n```\n\nWhat this does is just send back a unary response..nothing to see here, move along\n\n\n### The hard way\n\nNext is what this article is about.  `grpc_client_dynamic.go` does a couple of things:\n\n1. Load `echo.pb`\n\nFirst step is for your go app to even know about the protobuf...so we need to load it so protoreflect knows about it\n\n```golang\n\tprotoFile, err := ioutil.ReadFile(\"grpc_services/src/echo/echo.pb\")\n\n\tfileDescriptors := \u0026descriptorpb.FileDescriptorSet{}\n\terr = proto.Unmarshal(protoFile, fileDescriptors)\n\n\tpb := fileDescriptors.GetFile()[0]\n\tfd, err := protodesc.NewFile(pb, protoregistry.GlobalFiles)\n\n\terr = protoregistry.GlobalFiles.RegisterFile(fd)\n```\n\n2. Create Message\n\nNext we construct our `echo.EchoRequest` using the protodescriptor from step 1.\n\nI found two ways to do this:  in `3a` below, we will \"strongly type\" create a message and in `3b`, we will create a message using a JSON string.   (the latter is even more subject to simple typos)\n\n```golang\n\n```\n\n3. `(a)` Create Message Descriptor for both messages add fields\n\nIn the following, you know which type you want to create so we do this by hand:\n\n```golang\n    // create the inner message\n\techoRequestInnerMessageType, err := protoregistry.GlobalTypes.FindMessageByName(\"echo.Middle\")\n\techoRequestInnerMessageDescriptor := echoRequestInnerMessageType.Descriptor()\n\t// add a field\n\tinner_name := echoRequestInnerMessageDescriptor.Fields().ByName(\"name\")\n\treflectEchoInnerRequest := echoRequestInnerMessageType.New()\n\treflectEchoInnerRequest.Set(inner_name, protoreflect.ValueOfString(\"a\"))\n\n\t// now create the outer EchoRequest message\n\techoRequestMessageType, err := protoregistry.GlobalTypes.FindMessageByName(\"echo.EchoRequest\")\n\techoRequestMessageDescriptor := echoRequestMessageType.Descriptor()\n\n\t// setup the outer objects fields\n\tfname := echoRequestMessageDescriptor.Fields().ByName(\"first_name\")\n\tlname := echoRequestMessageDescriptor.Fields().ByName(\"last_name\")\n\tmname := echoRequestMessageDescriptor.Fields().ByName(\"middle_name\")\n\n\t// now add the fields and the Middle message\n\t// note the types, the message is of type Message\n\treflectEchoRequest := echoRequestMessageType.New()\n\treflectEchoRequest.Set(fname, protoreflect.ValueOfString(\"sal\"))\n\treflectEchoRequest.Set(lname, protoreflect.ValueOfString(\"mander\"))\n\treflectEchoRequest.Set(mname, protoreflect.ValueOfMessage(reflectEchoInnerRequest))\n\tfmt.Printf(\"EchoRequest: %v\\n\", reflectEchoRequest)\n\n\tin, err := proto.Marshal(reflectEchoRequest.Interface())\n\n\tfmt.Printf(\"Encoded EchoRequest using protoreflect %s\\n\", hex.EncodeToString(in))\n```\n\nNote that we're manually defining everything...its excruciating\n\n3. `(b)` Create Message using anypb and protojson\n\nIn the following, we will \"just create\" a message using its JSON format. Remember to set the `@type:` field in json\n\n```golang\n\tj := `{\t\"@type\": \"echo.EchoRequest\", \"firstName\": \"sal\", \"lastName\": \"mander\", \"middleName\": {\"name\": \"a\"}}`\n\ta, err := anypb.New(echoRequestMessageType.New().Interface())\n\n\terr = protojson.Unmarshal([]byte(j), a)\n\tfmt.Printf(\"Encoded EchoRequest using protojson and anypb %v\\n\", hex.EncodeToString(a.Value))\n```\n\n4. Encode it to the wireformat\n\nEither way, we need to convert the proto message into a wireformat.  For this we use [lencode](https://github.com/psanford/lencode)\n\n```golang\n\tvar out bytes.Buffer\n\tenc := lencode.NewEncoder(\u0026out, lencode.SeparatorOpt([]byte{0}))\n\n\t// (a) to send the manually generated message:\n\terr = enc.Encode(in)\n\n\t// (b) to send the json-\u003eprotobuf message\n\terr = enc.Encode(a.Value)\n```\n\nYou might be asking ...WTF is `lencode.SeparatorOpt([]byte{0})`?!\n\n\n...weeellll, thats just the wireformat position that signals compression..it works here.  See [parsing gRPC messages from Envoy TAP](https://github.com/psanford/lencode/issues/5)\n\n5. Send message\n\nWe're now ready to transmit the wireformat message to our grpc server\n\n```golang\n\tclient := http.Client{\n\t\tTransport: \u0026http2.Transport{\n\t\t\tTLSClientConfig: \u0026tlsConfig,\n\t\t},\n\t}\n\n\treader := bytes.NewReader(out.Bytes())\n\tresp, err := client.Post(\"https://localhost:50051/echo.EchoServer/SayHello\", \"application/grpc\", reader)\n```\n\n6. Recieve the wireformat response\n\nread in the bytes inside `resp.Body`\n\n7. decode the wireformat to a protobuf message\n\nUse `lencode` to unmarshall the payload\n\n```golang\n\trespMessage := lencode.NewDecoder(bytesReader, lencode.SeparatorOpt([]byte{0}))\n\trespMessageBytes, err := respMessage.Decode()\n```\n\n8. Convert the message to `EchoReply`\n\nWe now do the inverse of the outbound steps still using the descriptors we originally setup\n\n```golang\n\techoReplyMessageType, err := protoregistry.GlobalTypes.FindMessageByName(\"echo.EchoReply\")\n\n\techoReplyMessageDescriptor := echoReplyMessageType.Descriptor()\n\tpmr := echoReplyMessageType.New()\n\n\terr = proto.Unmarshal(respMessageBytes, pmr.Interface())\n\n\tmsg := echoReplyMessageDescriptor.Fields().ByName(\"message\")\n\n\tfmt.Printf(\"EchoReply.Message using protoreflect: %s\\n\", pmr.Get(msg).String())\n```\n\n9. print the contents of `EchoReply`\n\nWe now have the message back...we can print it.\n\ndone\n\n\nTO run it end-to end, keep the server running and invoke the client\n\n```bash\n$ go run grpc_client_dynamic.go \n\n\tLoading package echo\n\tRegistering MessageType: Middle\n\tRegistering MessageType: EchoRequest\n\tRegistering MessageType: EchoReply\n\n\tEchoRequest: first_name:\"sal\"  last_name:\"mander\"  middle_name:{name:\"a\"}\n\n\tEncoded EchoRequest using protoreflect 0a0373616c12066d616e6465721a030a0161\n\tEncoded EchoRequest using protojson and anypb 0a0373616c12066d616e6465721a030a0161\n\n\twire encoded EchoRequest: 00000000120a0373616c12066d616e6465721a030a0161\n\twire encoded EchoReply 00000000140a1248656c6c6f2073616c2061206d616e646572\n\n\tEncoded EchoReply 0a1248656c6c6f2073616c2061206d616e646572\n\tEchoReply.Message using protoreflect: Hello sal a mander\n\tEchoReply as string JSON: {\"message\":\"Hello sal a mander\"}\n```\n\nWhat the output shows is how we loaded the `echo.pb`, then constructed the Message from either explicitly creating it or by converting a JSON Message over. \n\nOnce that was done, we sent the wire-encoded message to the server and reversed the process.\n\n\nDid i mention you can also use `curl` to call the endpoint...\n\nthe trick is to use the wire encoded format (since,  you know, curl send stuff on the wire\n\n```bash\necho -n '00000000120a0373616c12066d616e6465721a030a0161' | xxd -r -p - frame.bin\n \ncurl -v  --raw -X POST --http2-prior-knowledge  \\\n    -H \"Content-Type: application/grpc\" \\\n    -H \"TE: trailers\" \\\n    --data-binary @frame.bin \\\n       http://localhost:50051/echo.EchoServer/SayHello -o resp.bin\n```\n\nto decode\n\n```bash\n$ xxd -p resp.bin \n00000000140a1248656c6c6f2073616c2061206d616e646572\n\n## remove the prefix headers 0000000014\n$ echo -n \"0a1248656c6c6f2073616c2061206d616e646572\" | xxd -r -p | protoc --decode_raw\n1: \"Hello sal a mander\"\n```\n\nNow look at the message decoded with wireshark\n\n![images/resp.png](images/resp.png)\n\n---\n\n### Using github.com/jhump/protoreflect\n\nThis repo also contains an end-to-end sample of [github.com/jhump/protoreflect](https://github.com/jhump/protoreflect).\n\nUsing that library makes certain things a lot easer as it wraps some of the legwork for you.  You can also \"just load\" a `.proto` file that includes specifications of the Message and gRPC server.\n\nTo use that,\n\n```bash\ncd jhump_client/\n$ go run grpc_client_jhump.go \n\u003e service echo.EchoServer\n  * method echo.EchoServer.SayHello (echo.EchoRequest) echo.EchoReply\n- message echo.Middle\n- message echo.EchoRequest\n- message echo.EchoReply\nLooking for serviceName echo.EchoServer methodName SayHello\nResponse: {\n\t\"message\": \"Hello sal a mander\"\n}\n```\n\n#### Wireshark decoding\n\nIf you want ot see the wireshark dissection of the protobufs, run\n\n```bash\necho \"\\\"$PWD/grpc_services/src/echo/\\\", \\\"TRUE\\\"\" \u003e ~/.config/wireshark/protobuf_search_paths\nwireshark trace.cap\n```\n\n---\n\n#### gRPC Reflection\n\nThe default gRPC server here also has [gRPC Reflection](https://github.com/grpc/grpc/blob/master/doc/server-reflection.md) enabled for inspection.\n\nTo use this, you have to install [grpc_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) (which, TBH, is way too cumbersome!)\n\nAnyway\n\n```bash\n$ grpc_cli ls localhost:50051\n\techo.EchoServer\n\tgrpc.health.v1.Health\n\tgrpc.reflection.v1alpha.ServerReflection\n\n$ grpc_cli ls localhost:50051 echo.EchoServer -l\n\tfilename: src/echo/echo.proto\n\tpackage: echo;\n\tservice EchoServer {\n\t\trpc SayHello(echo.EchoRequest) returns (echo.EchoReply) {}\n\t}\n\n\n$ grpc_cli ls localhost:50051 echo.EchoServer.SayHello -l\n\trpc SayHello(echo.EchoRequest) returns (echo.EchoReply) {}\n\n$ grpc_cli type localhost:50051 echo.EchoRequest\n\tmessage EchoRequest {\n\tstring first_name = 1 [json_name = \"firstName\"];\n\tstring last_name = 2 [json_name = \"lastName\"];\n\t.echo.Middle middle_name = 3 [json_name = \"middleName\"];\n\t}\n\n\ngrpc_cli call localhost:50051 echo.EchoServer.SayHello \"first_name: 'sal' last_name: 'mander' middle_name: {name: 'a'}\"\n\tconnecting to localhost:50051\n\tmessage: \"Hello sal a mander\"\n\tRpc succeeded with OK status\n```\n\n\n#### Streaming\n\nTo decode streaming, try to loop over till EOF on the payload\n\nsee [gRPC With Envoy TAP](https://github.com/salrashid123/envoy_tap/blob/main/grpc/parser/main.go#L88-L104)\n\nsomethign like \n\n```golang\n\t\t\t\trespMessage := lencode.NewDecoder(bytesReader, lencode.SeparatorOpt([]byte{0}))\n\t\t\t\tfor {\n\t\t\t\t\trespMessageBytes, err := respMessage.Decode()\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tif err == io.EOF {\n\t\t\t\t\t\t\tbreak\n\t\t\t\t\t\t}\n\t\t\t\t\t\tlog.Fatalf(\"could not Decode  %v\", err)\n\t\t\t\t\t\treturn err\n\t\t\t\t\t}\n\t\t\t\t\techoResponseMessageType, err := protoregistry.GlobalTypes.FindMessageByName(\"echo.EchoReply\")\n\t\t\t\t\tpmr := echoResponseMessageType.New()\n\t\t\t\t\techoResponseMessageDescriptor := echoResponseMessageType.Descriptor()\n\t\t\t\t\tmsg := echoResponseMessageDescriptor.Fields().ByName(\"message\")\n\t\t\t\t\terr = proto.Unmarshal(respMessageBytes, pmr.Interface())\n\n\t\t\t\t\tfmt.Printf(\"Encoded EchoResponse using protojson and anypb [%s]\\n\", pmr.Get(msg).String())\n\t\t\t\t}\n\t\t\t}\n```\n\n---\n\ndone\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fgrpc_wireformat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalrashid123%2Fgrpc_wireformat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalrashid123%2Fgrpc_wireformat/lists"}