{"id":29605504,"url":"https://github.com/bufbuild/hyperpb-go","last_synced_at":"2025-07-21T17:02:57.007Z","repository":{"id":305118818,"uuid":"1019753440","full_name":"bufbuild/hyperpb-go","owner":"bufbuild","description":"10x faster dynamic Protobuf parsing in Go that’s even 3x faster than generated code.","archived":false,"fork":false,"pushed_at":"2025-07-16T21:48:18.000Z","size":13462,"stargazers_count":178,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-18T12:46:36.837Z","etag":null,"topics":["grpc","protobuf","protoc","protocol-buffers"],"latest_commit_sha":null,"homepage":"https://buf.build/blog/hyperpb","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bufbuild.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":".github/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-14T20:19:44.000Z","updated_at":"2025-07-18T12:19:52.000Z","dependencies_parsed_at":"2025-07-18T12:56:54.589Z","dependency_job_id":null,"html_url":"https://github.com/bufbuild/hyperpb-go","commit_stats":null,"previous_names":["bufbuild/hyperpb-go"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bufbuild/hyperpb-go","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fhyperpb-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fhyperpb-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fhyperpb-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fhyperpb-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bufbuild","download_url":"https://codeload.github.com/bufbuild/hyperpb-go/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bufbuild%2Fhyperpb-go/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266152557,"owners_count":23884527,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grpc","protobuf","protoc","protocol-buffers"],"created_at":"2025-07-20T16:05:55.776Z","updated_at":"2025-07-20T16:05:59.488Z","avatar_url":"https://github.com/bufbuild.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"![The Buf logo](https://raw.githubusercontent.com/bufbuild/hyperpb-go/main/.github/buf-logo.svg)\n\n# hyperpb\n\n`hyperpb` is a highly optimized dynamic message library for Protobuf or read-only\nworkloads. It is designed to be a drop-in replacement for\n[`dynamicpb`][dynamicpb],\n`protobuf-go`'s canonical solution for working with completely dynamic messages.\n\n`hyperpb`'s parser is an efficient VM for a special instruction set, a variant of\ntable-driven parsing (TDP), pioneered by [the UPB project][upb].\n\nOur parser is very fast, beating `dynamicpb` by 10x, and often beating\n`protobuf-go`'s generated code by a factor of 2-3x, especially for workloads with\nmany nested messages.\n\n![Barchart of benchmarks for hyperpb](https://raw.githubusercontent.com/bufbuild/hyperpb-go/main/.github/benchmarks.svg)\n\nHere, we show two benchmark variants for `hyperpb`: out-of-the-box performance with no optimizations turned on, and real-time profile-guided optimization (PGO) with all optimizations we currently offer enabled.\n\n## Usage\n\nThe core conceit of `hyperpb` is that you must pre-compile a parser using\n`hyperpb.Compile` at runtime, much like regular expressions require that you\n`regexp.Compile` them. Doing this allows `hyperpb` to run optimization passes on\nyour message, and delaying it to runtime allows us to continuously improve\nlayout optimizations, without making any source-breaking changes.\n\nFor example, let's say that we want to compile a parser for some type baked into\nour binary, and parse some data with it.\n\n\u003c!-- weatherDataBytes values match data used in example_test.go and should be kept in sync --\u003e\n\n```go\npackage main\n\nimport (\n    \"fmt\"\n    \"log\"\n\n    \"buf.build/go/hyperpb\"\n    \"google.golang.org/protobuf/proto\"\n\n    weatherv1 \"buf.build/gen/go/bufbuild/hyperpb-examples/protocolbuffers/go/example/weather/v1\"\n)\n\n// Byte slice representation of a valid *weatherv1.WeatherReport.\nvar weatherDataBytes = []byte{\n    0x0a, 0x07, 0x53, 0x65, 0x61, 0x74, 0x74, 0x6c,\n    0x65, 0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x41, 0x44,\n    0x39, 0x33, 0x15, 0x66, 0x86, 0x22, 0x43, 0x1d,\n    0xcd, 0xcc, 0x34, 0x41, 0x25, 0xd7, 0xa3, 0xf0,\n    0x41, 0x2d, 0x33, 0x33, 0x13, 0x40, 0x30, 0x03,\n    0x12, 0x1d, 0x0a, 0x05, 0x4b, 0x48, 0x42, 0x36,\n    0x30, 0x15, 0xcd, 0x8c, 0x22, 0x43, 0x1d, 0x33,\n    0x33, 0x5b, 0x41, 0x25, 0x52, 0xb8, 0xe0, 0x41,\n    0x2d, 0x33, 0x33, 0xf3, 0x3f, 0x30, 0x03,\n}\n\nfunc main() {\n    // Compile a type for your message. Make sure to cache this!\n    // Here, we're using a compiled-in descriptor.\n    msgType := hyperpb.CompileMessageDescriptor(\n        (*weatherv1.WeatherReport)(nil).ProtoReflect().Descriptor(),\n    )\n\n    // Allocate a fresh message using that type.\n    msg := hyperpb.NewMessage(msgType)\n\n    // Parse the message, using proto.Unmarshal like any other message type.\n    if err := proto.Unmarshal(weatherDataBytes, msg); err != nil {\n        // Handle parse failure.\n        log.Fatalf(\"failed to parse weather data: %v\", err)\n    }\n\n    // Use reflection to read some fields. hyperpb currently only supports access\n    // by reflection. You can also look up fields by index using fields.Get(), which\n    // is less legible but doesn't hit a hashmap.\n    fields := msgType.Descriptor().Fields()\n\n    // Get returns a protoreflect.Value, which can be printed directly...\n    fmt.Println(msg.Get(fields.ByName(\"region\")))\n\n    // ... or converted to an explicit type to operate on, such as with List(),\n    // which converts a repeated field into something with indexing operations.\n    stations := msg.Get(fields.ByName(\"weather_stations\")).List()\n    for i := range stations.Len() {\n        // Get returns a protoreflect.Value too, so we need to convert it into\n        // a message to keep extracting fields.\n        station := stations.Get(i).Message()\n        fields := station.Descriptor().Fields()\n\n        // Here we extract each of the fields we care about from the message.\n        // Again, we could use fields.Get if we know the indices.\n        fmt.Println(\"station:\", station.Get(fields.ByName(\"station\")))\n        fmt.Println(\"frequency:\", station.Get(fields.ByName(\"frequency\")))\n        fmt.Println(\"temperature:\", station.Get(fields.ByName(\"temperature\")))\n        fmt.Println(\"pressure:\", station.Get(fields.ByName(\"pressure\")))\n        fmt.Println(\"wind_speed:\", station.Get(fields.ByName(\"wind_speed\")))\n        fmt.Println(\"conditions:\", station.Get(fields.ByName(\"conditions\")))\n    }\n}\n```\n\nCurrently, `hyperpb` only supports manipulating messages through the reflection\nAPI; it shines best when you need write a very generic service that\ndownloads types off the network and parses messages using those types, which\nforces you to use reflection.\n\nMutation is currently not supported; any operation which would mutate an\nalready-parsed message will panic. Which methods of `*hyperpb.Message` panic\nis included in the documentation.\n\n### Using types from a registry\n\nWe can use the `hyperpb.CompileFileDescriptorSet` function to parse a dynamic type and\nuse it to walk the fields of a message:\n\n```go\nfunc processDynamicMessage(\n    schema *descriptorpb.FileDescriptorSet,\n    messageName protoreflect.FullName,\n    data []byte,\n) error {\n    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName) // Remember to cache this!\n    if err != nil {\n        return err\n    }\n\n    msg := hyperpb.NewMessage(msgType)\n    if err := proto.Unmarshal(data, msg); err != nil {\n        return err\n    }\n\n    // Range will iterate over all of the populated fields in msg. Here we\n    // use Range with go1.24 iterator syntax.\n    for field, value := range msg.Range {\n        // Do something with each populated field.\n    }\n    return nil\n}\n```\n\nSince any generic, non-mutating operation will work with `hyperpb` messages,\nwe can use them as an efficient transcoding medium from the wire format, for\nruntime-loaded messages.\n\n```go\nfunc dynamicMessageToJSON(\n    schema *descriptorpb.FileDescriptorSet,\n    messageName protoreflect.FullName,\n    data []byte,\n) ([]byte, error) {\n    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)\n    if err != nil {\n        return nil, err\n    }\n\n    msg := hyperpb.NewMessage(msgType)\n    if err := proto.Unmarshal(data, msg); err != nil {\n        return nil, err\n    }\n\n    // Dump the message to JSON. This just works!\n    return protojson.Marshal(msg)\n}\n```\n\n`protovalidate` also works directly on reflection, so it works out-of-the-box:\n\n```go\nfunc validateDynamicMessage(\n    schema *descriptorpb.FileDescriptorSet,\n    messageName protoreflect.FullName,\n    data []byte,\n) error {\n    // Unmarshal like before.\n    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)\n    if err != nil {\n        return err\n    }\n\n    msg := hyperpb.NewMessage(msgType)\n    if err := proto.Unmarshal(data, msg); err != nil {\n        return err\n    }\n\n    // Run custom validation. This just works!\n    return protovalidate.Validate(msg)\n}\n```\n\n## Advanced Usage\n\n`hyperpb` is all about parsing as fast as possible, so there are a number of\noptimization knobs available. Calling `Message.Unmarshal` directly instead\nof `proto.Unmarshal` allows setting custom `UnmarshalOption`s:\n\n```go\nfunc unmarshalWithCustomOptions(\n    schema *descriptorpb.FileDescriptorSet,\n    messageName protoreflect.FullName,\n    data []byte,\n) error {\n    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)\n    if err != nil {\n        return err\n    }\n\n    msg := hyperpb.NewMessage(msgType)\n    return msg.Unmarshal(\n        data,\n        hyperpb.WithMaxDecodeMisses(16),\n        // Additional options...\n    )\n}\n```\n\nThe compiler also takes `CompileOptions`, such as for configuring how extensions\nare resolved:\n\n```go\nmsgType, err := hyperpb.CompileFileDescriptor(\n    schema,\n    messageName,\n    hyperpb.WithExtensionsFromTypes(typeRegistry),\n    // Additional options...\n)\n```\n\n### Memory Reuse\n\n`hyperpb` also has a memory-reuse mechanism that side-steps the Go garbage\ncollector for improved allocation latency. `hyperpb.Shared` is book-keeping\nstate and resources shared by all messages resulting from the same parse.\nAfter the message goes out of scope, these resources are ordinarily reclaimed\nby the garbage collector.\n\nHowever, a `hyperpb.Shared` can be retained after its associated message goes\naway, allowing for re-use. Consider the following example of a request handler:\n\n```go\ntype requestContext struct {\n    shared *hyperpb.Shared\n    types map[string]*hyperpb.MessageType\n    // Additional context fields...\n}\n\nfunc (c *requestContext) Handle(req Request) {\n    msgType := c.types[req.Type]\n    msg := c.shared.NewMessage(msgType)\n    defer c.shared.Free()\n\n    c.process(msg, req, ...)\n}\n```\n\nBeware that `msg` must not outlive the call to `Shared.Free`; failure to do so\nwill result in memory errors that Go cannot protect you from.\n\n### Profile-Guided Optimization (PGO)\n\n`hyperpb` supports online PGO for squeezing extra performance out of the parser\nby optimizing the parser with knowledge of what the average message actually\nlooks like. For example, using PGO, the parser can predict the expected size of\nrepeated fields and allocate more intelligently.\n\nFor example, suppose you have a corpus of messages for a particular type. You\ncan build an optimized type, using that corpus as the profile, using\n`Type.Recompile`:\n\n```go\nfunc compilePGO(\n    md protoreflect.MessageDescriptor,\n    corpus [][]byte,\n) (*hyperpb.MessageType, error) {\n    // Compile the type without any profiling information.\n    msgType := hyperpb.CompileMessageDescriptor(md)\n\n    // Construct a new profile recorder.\n    profile := msgType.NewProfile()\n\n    // Parse all of the specimens in the corpus, making sure to record a profile\n    // for all of them.\n    s := new(hyperpb.Shared)\n    for _, specimen := range corpus {\n        if err := s.NewMessage(msgType).Unmarshal(\n            specimen,\n            hyperpb.WithRecordProfile(profile, 1.0),\n        ); err != nil {\n            return nil, err\n        }\n        s.Free()\n    }\n\n    // Recompile with the profile.\n    return msgType.Recompile(profile), nil\n}\n```\n\nUsing a custom sampling rate in `hyperpb.WithRecordProfile`, it's possible to\nsample data on-line as part of a request flow, and recompile dynamically:\n\n```go\ntype requestContext struct {\n    shared *hyperpb.Shared\n    types map[string]*typeInfo\n    // Additional context fields...\n}\n\ntype typeInfo struct {\n    msgType atomic.Pointer[hyperpb.MessageType]\n    prof atomic.Pointer[hyperpb.Profile]\n    seen atomic.Int64\n}\n\nfunc (c *requestContext) Handle(req Request) {\n    // Look up the type in the context's type map.\n    typeInfo := c.types[req.Type]\n\n    // Parse the type as usual.\n    msg := c.shared.NewMessage(typeInfo.msgType.Load())\n    defer c.shared.Free()\n\n    if err := msg.Unmarshal(\n        data,\n        // Only profile 1% of messages.\n        hyperpb.WithRecordProfile(typeInfo.prof.Load(), 0.01),\n    ); err != nil {\n        // Process error...\n    }\n    typeInfo.seen.Add(1)\n\n    // Every 100,000 messages, spawn a goroutine to asynchronously recompile the type.\n    if typeInfo.seen.Load() % 100000 == 0 {\n        go func() {\n            prof := typeInfo.prof.Load()\n            if !typeInfo.prof.CompareAndSwap(prof, nil) {\n                // Avoid a race condition.\n                return\n            }\n\n            // Recompile the type. This is gonna be really slow, because\n            // the compiler is slow, which is why we're doing it asynchronously.\n            typeInfo.msgType.Store(typeInfo.msgType.Load().Recompile(typeInfo.prof.Load()))\n            typeInfo.prof.Store(typeInfo.msgType.Load().NewProfile())\n        }\n    }\n\n    // Do something with msg.\n}\n```\n\n## Compatibility\n\n`hyperpb` is experimental software, and the API may change drastically before\n`v1`. It currently implements all Protobuf language constructs. It does not\nimplement mutation of parsed messages, however.\n\n## Contributing\n\nFor a detailed explanation of the implementation details of `hyperpb`, see\nthe [`DESIGN.md`](DESIGN.md) file. Contributions that significantly change the\nparser will require benchmarks; you can run them with `make bench`.\n\n## Legal\n\nOffered under the [Apache 2 license][license].\n\n[dynamicpb]: https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb\n[upb]: https://github.com/protocolbuffers/protobuf/tree/main/upb\n[license]: https://github.com/bufbuild/hyperpb-go/blob/main/LICENSE","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufbuild%2Fhyperpb-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbufbuild%2Fhyperpb-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbufbuild%2Fhyperpb-go/lists"}