{"id":13643521,"url":"https://github.com/VictoriaMetrics/easyproto","last_synced_at":"2025-04-21T02:30:31.190Z","repository":{"id":217162057,"uuid":"743212031","full_name":"VictoriaMetrics/easyproto","owner":"VictoriaMetrics","description":"Simple building blocks for protobuf marshaling and unmarshaling","archived":false,"fork":false,"pushed_at":"2024-06-07T07:33:03.000Z","size":43,"stargazers_count":177,"open_issues_count":4,"forks_count":3,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-04-08T16:11:21.646Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VictoriaMetrics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-14T17:04:07.000Z","updated_at":"2025-03-25T15:11:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"b31e3acc-66f5-4c5e-99dc-8d1a8ba0a26b","html_url":"https://github.com/VictoriaMetrics/easyproto","commit_stats":null,"previous_names":["victoriametrics/easyproto"],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VictoriaMetrics%2Feasyproto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VictoriaMetrics%2Feasyproto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VictoriaMetrics%2Feasyproto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VictoriaMetrics%2Feasyproto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VictoriaMetrics","download_url":"https://codeload.github.com/VictoriaMetrics/easyproto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249986022,"owners_count":21356310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:48.759Z","updated_at":"2025-04-21T02:30:30.850Z","avatar_url":"https://github.com/VictoriaMetrics.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"[![GoDoc](https://godoc.org/github.com/VictoriaMetrics/easyproto?status.svg)](http://godoc.org/github.com/VictoriaMetrics/easyproto)\n\n# easyproto\n\nPackage [github.com/VictoriaMetrics/easyproto](http://godoc.org/github.com/VictoriaMetrics/easyproto) provides simple building blocks\nfor marshaling and unmarshaling of [protobuf](https://protobuf.dev/) messages with [proto3 encoding](https://protobuf.dev/programming-guides/encoding/).\n\n## Features\n\n- There is no need in [protoc](https://grpc.io/docs/protoc-installation/) or [go generate](https://go.dev/blog/generate) -\n  just write simple maintainable code for marshaling and unmarshaling protobuf messages.\n- `easyproto` doesn't increase your binary size by tens of megabytes unlike traditional `protoc`-combiled code may do.\n- `easyproto` allows writing zero-alloc code for marshaling and unmarshaling of arbitrary complex protobuf messages. See [examples](#examples).\n\n## Restrictions\n\n- It supports only [proto3 encoding](https://protobuf.dev/programming-guides/encoding/), e.g. it doesn't support `proto2` encoding\n  features such as [proto2 groups](https://protobuf.dev/programming-guides/proto2/#groups).\n- It doesn't provide helpers for marshaling and unmarshaling of [well-known types](https://protobuf.dev/reference/protobuf/google.protobuf/),\n  since they aren't used too much in practice.\n\n## Examples\n\nSuppose you need marshaling and unmarshaling of the following `timeseries` message:\n\n```proto\nmessage timeseries {\n  string name = 1;\n  repeated sample samples = 2;\n}\n\nmessage sample {\n  double value = 1;\n  int64 timestamp = 2;\n}\n```\n\nAt first let's create the corresponding data structures in Go:\n\n```go\ntype Timeseries struct {\n\tName    string\n\tSamples []Sample\n}\n\ntype Sample struct {\n\tValue     float64\n\tTimestamp int64\n}\n```\n\nSince you write the code on yourself without any `go generate` and `protoc` invocations,\nyou are free to use arbitrary fields and methods in these structs. You can also specify the most suitable types for these fields.\nFor example, the `Sample` struct may be written as the following if you need an ability to detect empty values and timestamps:\n\n```go\ntype Sample struct {\n\tValue     *float64\n\tTimestamp *int64\n}\n```\n\n* [How to marshal `Timeseries` struct to protobuf message](#marshaling)\n* [How to unmarshal protobuf message to `Timeseries` struct](#unmarshaling)\n\n### Marshaling\n\nThe following code can be used for marshaling `Timeseries` struct to protobuf message:\n\n```go\nimport (\n\t\"github.com/VictoriaMetrics/easyproto\"\n)\n\n// MarshalProtobuf marshals ts into protobuf message, appends this message to dst and returns the result.\n//\n// This function doesn't allocate memory on repeated calls.\nfunc (ts *Timeseries) MarshalProtobuf(dst []byte) []byte {\n\tm := mp.Get()\n\tts.marshalProtobuf(m.MessageMarshaler())\n\tdst = m.Marshal(dst)\n\tmp.Put(m)\n\treturn dst\n}\n\nfunc (ts *Timeseries) marshalProtobuf(mm *easyproto.MessageMarshaler) {\n\tmm.AppendString(1, ts.Name)\n\tfor _, s := range ts.Samples {\n\t\ts.marshalProtobuf(mm.AppendMessage(2))\n\t}\n}\n\nfunc (s *Sample) marshalProtobuf(mm *easyproto.MessageMarshaler) {\n\tmm.AppendDouble(1, s.Value)\n\tmm.AppendInt64(2, s.Timestamp)\n}\n\nvar mp easyproto.MarshalerPool\n```\n\nNote that you are free to modify this code according to your needs, since you write and maintain it.\nFor example, you can construct arbitrary protobuf messages on the fly without the need to prepare the source struct for marshaling:\n\n```go\nfunc CreateProtobufMessageOnTheFly() []byte {\n\t// Dynamically construct timeseries message with 10 samples\n\tvar m easyproto.Marshaler\n\tmm := m.MessageMarshaler()\n\tmm.AppendString(1, \"foo\")\n\tfor i := 0; i \u003c 10; i++ {\n\t\tmmSample := mm.AppendMessage(2)\n\t\tmmSample.AppendDouble(1, float64(i)/10)\n\t\tmmSample.AppendInt64(2, int64(i)*1000)\n\t}\n\treturn m.Marshal(nil)\n}\n```\n\nThis may be useful in tests.\n\n### Unmarshaling\n\nThe following code can be used for unmarshaling [`timeseries` message](#examples) into `Timeseries` struct:\n\n```go\n// UnmarshalProtobuf unmarshals ts from protobuf message at src.\nfunc (ts *Timeseries) UnmarshalProtobuf(src []byte) (err error) {\n\t// Set default Timeseries values\n\tts.Name = \"\"\n\tts.Samples = ts.Samples[:0]\n\n\t// Parse Timeseries message at src\n\tvar fc easyproto.FieldContext\n\tfor len(src) \u003e 0 {\n\t\tsrc, err = fc.NextField(src)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"cannot read next field in Timeseries message\")\n\t\t}\n\t\tswitch fc.FieldNum {\n\t\tcase 1:\n\t\t\tname, ok := fc.String()\n\t\t\tif !ok {\n\t\t\t\treturn fmt.Errorf(\"cannot read Timeseries name\")\n\t\t\t}\n\t\t\t// name refers to src. This means that the name changes when src changes.\n\t\t\t// Make a copy with strings.Clone(name) if needed.\n\t\t\tts.Name = name\n\t\tcase 2:\n\t\t\tdata, ok := fc.MessageData()\n\t\t\tif !ok {\n\t\t\t\treturn fmt.Errorf(\"cannot read Timeseries sample data\")\n\t\t\t}\n\t\t\tts.Samples = append(ts.Samples, Sample{})\n\t\t\ts := \u0026ts.Samples[len(ts.Samples)-1]\n\t\t\tif err := s.UnmarshalProtobuf(data); err != nil {\n\t\t\t\treturn fmt.Errorf(\"cannot unmarshal sample: %w\", err)\n\t\t\t}\n\t\t}\n\t}\n\treturn nil\n}\n\n// UnmarshalProtobuf unmarshals s from protobuf message at src.\nfunc (s *Sample) UnmarshalProtobuf(src []byte) (err error) {\n\t// Set default Sample values\n\ts.Value = 0\n\ts.Timestamp = 0\n\n\t// Parse Sample message at src\n\tvar fc easyproto.FieldContext\n\tfor len(src) \u003e 0 {\n\t\tsrc, err = fc.NextField(src)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"cannot read next field in sample\")\n\t\t}\n\t\tswitch fc.FieldNum {\n\t\tcase 1:\n\t\t\tvalue, ok := fc.Double()\n\t\t\tif !ok {\n\t\t\t\treturn fmt.Errorf(\"cannot read sample value\")\n\t\t\t}\n\t\t\ts.Value = value\n\t\tcase 2:\n\t\t\ttimestamp, ok := fc.Int64()\n\t\t\tif !ok {\n\t\t\t\treturn fmt.Errorf(\"cannot read sample timestamp\")\n\t\t\t}\n\t\t\ts.Timestamp = timestamp\n\t\t}\n\t}\n\treturn nil\n}\n```\n\nYou are free to modify this code according to your needs, since you wrote it and you maintain it.\n\nIt is possible to extract the needed data from arbitrary protobuf messages without the need to create a destination struct.\nFor example, the following code extracts `timeseries` name from protobuf message, while ignoring all the other fields:\n\n```go\nfunc GetTimeseriesName(src []byte) (name string, err error) {\n\tvar fc easyproto.FieldContext\n\tfor len(src) \u003e 0 {\n\t\tsrc, err = fc.NextField(src)\n\t\tif src != nil {\n\t\t\treturn \"\", fmt.Errorf(\"cannot read the next field\")\n\t\t}\n\t\tif fc.FieldNum == 1 {\n\t\t\tname, ok := fc.String()\n\t\t\tif !ok {\n\t\t\t\treturn \"\", fmt.Errorf(\"cannot read timeseries name\")\n\t\t\t}\n\t\t\t// Return a copy of name, since name refers to src.\n\t\t\treturn strings.Clone(name), nil\n\t\t}\n\t}\n\treturn \"\", fmt.Errorf(\"timeseries name isn't found in the message\")\n}\n```\n\n## Users\n\n`easyproto` is used in the following projects:\n\n- [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVictoriaMetrics%2Feasyproto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FVictoriaMetrics%2Feasyproto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FVictoriaMetrics%2Feasyproto/lists"}