{"id":13508974,"url":"https://github.com/ahamez/protox","last_synced_at":"2025-05-16T07:06:45.708Z","repository":{"id":14604406,"uuid":"82186020","full_name":"ahamez/protox","owner":"ahamez","description":"A reasonably fast, easy to use and 100% conformant Elixir library for Google Protocol Buffers (aka protobuf)","archived":false,"fork":false,"pushed_at":"2025-05-12T05:51:47.000Z","size":1959,"stargazers_count":285,"open_issues_count":0,"forks_count":19,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-05-12T06:36:47.602Z","etag":null,"topics":["elixir","protobuf","protobuf-message","protobuf-runtime","protoc","protocol-buffers"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahamez.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["ahamez"]}},"created_at":"2017-02-16T13:55:34.000Z","updated_at":"2025-05-12T05:51:49.000Z","dependencies_parsed_at":"2023-12-13T10:31:49.431Z","dependency_job_id":"542767d4-741f-4bb9-88d9-e69450ef5f77","html_url":"https://github.com/ahamez/protox","commit_stats":{"total_commits":1014,"total_committers":20,"mean_commits":50.7,"dds":"0.12623274161735698","last_synced_commit":"c229fd33c02bddffcad54eb4c0d27e70bd82cc4c"},"previous_names":[],"tags_count":69,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahamez%2Fprotox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahamez%2Fprotox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahamez%2Fprotox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahamez%2Fprotox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahamez","download_url":"https://codeload.github.com/ahamez/protox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254485066,"owners_count":22078767,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","protobuf","protobuf-message","protobuf-runtime","protoc","protocol-buffers"],"created_at":"2024-08-01T02:01:01.203Z","updated_at":"2025-05-16T07:06:40.699Z","avatar_url":"https://github.com/ahamez.png","language":"Elixir","funding_links":["https://github.com/sponsors/ahamez"],"categories":["Protocols"],"sub_categories":[],"readme":"# Protox\n\n[![Elixir CI](https://github.com/ahamez/protox/actions/workflows/elixir.yml/badge.svg)](https://github.com/ahamez/protox/actions/workflows/elixir.yml) [![Coverage Status](https://coveralls.io/repos/github/ahamez/protox/badge.svg?branch=master)](https://coveralls.io/github/ahamez/protox?branch=master) [![Hex.pm Version](http://img.shields.io/hexpm/v/protox.svg)](https://hex.pm/packages/protox) [![Hex Docs](https://img.shields.io/badge/hex-docs-brightgreen.svg)](https://hexdocs.pm/protox/) [![License](https://img.shields.io/hexpm/l/protox.svg)](https://github.com/ahamez/protox/blob/master/LICENSE)\n\nProtox is an Elixir library for working with [Google's Protocol Buffers](https://developers.google.com/protocol-buffers), versions 2 and 3, supporting binary encoding and decoding.\n\nThe primary objective of Protox is **reliability**: it uses [property testing](https://github.com/alfert/propcheck), [mutation testing](https://github.com/devonestes/muzak) and has a [near 100% code coverage](https://coveralls.io/github/ahamez/protox?branch=master). Protox [passes all the tests](#conformance) of the conformance checker provided by Google.\n\n\n\u003e [!NOTE]\n\u003e If you're using version 1, please see how to migrate to version 2 [here](documentation/v1_to_v2_migration.md).\n\n## Example\n\nGiven the following protobuf definition:\n```proto\nmessage Msg{\n  int32 a = 1;\n  map\u003cint32, string\u003e b = 2;\n}\n```\n\nProtox will create a regular Elixir `Msg` struct:\n```elixir\niex\u003e msg = %Msg{a: 42, b: %{1 =\u003e \"a map entry\"}}\niex\u003e {:ok, iodata, iodata_size} = Msg.encode(msg)\n\niex\u003e binary = # read binary from a socket, a file, etc.\niex\u003e {:ok, msg} = Msg.decode(binary)\n```\n\n## Usage\n\nYou can use Protox in two ways:\n\n1. pass the protobuf schema ([as an inlined schema](#usage-with-an-inlined-schema) or as a [list of files](#usage-with-files)) to the `Protox` macro;\n2. [generate](#files-generation) Elixir source code files with the mix task `protox.generate`.\n\n## Table of contents\n\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n- [Usage with an inlined schema](#usage-with-an-inlined-schema)\n- [Usage with files](#usage-with-files)\n- [Encode](#encode)\n- [Decode](#decode)\n- [Packages and namespaces](#packages-and-namespaces)\n- [Specify include path](#specify-include-path)\n- [Files generation](#files-generation)\n- [Unknown fields](#unknown-fields)\n- [Unsupported features](#unsupported-features)\n- [Implementation choices](#implementation-choices)\n- [Generated code reference and types mapping](#generated-code-reference-and-types-mapping)\n- [Conformance](#conformance)\n- [Benchmark](#benchmark)\n- [Contributing](#contributing)\n\n## Prerequisites\n\n- Elixir \u003e= 1.15 and OTP \u003e= 26\n- [protoc](https://github.com/protocolbuffers/protobuf/releases) \u003e= 3.0 *This dependency is only required at compile-time*. It must be available in `$PATH`.\n\n## Installation\n\nAdd `:protox` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [{:protox, \"~\u003e 2.0\"}]\nend\n```\n\n## Usage with an inlined schema\n\nThe following example generates two modules, `Baz` and `Foo`:\n\n```elixir\ndefmodule MyModule do\n  use Protox, schema: \"\"\"\n  syntax = \"proto3\";\n\n  message Baz {\n  }\n\n  message Foo {\n    int32 a = 1;\n    map\u003cint32, Baz\u003e b = 2;\n  }\n  \"\"\"\nend\n```\n\n\u003e [!NOTE]\n\u003e The module in which the `Protox` macro is called is ignored and does not appear in the names of the generated modules. To include the enclosing module’s name, use the `namespace` option, see [here](#prepend-namespaces).\n\n## Usage with files\n\nUse the `:files` option to pass a list of files:\n\n```elixir\ndefmodule MyModule do\n  use Protox, files: [\n    \"./defs/foo.proto\",\n    \"./defs/bar.proto\",\n    \"./defs/baz/fiz.proto\"\n  ]\nend\n```\n\n## Encode\n\nHere's how to encode a message to binary protobuf:\n\n```elixir\nmsg = %Foo{a: 3, b: %{1 =\u003e %Baz{}}}\n{:ok, iodata, iodata_size} = Protox.encode(msg)\n# or using the bang version\n{iodata, iodata_size} = Protox.encode!(msg)\n```\n\nIt's also possible to call `encode/1` and `encode!/1` directly on the generated structures:\n\n```elixir\n{:ok, iodata, iodata_size} = Foo.encode(msg)\n{iodata, iodata_size} = Foo.encode!(msg)\n```\n\n\u003e [!NOTE]\n\u003e `encode/1` and `encode!/1` return an [IO data](https://hexdocs.pm/elixir/IO.html#module-use-cases-for-io-data) for efficiency reasons. Such IO data can be used directly with files or sockets write operations:\n\u003e ```elixir\n\u003e iex\u003e {iodata, _iodata_size} = Protox.encode!(%Foo{a: 3, b: %{1 =\u003e %Baz{}}})\n\u003e {[\"\\b\", \u003c\u003c3\u003e\u003e, \u003c\u003c18, 4, 8\u003e\u003e, \u003c\u003c1\u003e\u003e, \u003c\u003c18\u003e\u003e, [\u003c\u003c0\u003e\u003e, []]], 8}\n\u003e iex\u003e {:ok, file} = File.open(\"msg.bin\", [:write])\n\u003e {:ok, #PID\u003c0.1023.0\u003e}\n\u003e iex\u003e IO.binwrite(file, iodata)\n\u003e :ok\n\u003e ```\n\u003e\n\u003e Use [`:binary.list_to_bin/1`](https://erlang.org/doc/man/binary.html#list_to_bin-1) or [`IO.iodata_to_binary`](https://hexdocs.pm/elixir/IO.html#iodata_to_binary/1) if you need to get a binary from an IO data.\n\n\n## Decode\n\nHere's how to decode a message from binary protobuf:\n\n```elixir\n{:ok, msg} = Protox.decode(\u003c\u003c8, 3, 18, 4, 8, 1, 18, 0\u003e\u003e, Foo)\n# or using the bang version\nmsg = Protox.decode!(\u003c\u003c8, 3, 18, 4, 8, 1, 18, 0\u003e\u003e, Foo)\n```\n\nIt's also possible to call `decode/1` and `decode!/1` directly on the generated structures:\n\n```elixir\n{:ok, msg} = Foo.decode(\u003c\u003c8, 3, 18, 4, 8, 1, 18, 0\u003e\u003e)\nmsg = Foo.decode!(\u003c\u003c8, 3, 18, 4, 8, 1, 18, 0\u003e\u003e)\n```\n\n## Packages and namespaces\n\n### Packages\n\nProtox honors the [`package`](https://protobuf.dev/programming-guides/proto3/#packages) directive:\n\n```proto\npackage abc.def;\nmessage Baz {}\n```\n\nThe example above will be translated to `Abc.Def.Baz` (note the [camelization](#implementation-choices) of package `abc.def` to `Abc.Def`).\n\n### Prepend namespaces\nIn addition, Protox provides the possibility to prepend a namespace with the `:namespace` option:\n\n```elixir\ndefmodule Bar do\n  use Protox, schema: \"\"\"\n    syntax = \"proto3\";\n\n    package abc;\n\n    message Msg {\n        int32 a = 1;\n      }\n    \"\"\",\n    namespace: __MODULE__\nend\n```\n\nIn this example, the module `Bar.Abc.Msg` is generated:\n\n```elixir\nmsg = %Bar.Abc.Msg{a: 42}\n```\n\n## Specify include path\n\nOne or more include paths (directories in which to search for imports) can be specified using the `:paths` option:\n\n```elixir\ndefmodule Baz do\n  use Protox,\n    files: [\n      \"./defs1/prefix/foo.proto\",\n      \"./defs1/prefix/bar.proto\",\n      \"./defs2/prefix/baz/baz.proto\"\n    ],\n    paths: [\n      \"./defs1\",\n      \"./defs2\"\n    ]\nend\n```\n\n\u003e [!NOTE]\n\u003e It corresponds to the `-I` option of protoc.\n\n## Files generation\n\nIt's possible to generate Elixir source code files with the mix task `protox.generate`:\n\n```shell\nprotox.generate --output-path=/path/to/messages.ex protos/foo.proto protos/bar.proto\n```\n\nThe files will be usable in any project as long as Protox is declared in the dependencies as functions from its runtime are used.\n\n\u003e [!NOTE]\n\u003e protoc is not needed to compile the generated files.\n\n### Options\n\n* `--output-path`\n\n  The path to the file to be generated or to the destination folder when generating multiple files.\n\n* `--include-path`\n\n  Specifies the [include path](#specify-include-path). If multiple include paths are needed, add more `--include-path` options.\n\n* `--multiple-files`\n\n  Generates one file per Elixir module. It's useful for definitions with a lot of messages as the compilation will be parallelized.\n  When generating multiple files, the `--output-path` option must point to a directory.\n\n* `--namespace`\n\n  [Prepends a namespace](#prepend-namespaces) to all generated modules.\n\n## Unknown fields\n\n[Unknown fields](https://developers.google.com/protocol-buffers/docs/proto3#unknowns) are fields that are present on the wire but which do not correspond to an entry in the protobuf definition. Typically, it occurs when the sender has a newer version of the protobuf definition. It enables backwards compatibility as the receiver with an old version of the protobuf definition will still be able to decode old fields.\n\nWhen unknown fields are encountered at decoding time, they are kept in the decoded message. It's possible to access them with the  `unknown_fields/1` function defined with the message.\n\n```elixir\niex\u003e msg = Msg.decode!(\u003c\u003c8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83\u003e\u003e)\n%Msg{a: 42, b: \"\", z: -42, __uf__: [{5, 2, \u003c\u003c121, 97, 121, 101\u003e\u003e}]}\n\niex\u003e Msg.unknown_fields(msg)\n[{5, 2, \u003c\u003c121, 97, 121, 101\u003e\u003e}]\n```\n\nYou must use `unknown_fields/1` as the name of the field (e.g. `__uf__` in the above example) is generated at compile-time to avoid collision with the actual fields of the Protobuf message. This function returns a list of tuples `{tag, wire_type, bytes}`. For more information, please see the [protobuf encoding guide](https://developers.google.com/protocol-buffers/docs/encoding).\n\n\u003e [!NOTE]\n\u003e Unknown fields are retained when re-encoding the message.\n\n## Unsupported features\n\n* The [Any](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#any) well-known type is partially supported: you can manually unpack the embedded message after decoding and conversely pack it before encoding;\n* Groups ([deprecated in protobuf](https://protobuf.dev/programming-guides/proto2/#groups));\n* All [options](https://developers.google.com/protocol-buffers/docs/proto3#options) other than `packed` and `default` are ignored as they concern other languages implementation details.\n\n## Implementation choices\n\n*  (__Protobuf 2__) __Required fields__ Protox enforces the presence of required fields; an error is raised when encoding a message with missing required field:\n    ```elixir\n    defmodule Bar do\n      use Protox, schema: \"\"\"\n        syntax = \"proto2\";\n\n        message Required {\n          required int32 a = 1;\n        }\n      \"\"\"\n    end\n\n    iex\u003e Protox.encode!(%Required{})\n    ** (Protox.RequiredFieldsError) Some required fields are not set: [:a]\n    ```\n\n* __Enum aliases__ When decoding, the last encountered constant is used. For instance, in the following example, `:BAR` is always used if the value `1` is read on the wire:\n    ```protobuf\n    enum E {\n      option allow_alias = true;\n      FOO = 0;\n      BAZ = 1;\n      BAR = 1;\n    }\n    ```\n\n* (__Protobuf 2__) __Unset optional fields__ are assigned `nil`. You can use the generated `default/1` function to get the default value of a field:\n    ```elixir\n    defmodule Bar do\n      use Protox,\n      schema: \"\"\"\n        syntax = \"proto2\";\n\n        message Foo {\n          optional int32 a = 1 [default = 42];\n        }\n      \"\"\"\n    end\n\n    iex\u003e %Foo{}.a\n    nil\n\n    iex\u003e Foo.default(:a)\n    {:ok, 42}\n    ```\n\n* (__Protobuf 3__) __Unset fields__ are assigned to their [default values](https://developers.google.com/protocol-buffers/docs/proto3#default). However, if you use the `optional` keyword (available in protoc \u003e= 3.15), then unset fields are assigned `nil`:\n    ```elixir\n    defmodule Bar do\n      use Protox,\n      schema: \"\"\"\n        syntax = \"proto3\";\n\n        message Foo {\n          int32 a = 1;\n          optional int32 b = 2;\n        }\n      \"\"\"\n    end\n\n    iex\u003e %Foo{}.a\n    0\n\n    iex\u003e Foo.default(:a)\n    {:ok, 0}\n\n    iex\u003e %Foo{}.b\n    nil\n\n    iex\u003e Foo.default(:b)\n    {:error, :no_default_value}\n    ```\n\n* __Messages and enums names__ are converted using the [`Macro.camelize/1`](https://hexdocs.pm/elixir/Macro.html#camelize/1) function.\n  Thus, in the following example, `non_camel_message` becomes `NonCamelMessage`, but the field `non_camel_field` is left unchanged:\n    ```elixir\n    defmodule Bar do\n      use Protox,\n      schema: \"\"\"\n        syntax = \"proto3\";\n\n        message non_camel_message {\n        }\n\n        message CamelMessage {\n          int32 non_camel_field = 1;\n        }\n      \"\"\"\n    end\n\n    iex\u003e msg = %NonCamelMessage{}\n    %NonCamelMessage{__uf__: []}\n\n    iex\u003e msg = %CamelMessage{}\n    %CamelMessage{__uf__: [], non_camel_field: 0}\n    ```\n\n## Generated code reference and types mapping\n\n- The detailed reference of the generated code is available in [documentation/reference.md](documentation/reference.md).\n- Please see [documentation/types_mapping.md](documentation/types_mapping.md) to see how protobuf types are mapped to Elixir types.\n\n\n## Conformance\n\nThe Protox library has been thoroughly tested using the conformance checker [provided by Google](https://github.com/protocolbuffers/protobuf/tree/master/conformance).\n\nTo launch these conformance tests, use the `protox.conformance` mix task:\n  ```\n  $ mix protox.conformance\n  WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n  I0000 00:00:1738246114.224098 3490144 conformance_test_runner.cc:394] ./protox_conformance\n  CONFORMANCE TEST BEGIN ====================================\n\n  CONFORMANCE SUITE PASSED: 1368 successes, 1307 skipped, 0 expected failures, 0 unexpected failures.\n\n  WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n  I0000 00:00:1738246115.065491 3495574 conformance_test_runner.cc:394] ./protox_conformance\n  CONFORMANCE TEST BEGIN ====================================\n\n  CONFORMANCE SUITE PASSED: 0 successes, 414 skipped, 0 expected failures, 0 unexpected failures.\n  ```\n\n\u003e[!NOTE]\n\u003e A report will be generated in the directory `conformance_report`.\n\n\n## Benchmark\n\nPlease see [benchmark/launch_benchmark.md](benchmark/launch_benchmark.md) for more information on how to launch benchmark.\n\n## Contributing\n\nPlease see [CONTRIBUTING.md](./CONTRIBUTING.md) for more information on how to contribute.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahamez%2Fprotox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahamez%2Fprotox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahamez%2Fprotox/lists"}