{"id":13681929,"url":"https://github.com/AppsFlyer/pronto","last_synced_at":"2025-04-30T06:32:57.036Z","repository":{"id":44548466,"uuid":"374575492","full_name":"AppsFlyer/pronto","owner":"AppsFlyer","description":"Clojure support for protocol buffers","archived":false,"fork":false,"pushed_at":"2024-05-22T09:57:20.000Z","size":932,"stargazers_count":106,"open_issues_count":7,"forks_count":7,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-11T17:14:49.513Z","etag":null,"topics":["clojure","protobuf","protobuf-java","protobuf3","protocol-buffers"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AppsFlyer.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-07T07:33:51.000Z","updated_at":"2024-09-06T17:53:27.000Z","dependencies_parsed_at":"2024-01-08T18:03:37.971Z","dependency_job_id":"8150d904-6647-46b8-aaa1-f4f698dcd7c4","html_url":"https://github.com/AppsFlyer/pronto","commit_stats":{"total_commits":217,"total_committers":9,"mean_commits":24.11111111111111,"dds":"0.11981566820276501","last_synced_commit":"b5f48afadfcb4a5b48b468db9669a44122856c47"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AppsFlyer%2Fpronto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AppsFlyer%2Fpronto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AppsFlyer%2Fpronto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AppsFlyer%2Fpronto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AppsFlyer","download_url":"https://codeload.github.com/AppsFlyer/pronto/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224201779,"owners_count":17272644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","protobuf","protobuf-java","protobuf3","protocol-buffers"],"created_at":"2024-08-02T13:01:37.835Z","updated_at":"2024-11-12T01:30:43.307Z","avatar_url":"https://github.com/AppsFlyer.png","language":"Clojure","readme":"# pronto\n\n[![Coverage Status](https://coveralls.io/repos/github/AppsFlyer/pronto/badge.svg?branch=master)](https://coveralls.io/github/AppsFlyer/pronto?branch=master)\n[![Clojars Project](https://img.shields.io/clojars/v/com.appsflyer/pronto.svg)](https://clojars.org/com.appsflyer/pronto)\n[![cljdoc badge](https://cljdoc.org/badge/com.appsflyer/pronto)](https://cljdoc.org/d/com.appsflyer/pronto/CURRENT)\n\nA library for using [Protocol Buffers](https://github.com/protocolbuffers/protobuf) 3 in Clojure.\n\n## Rationale\n\nThe guiding principles behind `pronto` are:\n\n* **Idiomatic interaction**: Use Protocol Buffer POJOs (`protoc` generated) as though they were native Clojure data structures, allowing for data-driven programming.\n* **Minimalistic**: `pronto` is behavioral only: it is only concerned with making POJOs mimic Clojure collections. Data is still stored in the POJOs, and no\nkind of reflective/dynamic APIs are used. This also has the benefit that [unknown fields](https://developers.google.com/protocol-buffers/docs/proto3#unknowns) are not lost \nduring serialization.\n* **Runtime Type Safety**: The schema cannot be broken - `pronto` fails-fast when `assoc`ing a key not present in the schema or a value of the wrong type.\nThis guarantees that schema errors are detected immediately rather than at some undefined time in the future (perhaps too late) or worse -- dropped and\nignored completely.\n* **Performant**: Present a minimal CPU/memory overhead: `pronto` compiles very thin wrapper classes and avoids reflection completely.\n\n## Installation\nAdd a dependency to your `project.clj` file:\n\n           [com.appsflyer/pronto \"3.0.0\"]\n\nNote that the library comes with no Java protobuf dependencies of its own and they are expected to be provided by consumers of the library, with a minimal version of `3.15.0`.\n\n## How does it work?\n\nThe main abstraction in `pronto` is the `proto-map`, a type of map which can be used as a regular Clojure map, but rejects any operations\nwhich would break its schema. The library generates a bespoke `proto-map` class for every `protoc`-generated Java class (POJO).\n\nEvery `proto-map` \n* Holds an underlying instance of the actual POJO.\n* Can be used as Clojure maps and support most Clojure semantics and abstractions by implementing all the appropriate internal Clojure interfaces \n(see [fine print](#fine-print-please-read)).\n* Is immutable, i.e, `assoc`ing creates a new proto-map instance around a new POJO instance.\n\n## Quick example\n\nLet's use this [example](https://github.com/AppsFlyer/pronto/blob/master/resources/proto/people.proto):\n\n```clj\n(import 'protogen.generated.People$Person)\n\n(require '[pronto.core :as p])\n\n(p/defmapper my-mapper [People$Person])\n```\n\n`defmapper` is a macro which generates new `proto-map` classes for the supplied class and for any message type dependency it has. It also defines a var at the call-site, which serves as a handle to interact with the library later on.\n\nNow we can work with protobuf while writing idiomatic Clojure code:\n\n```clj\n(-\u003e (p/proto-map mapper People$Person) ;; create a Person proto-map\n    (assoc :name \"Rich\" :id 0 :pet_names [\"FOO\" \"BAR\"])\n    (update :pet_names #(map clojure.string/lower-case %))\n    (assoc-in [:address :street] \"Broadway\"))\n```\n\nInternally, field reads and writes are delegated directly to the underlying POJO.\nFor example, `(:name person-map)` will call `Person.getName` and `(assoc person-map :name \"John\")` will call `Person.Builder.setName`.\n\nSchema-breaking operations will fail:\n\n```clj\n(assoc person-map :no-such-key 12345)\n=\u003e Execution error (IllegalArgumentException) at user.People$PersonMap/assoc\nNo such field :no-such-key\n\n(assoc person-map :name 12345)\n=\u003e Execution error:  {:error :invalid-type,\n                      :class protogen.generated.People$Person,\n                      :field \"name\",\n                      :expected-type java.lang.String,\n                      :actual-type java.lang.Long,\n                      :value 12345}\n```\n\n## Fine print - please read\n\nIt is important to realize that while `proto-maps`s look and feel like Clojure maps for the most part, their semantics\nare not always identical. Clojure maps are dynamic and open; Protocol-buffers are static and closed. This leads to\nseveral design decisions, where we usually preferred to stick to Protocol-buffers' semantics rather than Clojure's.\nThis is done in order to remove ambiguity, and because we assume that a protocol-buffers user would like to ensure\nthe properties for which they decided to use them in the first place are maintained.\n\nThe main differences and the reasoning behind them are as follows:\n\n* A `proto-map` contains the **entire set of keys** defined in a schema (as Clojure keywords) -- the schema is the source of truth and it is always present in its entirety.\n* `dissoc` is unsupported -- for the reason above.\n* Trying to `get` a key not in the map will throw an error, rather than return `nil`, for two reasons; First, `proto-maps` are closed\nand can't be used as a general-purpose container of key-value pairs. Therefor, this is probably a mistake and we'd like to give the user immediate feedback.\nSecond, returning `nil` could lead to strange ambiguities -- see below.\n* Associng a key not present in the schema is an error -- maintain schema correctness.\n* Associng a value of the wrong type is an error -- maintain schema correctness.\n* To `nil` or not to `nil`: protocol buffers in Java have no notion of scalar nullability. Scalar fields are always initialized and present.\nWhen unset, they take on their \"zero-value\" rather than `null`. However, for message type fields it is possible to check whether set or not.\n  * Scalar fields will never be `nil`. When unset, their value will be whatever the default value is for the respective type. However, protobuf provides\n [boxed](#well-known-types)  wrappers for primitive types, which `pronto` automatically recognizes and inlines into the proto-map.\n  * The value message type fields will be `nil` when they are unset. Associng `nil` to a message type field will clear it.\n\n## Usage guide\n\n### Creating a new map:\n\n```clj\n(import 'protogen.generated.People$Person)\n\n(require '[pronto.core :as p])\n\n(p/defmapper my-mapper People$Person)\n\n;; Create a new empty Person proto-map:\n(p/proto-map my-mapper People$Person)\n\n;; Serialize a byte array into a proto-map (and accompanying POJO):\n(p/proto-map-\u003ebytes my-proto-map)\n\n;; Deserialize byte array into a proto-map (and accompanying POJO):\n(p/bytes-\u003eproto-map my-mapper People$Person (read-person-byte-array-from-kafka))\n\n;; Generate a new proto-map from a Clojure map adhering to the schema:\n(p/clj-map-\u003eproto-map my-mapper People$Person {:id 0 :name \"hello\" :address {:city \"London\"}})\n\n;; Wrap around an existing instance of a POJO:\n(let [person (. (People$Person/newBuilder) build)]\n  (p/proto-\u003eproto-map my-mapper person))\n  \n;; Get the underlying POJO of a proto-map:\n(p/proto-map-\u003eproto my-proto-map)\n```\n\n### Pro tip: On `proto-map`s scope\nWhen creating data you can control when exactly you stop working with maps and start working with `proto-map`s. A `proto-map` has the advantage of failing fast. Hence `assoc`ing an invalid field (wrong type, non-existent enum etc.) generates failures at the crime scene. This is a _good_ thing since you want to locate the bug quickly. However, this comes with the cost of creating `proto-maps`.\n\n```clj\n(defn person-with-address [city]\n  (let [addr (p/clj-map-\u003eproto-map my-mapper People$Address {:city city})]\n    (p/clj-map-\u003eproto-map my-mapper People$Person {:id 0 :name \"hello\" :address addr})))\n```\n\nis mouthful. While it fails for every mistake _at the right place_ deeply nested structures creation quickly becomes bloated this way. \n\nHowever, this is also a valid code:\n\n```clj\n(defn person-with-address [city]\n  (-\u003e\u003e {:id 0 :name \"hello\" :address {:city city}}\n       (p/clj-map-\u003eproto-map my-mapper People$Person))\n```\n\nIt has the downside that you might have gotten either `Person` or `Address` wrong, but figuring which one is still easy enough. The point to move from plain maps into `proto-map`s can be chosen freely and should balance this tradeoff. \n\n### Protocol Buffers - Clojure interop\n\n#### Fields\n\nAs discussed [previously](#fine-print-please-read), a `proto-map` contains the **entire set of keys** defined in a schema, represented by a keyword of their original\nfield name in the `.proto` file. \n\nHowever, you can control the naming strategy of keys. For example, if you'd like to use kebab-case:\n\n```clj\n(require '[pronto.utils :as u])\n(p/defmapper my-mapper People$Person \n    :key-name-fn u/-\u003ekebab-case)\n```\n\n#### Scalar fields\nScalar fields are straight-forward in that that they follow the [protobuf Java scalar mappings](https://developers.google.com/protocol-buffers/docs/proto3#scalar).\n\nClojure-specific numeric types such as `Ratio` and `BigInt` are supported as well, and when `assoc`ing them to a map they are converted automatically\nto the underlying field's type.\n\nIt is also important to note that Clojure uses `long`s to represent natural numbers, and these will be down-casted to `int` for integer fields.\n\nIn any case, handling of overflows is left to the user.\n\n#### Message types\nWhen calling `defmapper`, the macro will also find all message types on which the class depends, and generate specialized wrapper types for them as well,\nso you do not have to call `defmapper` recursively yourselves.\n\nWhen reading a field whose type is a message type, a `proto-map` is returned.\n\nIt is possible to assoc both a `proto-map` into a message type field, or a regular Clojure map -- as long as it adheres to the schema.\n\n#### Repeated and maps\nValues of repeated/map fields are returned as Clojure maps/vectors:\n\n```clj\n(:pet_names person-map)\n=\u003e [\"foo\" \"bar\"]\n(:relations person-map)\n=\u003e {\"friend\" {:name \"Joe\" ... } \"cousin\" {:name \"Vinny\" ... }}\n```\n\n#### Enums\nEnumerations are also represented by a keyword:\n\n```clj\n(import 'protogen.generated.People$Like)\n(p/defmapper my-mapper People$Like)\n\n(:level (p/proto-map my-mapper People$Like)) ;; either Level/LOW, Level/MEDIUM, Level/HIGH\n=\u003e :LOW\n```\nIt is possible to use kebab-case (or any other case) for enums. \n\n```clj\n(p/defmapper my-mapper People$Like\n    :enum-value-fn u/-\u003ekebab-case)\n(:level (p/proto-map my-mapper People$Like))\n=\u003e :low\n```\n\nEither a keyword or a Java enum value may be assoced:\n\n```clj\n(assoc (p/proto-map mapper People$Like) :level :HIGH)\n\n(assoc (p/proto-map mapper People$Like) :level People$Level/HIGH)\n```\n\n#### One-of's\none-of's behave like other fields. This means that even when unset, the optional\nfields still exist in the schema with their default values or `nil` in the case of message types.\n\nTo check which one-of is set, use `which-one-of` or `one-of`.\n\nFor example, given this schema:\n```protobuf\nmessage Address {\n  string city = 1;\n  string street = 2;\n  int32 house_num = 3;\n  oneof home {\n    House house = 4;\n    Apartment apartment = 5;\n  }\n}\n```\n\n```clj\n(p/which-one-of (p/proto-map People$Address) :home)\n=\u003e nil\n\n(p/one-of (p/proto-map People$Address) :home)\n=\u003e nil\n\n(p/which-one-of (p/clj-map-\u003eproto-map People$Address {:house {:num_rooms 3}}) :home)\n=\u003e :house\n\n(p/one-of (p/clj-map-\u003eproto-map People$Address {:house {:num_rooms 3}}) :home)\n=\u003e {:num_rooms 3}\n```\n\n#### ByteStrings\n`ByteString`s are not wrapped, and returned raw in order to provide direct access to the byte array.\n\nHowever, ByteString's are naturally `seqable` since they implement `java.lang.Iterable`.\n\n#### Well-Known-Types\n\n[Well known types](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto) fields will be inlined into the message.\nThis means that rather than calling `(-\u003e my-proto-map :my-string-value :value)` you can simply write `(:my-string-value my-proto-map)`. Note that since\nwell-known-types are message types, this may return `nil` when the field is unset -- allowing us to model schemas which support null scalar fields.\n\n\n#### Encoders\n\nWhile protobuf allows us to describe our domain model, the Java generated types are not always a great programmatic fit. Consider the following schema:\n\n```protobuf\nmessage UUID {\n   int64 msb = 1; // most significat bits\n   int64 lsb = 2; // least significat bits\n}\n\nmessage Person {\n   UUID id = 1;\n}\n\n```\nReading a person's `id` field would return a ```{:lsb \u003clsb\u003e :msb \u003cmsb\u003e}``` proto-map.\n\nEncoders allow us to define an alternative type (rather than the POJO class) that will be used for proto-map fields of that type:\n```clj\n(defmapper mapper [protogen.generated.People$Person]\n  :encoders {protogen.generated.People$UUID\n             {:from-proto (fn [^protogen.generated.People$UUID proto-uuid]\n                            (java.util.UUID. (.getMsb proto-uuid) (.getLsb proto-uuid)))                           \n              :to-proto   (fn [^java.util.UUID java-uuid]\n                            (let [b (People$UUID/newBuilder)]\n                              (.setMsb b (.getMostSignificantBits java-uuid)\n                              (.setLsb b (.getLeastSignificantBits java-uuid))\n                              (.build b))))}})\n\n(proto-map mapper People$Person :id (java.util.UUID/randomUUID))\n=\u003e {:id #uuid \"2a1ef325-c7c2-42d4-815d-6bb1b9ed2e63\"} \n\n```\nThis encourages DRYer code, since these kinds of proto\u003c-\u003eclj conversions can be defined as a single encoder, rather than handled across the codebase.\n\n#### Interoping proto-maps with Java code\n\nIt is sometimes necessary to interop with Java code that expects a POJO instance. For example, consider the following method signature:\n\n\n```java\npublic class Utils {\n  public static void foo(com.google.protobuf.Duration duration) { ... }   \n}  \n```\n\nThis method receives a `com.google.protobuf.Duration`, a generated class that was compiled from the [duration schema](https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/duration.proto) that is part of the protobuf distribution. \n\nSince proto-maps are thin wrappers, we can always refer back to the underlying POJO and interop successfully:\n\n```clj\n(require '[pronto.core :as p])\n(import 'com.google.protobuf.Duration)\n\n(p/defmapper m [Duration])\n\n(Utils/foo (p/proto-map-\u003eproto (p/proto-map m Duration)))\n```\n\nIf your Java code operates on the protoc generated interfaces rather than concrete typs, it is also possible to pass the proto-map directly:\n\n```java\npublic static void foo(com.google.protobuf.DurationOrBuilder duration) { ... }\n```\n\n```clj\n(Utils/foo (p/proto-map m Duration))\n```\n\n## [Performance](doc/performance.md)\n\nPlease read the [performance introduction](doc/performance.md).\n\n## Schema utils\n\nTo inspect a schema at the REPL use `pronto.schema/schema`, which returns the (Clojurified) schema as data:\n\n```clj\n(require '[pronto.schema :refer [schema]])\n\n(schema People$Person)\n=\u003e {:diet #{\"UNKNOWN_DIET\" \"OMNIVORE\" \"VEGETARIAN\" \"VEGAN\"} ;; an enum\n    :address People$Address ;; address field\n    :address_book {String People$PersonDetails} ;; a map string-\u003ePersonDetails\n    :age  int\n    :friends [People$Person] ;; a repeated Person fields\n    :name String}\n```\nDrilling-down is also possible:\n```clj\n(p/schema People$Person :address)\n=\u003e {:country String :city String :house_num int}\n```\n\nPlease note that unlike the rest of the library, `schema` uses runtime reflection and is meant as a convenience method to be used during development. \n\n","funding_links":[],"categories":["Clojure","Protocol Buffers and gRPC"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAppsFlyer%2Fpronto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAppsFlyer%2Fpronto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAppsFlyer%2Fpronto/lists"}