{"id":13692864,"url":"https://github.com/damballa/abracad","last_synced_at":"2025-10-21T23:55:56.609Z","repository":{"id":8419780,"uuid":"10006304","full_name":"damballa/abracad","owner":"damballa","description":"A Clojure library for de/serializing Clojure data structures with Avro.","archived":false,"fork":false,"pushed_at":"2021-12-02T16:57:45.000Z","size":192,"stargazers_count":115,"open_issues_count":3,"forks_count":35,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-04-02T04:50:11.955Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/damballa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.apache-v2","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-05-11T22:45:55.000Z","updated_at":"2024-07-07T16:44:37.000Z","dependencies_parsed_at":"2022-08-27T03:52:05.287Z","dependency_job_id":null,"html_url":"https://github.com/damballa/abracad","commit_stats":null,"previous_names":[],"tags_count":19,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/damballa%2Fabracad","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/damballa%2Fabracad/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/damballa%2Fabracad/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/damballa%2Fabracad/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/damballa","download_url":"https://codeload.github.com/damballa/abracad/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252013625,"owners_count":21680388,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T17:01:02.916Z","updated_at":"2025-10-21T23:55:51.582Z","avatar_url":"https://github.com/damballa.png","language":"Clojure","funding_links":[],"categories":["Avro \u0026 Confluent Schema Registry","\u003ca name=\"Clojure\"\u003e\u003c/a\u003eClojure"],"sub_categories":[],"readme":"# abracad\n\n[![Build Status](https://secure.travis-ci.org/damballa/abracad.png)](http://travis-ci.org/damballa/abracad)\n\nAbracad is a Clojure library for de/serializing Clojure data\nstructures with Avro, leveraging the Java Avro implementation.\n\nAbracad supports: a generic mapping between Avro and Clojure data for\narbitrary schemas; customized protocol-based mappings between Avro\nrecords and any JVM types; and “schema-less” EDN-in-Avro serialization\nof arbitrary Clojure data.\n\n## Installation\n\nAbracad is available on Clojars.  Add this `:dependency` to your\nLeiningen `project.clj`:\n\n```clj\n[com.damballa/abracad \"0.4.13\"]\n```\n\n## Usage\n\nExample usage follows; [detailed API documentation][api] available,\ngenerated via [codox][codox].\n\n[api]: http://damballa.github.io/abracad/\n[codox]: https://github.com/weavejester/codox\n\n### Schemas\n\nAvro schemas may be parsed from JSON (from either strings or input\nstreams), from the Clojure data representation of a JSON schema, or\nfrom existing Avro Schema objects.\n\n```clj\n(require '[abracad.avro :as avro])\n\n(def schema\n  (avro/parse-schema\n   {:type :record\n    :name \"LongList\"\n    :aliases [\"LinkedLongs\"]\n    :fields [{:name \"value\", :type :long}\n             {:name \"next\", :type [\"LongList\", :null]}]}))\n```\n\nThe `parse-schema` function may be passed multiple schemas, in which\ncase later schemas may reference types defined in earlier schemas.\nThe result is the schema generated from the final argument.\n\n### Basic de/serialization\n\nAbracad provides functions which act as a thin layer over the Java\nAvro interface, plus Clojure generic datum reader and writer\nimplementations which allow Clojure data structures to be directly\nde/serialized.\n\n```clj\n(with-open [adf (avro/data-file-writer \"snappy\" schema \"example.avro\")]\n  (.append adf {:value 0, :next nil})\n  (.append adf {:value 8, :next {:value 16, :next nil}}))\n\n(with-open [adf (avro/data-file-reader \"example.avro\")]\n  (doall (seq adf)))\n;;=\u003e ({:value 0, :next nil} {:value 8, :next {:value 16, :next nil}})\n```\n\nThe Avro type deserialization mappings are as follows:\n\n  - Numeric primitives deserialize as their Java counterparts\n  - Strings currently always deserialize as Strings\n  - Enums deserialize as keywords\n  - Arrays currently always deserialize as persistent vectors\n  - Maps deserialize as persistent maps\n  - Fixed values currently always deserialize as primitive byte arrays\n  - Bytes values currently always deserialize as primitive byte arrays\n  - Records deserialize as maps with keyword field names and `:type` metadata\n    indicating the Avro schema name\n\nThe Avro specification allows field names to contain the `_` character but\ndisallows the `-` character.  Clojure keywords frequently contain `-` but rarely\ncontain `_`.  Abracad attempts to work around this difference by mapping `_` in\nAvro field names to `-` in Clojure symbols and vice-versa.  The current\nimplementation of this conversion does *not* handle keywords containing `_`\ninstead, which is probably a bug.  This mangling may be disabled by binding\n`abracad.avro.util/*mangle-names*` to `false`.\n\n#### Record de/serialization tweaking\n\nIn addition to the generic map de/serialization, records may also be\ngenerically de/serialized as vectors.  During serialization, whenever\na record is expected and a vector is encountered, the vector will be\nserialized by matching fields by position, so long as the expected and\nprovided numbers of fields match.\n\nDuring deserialization, a record schema with the annotation\n`:abracad.reader` set to `\"vector\"` will be deserialized as a vector,\nwith fields encoded by position.\n\n```clj\n(let [schema (avro/parse-schema\n              {:name \"example\", :type \"record\",\n               :fields [{:name \"left\", :type \"string\"}\n                        {:name \"right\", :type \"long\"}]\n               :abracad.reader \"vector\"})]\n  (-\u003e\u003e [\"foo\" 31337]\n       (avro/binary-encoded schema)\n       (avro/decode schema)))\n;;=\u003e [\"foo\" 31337\"]\n```\n\nMaps serialized as records will be checked to ensure that they do not\nhave any extra entries not encoded by the schema, raising an exception\nif extra entries are present.  This check may be avoided for\nindividual records by including `:type` metadata matching the schema.\nThe check may be en/disabled recursively for a record and all\ncontained records via the `:abracad.avro/unchecked` metadata.\n\n### Custom record de/serialization\n\nDuring union and record serialization, Abracad uses a protocol to\ndetermine an object's Avro schema name and to access its fields.\nDuring deserialization, Abracad uses a facility directly analogous to\nthe Clojure Reader `*data-readers*` facility to find custom\ndeserialization constructor functions.  These may be used to extend\nAvro de/serialization to arbitrary existing types.\n\n```clj\n(import 'java.net.InetAddress)\n\n(extend-type InetAddress\n  avro/AvroSerializable\n  (schema-name [_] \"ip.address\")\n  (field-get [this field] (case field :address (.getAddress this)))\n  (field-list [this] #{:address}))\n\n(defn -\u003eInetAddress\n  [address] (InetAddress/getByAddress address))\n\n(def schema\n  (avro/parse-schema\n   {:type :record\n    :name 'ip.address\n    :fields [{:name :address\n              :type [{:type :fixed, :name \"IPv4\", :size 4}\n                     {:type :fixed, :name \"IPv6\", :size 16}]}]}))\n\n(binding [avro/*avro-readers* {'ip/address #'-\u003eInetAddress}]\n  (with-open [adf (avro/data-file-writer schema \"example.avro\")]\n    (.append adf (InetAddress/getByName \"8.8.8.8\"))\n    (.append adf (InetAddress/getByName \"8::8\")))\n\n  (with-open [adf (avro/data-file-reader \"example.avro\")]\n    (doall (seq adf))))\n;;=\u003e (#\u003cInet4Address /8.8.8.8\u003e #\u003cInet6Address /8:0:0:0:0:0:0:8\u003e)\n```\n\n### EDN-in-Avro\n\nAbracad supports expressing EDN data structures as Avro records in the\n`abracad.avro.edn` Avro namespace.  The `new-schema` function in the\nsame-named Clojure namespace returns schemas which express a superset\nof EDN capturing most commonly-used Clojure constructs.  These allow\nusing Avro for Clojure data without pre-defining application-specific\nschemas.\n\n```clj\n(require '[abracad.avro.edn :as aedn])\n\n(def schema (aedn/new-schema))\n\n(-\u003e\u003e {:foo ['bar \"baz\" 1337]}\n     (avro/binary-encoded schema)\n     (avro/decode schema))\n;;=\u003e {:foo [bar \"baz\" 1337]}\n```\n\n### Hadoop MapReduce integration\n\nAvro 1.7.5 and later supports configurable “data models” for datum\nreading, writing, and comparison in Hadoop MapReduce jobs.  Abracad\n0.4.0 and later provides a `ClojureData` class which can be passed to\nthe `AvroJob/setDataModelClass` static method in order to map job Avro\ninput and output directly to and from Clojure data structures.\n\n## TODO\n\nThese are the early days.  Still to be done:\n\n  - Kick the tires on the interface.  There may be glaring holes.\n  - Write more exhaustive tests, to cover the full range of types.\n  - Figure out a cleaner way of handling `_` vs `-`.\n  - Dynamically generate schema-specific datum reader/writer\n    implementations.  All the speed of generating \u0026 compiling\n    de/serialization classes from schemas, but with none of the\n    ahead-of-time hassle.\n\n## License\n\nCopyright © 2013-2015 Damballa Inc. and contributors.\n\nDistributed under your choice of the Eclipse Public License or the\nApache License, Version 2.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamballa%2Fabracad","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdamballa%2Fabracad","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdamballa%2Fabracad/lists"}