{"id":16405558,"url":"https://github.com/igrishaev/deed","last_synced_at":"2025-04-09T22:11:43.459Z","repository":{"id":255689698,"uuid":"851725823","full_name":"igrishaev/deed","owner":"igrishaev","description":"Fast, flexible, 0-deps (de)serialization library for Clojure","archived":false,"fork":false,"pushed_at":"2024-12-07T12:31:22.000Z","size":309,"stargazers_count":55,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-09T22:11:38.561Z","etag":null,"topics":["clojure","serialization"],"latest_commit_sha":null,"homepage":"https://github.com/igrishaev/deed","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/igrishaev.png","metadata":{"files":{"readme":"readme.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-03T15:54:35.000Z","updated_at":"2025-04-01T12:07:24.000Z","dependencies_parsed_at":"2024-09-06T18:48:50.829Z","dependency_job_id":"7a8f2236-15a2-44e9-8e5a-e1b79759b26e","html_url":"https://github.com/igrishaev/deed","commit_stats":{"total_commits":155,"total_committers":1,"mean_commits":155.0,"dds":0.0,"last_synced_commit":"bde65422da04e14edd5b2278999e5159783a54b2"},"previous_names":["igrishaev/pinny","igrishaev/deed"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrishaev%2Fdeed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrishaev%2Fdeed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrishaev%2Fdeed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/igrishaev%2Fdeed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/igrishaev","download_url":"https://codeload.github.com/igrishaev/deed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248119294,"owners_count":21050755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","serialization"],"created_at":"2024-10-11T06:06:19.147Z","updated_at":"2025-04-09T22:11:43.436Z","avatar_url":"https://github.com/igrishaev.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Deed\n\nA fast, zero-deps binary encoding and decoding library for Clojure.\n\n## Table of Contents\n\n\u003c!-- toc --\u003e\n\n- [About](#about)\n- [Motivation](#motivation)\n- [Installation \u0026 Requirements](#installation--requirements)\n- [Quick Demo](#quick-demo)\n- [API](#api)\n * [Simple Encode and Decode](#simple-encode-and-decode)\n * [Encoding to Memory](#encoding-to-memory)\n * [Sequential Encoding and Decoding](#sequential-encoding-and-decoding)\n * [Low-Level API](#low-level-api)\n * [API Options](#api-options)\n- [GZipped Streams](#gzipped-streams)\n- [Versioning and Backward Compatibility](#versioning-and-backward-compatibility)\n- [Appending to a File](#appending-to-a-file)\n- [Handle Unsupported Types](#handle-unsupported-types)\n- [Supported Types](#supported-types)\n- [Extending Custom Types](#extending-custom-types)\n * [Encode](#encode)\n * [Decode](#decode)\n * [Macros](#macros)\n * [Grouping Fields](#grouping-fields)\n- [Handling Defrecords](#handling-defrecords)\n- [Contrib](#contrib)\n * [Base64](#base64)\n * [VectorZ](#vectorz)\n- [Binary Format](#binary-format)\n- [Benchmarks](#benchmarks)\n\n\u003c!-- tocstop --\u003e\n\n## About\n\n[vectorz]: https://github.com/mikera/vectorz\n\nDeed is a library to dump any value into a byte array and read it back. It\nsupports plenty of types out from the box: Java primitives, most of the Clojure\ntypes, Java collections, date and time, and so on. It supports even such tricky\ntypes as atoms, refs, and input streams. The full list of supported types is\nshown in the [\"Supported Types\"](#supported-types) section below.\n\nDeed can be extended with custom types with ease. There is a contrib package\nthat extends encoding and decoding logic for vectors from the the well-known\n[mikera/vectorz][vectorz] library.\n\nDeed is written in pure Java and thus is pretty fast (see the\n[\"Benchmarks\"](#benchmarks) section). It's about 30-50% faster than Nippy.\n\nIt doesn't rely on the built-in Java `Serializable` interface for security\nreasons. Every type is processed manually.\n\nDeed provides convenient API for reading the frozen data lazily one by one.\n\n## Motivation\n\nObviously you would ask why doing this if we already have Nippy? This is what I\nhad in mind while working on Deed:\n\n1. The library must be **absolutely free** from dependencies. This is true for\n the `deed-core` package: it's written in pure Java with no dependencies at\n all. By adding it into a project, you won't blow up you uberjar, nor you will\n have troubles with building a native image with GraalVM.\n\n2. Any part of Deed that requires 3rd-party stuff must be a sub-library. So you\n have precise control of what you use and what you don't\n\n3. Unlike Nippy, Deed never falls back to native Java serialization. There is no\n such an option. Thus, your application cannot be attacked by someone who has\n forged a binary dump.\n\n4. Deed is simple: it blindly works with input- and output byte streams having\n no idea what's behind them. It doesn't take compression nor encryption into\n account -- yet there are utilities for streams.\n\n5. The library provides API which personally I consider more convenient than\n Nippy's. Namely, Deed can lazily iterate on a series of encoded data instead\n of reading the whole dump at once.\n\n6. Finally, why not using popular and cross-platform formats like JSON, Message\n Pack, or YAML? Well, because of poor types support. JSON has only primitive\n types and collections, and nothing else. Extending it with custom types is\n always a pain. At the same time, I want my decoded data be as close to the\n origin data as possible, say, `LocalDateTime` be an instance of\n `LocalDateTime` but not a string or `java.util.Date`. Sometimes, preserving\n metadata is crucial. To handle all of these cases, there now a way other than\n making your own library.\n\n## Installation \u0026 Requirements\n\nDeed requires Java version at least 16 to run. Tested with Clojure 1.9.0.\n\n**The core module** with basic encode and decode capabilities:\n\n~~~clojure\n;; lein\n[com.github.igrishaev/deed-core \"0.1.0\"]\n\n;; deps\ncom.github.igrishaev/deed-core {:mvn/version \"0.1.0\"}\n~~~\n\n**Base64 module** do encode and decode from/into base64 on the fly:\n\n~~~clojure\n;; lein\n[com.github.igrishaev/deed-base64 \"0.1.0\"]\n\n;; deps\ncom.github.igrishaev/deed-base64 {:mvn/version \"0.1.0\"}\n~~~\n\n**Vectorz module** extends Deed with a number of `Vector*` classes from the\n[mikera/vectorz][vectorz] library:\n\n~~~clojure\n;; lein\n[com.github.igrishaev/deed-vectorz \"0.1.0\"]\n\n;; deps\ncom.github.igrishaev/deed-vectorz {:mvn/version \"0.1.0\"}\n~~~\n\n## Quick Demo\n\nThis is a very brief example of how to dump the data into a file. Prepare the\nnamespace with imports:\n\n~~~clojure\n(ns demo\n (:require\n [clojure.java.io :as io]\n [deed.core :as deed]))\n~~~\n\nDeclare the file and the data:\n\n~~~clojure\n(def file\n (io/file \"dump.deed\"))\n\n(def data\n {:number 1\n :string \"hello\"\n :bool true\n :nil nil\n :symbol 'hello/test\n :simple-kw :test\n :complex-kw :foo/bar\n :vector [1 2 :test nil 42 \"hello\"]\n :map {:test {:bar {'dunno {\"lol\" [:kek]}}}}\n :set #{:a :b :c}\n :atom (atom {:abc 42})})\n~~~\n\nPass them into the `encode-to` function as follows:\n\n~~~clojure\n(deed/encode-to data file)\n~~~\n\nAnd this is it! If you examine the file with any kind of a hex editor, you'll\nsee a binary payload:\n\n~~~clojure\nxxd /path/to/dump.deed\n\n00000000: 0001 0001 0000 0000 0000 0000 0000 0000 ................\n00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................\n00000020: 0000 003b 0000 000b 004a 0000 0006 6e75 ...;.....J....nu\n00000030: 6d62 6572 000e 004a 0000 0006 7379 6d62 mber...J....symb\n00000040: 6f6c 004b 0000 000a 6865 6c6c 6f2f 7465 ol.K....hello/te\n00000050: 7374 004a 0000 000a 636f 6d70 6c65 782d st.J....complex-\n00000060: 6b77 004a 0000 0007 666f 6f2f 6261 7200 kw.J....foo/bar.\n00000070: 4a00 0000 0673 7472 696e 6700 2b00 0000 J....string.+...\n00000080: 0568 656c 6c6f 004a 0000 0006 7665 6374 .hello.J....vect\n00000090: 6f72 002e 0000 0006 000e 000c 0000 0000 or..............\n000000a0: 0000 0002 004a 0000 0004 7465 7374 0000 .....J....test..\n000000b0: 000c 0000 0000 0000 002a 002b 0000 0005 .........*.+....\n000000c0: 6865 6c6c 6f00 4a00 0000 036e 696c 0000 hello.J....nil..\n000000d0: 004a 0000 0004 626f 6f6c 0029 004a 0000 .J....bool.).J..\n000000e0: 0003 7365 7400 3300 0000 0300 4a00 0000 ..set.3.....J...\n000000f0: 0163 004a 0000 0001 6200 4a00 0000 0161 .c.J....b.J....a\n00000100: 004a 0000 0009 7369 6d70 6c65 2d6b 7700 .J....simple-kw.\n00000110: 4a00 0000 0474 6573 7400 4a00 0000 0461 J....test.J....a\n00000120: 746f 6d00 3000 3b00 0000 0100 4a00 0000 tom.0.;.....J...\n00000130: 0361 6263 000c 0000 0000 0000 002a 004a .abc.........*.J\n00000140: 0000 0003 6d61 7000 3b00 0000 0100 4a00 ....map.;.....J.\n00000150: 0000 0474 6573 7400 3b00 0000 0100 4a00 ...test.;.....J.\n00000160: 0000 0362 6172 003b 0000 0001 004b 0000 ...bar.;.....K..\n00000170: 0005 6475 6e6e 6f00 3b00 0000 0100 2b00 ..dunno.;.....+.\n00000180: 0000 036c 6f6c 002e 0000 0001 004a 0000 ...lol.......J..\n00000190: 0003 6b65 6b ..kek\n~~~\n\nNow that you have a .deed file, read it back using the `decode-from` function:\n\n~~~clojure\n(def data-back\n (deed/decode-from file))\n~~~\n\nPrint the `data-back` to ensure it keeps the origin types:\n\n~~~clojure\n{:number 1,\n :symbol hello/test,\n :complex-kw :foo/bar,\n :string \"hello\",\n :vector [1 2 :test nil 42 \"hello\"],\n :nil nil,\n :bool true,\n :set #{:c :b :a},\n :simple-kw :test,\n :atom #\u003cAtom@75f6dd87: {:abc 42}\u003e,\n :map {:test {:bar {dunno {\"lol\" [:kek]}}}}}\n~~~\n\nCompare them to ensure we didn't loose anything. Since atoms cannot be compared,\nwe `dissoc` them from both sides:\n\n~~~clojure\n(= (dissoc data :atom) (dissoc data-back :atom))\n;; true\n~~~\n\nDeed supports plenty of built-in Clojure and Java types. The next section\ndescribes which API it provides to manage the data.\n\n## API\n\n### Simple Encode and Decode\n\nThe `encode-to` function accepts two parameters: a value and an output. The\nvalue is an instance of any [supported type](#supported-types) (a map, a vector,\netc).\n\nThe output is anything that can be coerced to the output stream using the\n`io/output-stream` function. It could be a file, another output stream, a byte\narray, or similar. If you pass a string, it's treated as a file name which will\nbe created.\n\nExamples:\n\n~~~clojure\n(deed/encode-to {:foo 123} \"test.deed\")\n\n(deed/encode-to {:foo 123} (io/file \"test2.deed\"))\n\n(deed/encode-to {:foo 123} (-\u003e \"test3.deed\"\n io/file\n io/output-stream))\n~~~\n\nAll the three invocations above dump the same map `{:foo 123}` into different\nfiles.\n\nTo read the data back, invoke the `decode-from` function. It accepts anything\nthat can be coerced into an input stream using the `io/input-stream`\nfunction. It might be a file, another stream, a byte array, or a name of a file.\n\n~~~clojure\n(deed/decode-from \"test.deed\")\n;; {:foo 123}\n\n(deed/decode-from (io/file \"test2.deed\"))\n;; {:foo 123}\n\n(deed/decode-from (-\u003e \"test3.deed\"\n io/file\n io/input-stream))\n;; {:foo 123}\n~~~\n\n### Encoding to Memory\n\nThe functions below rely on external IO resources. If you want to dump the data\ninto memory, use the `encode-to-bytes` function. It returns a byte array without\nany IO interaction:\n\n~~~clojure\n(def buf\n (deed/encode-to-bytes {:test 123}))\n\n(println (vec buf))\n\n[0 1 0 1 0 0 ... 59 0 0 0 1 0 74 0 0 0 4 116 101 115 116 0 12 0 0 0 0 0 0 0 123]\n~~~\n\nThere is no an opposite `decode-from-bytes` function because a byte array is\nalready a data source for the `decode-from` function. Just pass the result into\nit:\n\n~~~clojure\n(deed/decode-from buf)\n;; {:test 123}\n~~~\n\n### Sequential Encoding and Decoding\n\nWe often dump vast collections to explore them afterwards. Say, you're going to\nwrite 10M of database rows into a file to find a broken row with a script.\n\nThe functions above encode and decode a single value. That's OK for primitive\ntypes and maps but not good for vast collections. For example, if you encode a\nvector of 10M items into a file and read it back, you'll get the same vector of\n10M items. That's unlikely you need all of these at once: you'd better to\niterate on them one by one.\n\nThis is the case that `encode-seq-to` and `decode-seq-from` functions cover. The\nfirst function accepts a collection of items and writes them sequentially. It's\nnot a vector any longer but a series of items written one after another. The\n`encode-seq-to` invocation returns the number of items written:\n\n~~~clojure\n(deed/encode-seq-to [1 2 3] \"test.deed\")\n;; 3\n~~~\n\nInstead of a vector, there might be a lazy sequence, or anything that can be\niterated.\n\nIf you read the dump using `decode-from`, you'll get the first item only:\n\n~~~clojure\n(deed/decode-from \"test.deed\")\n1\n~~~\n\nTo read all of them, use `decode-seq-from`:\n\n~~~clojure\n(deed/decode-seq-from \"test.deed\")\n[1 2 3]\n~~~\n\nWhat is the point to use sequential encoding? Because with a special API, you\ncan read them lazily one by one.\n\nThe `with-decoder` macro takes two parameters: the binding symbol and the\ninput. Internally, it creates a `Decoder` instance and binds it to the first\nsymbol. The `decode-seq` function returns a lazy sequence of items from a\ndecoder:\n\n~~~clojure\n(deed/with-decoder [d \"test.deed\"]\n (doseq [item (deed/decode-seq d)]\n (println item)))\n;; 1\n;; 2\n;; 3\n~~~\n\nYou must process items before you exit the `with-decoder` macro.\n\nThe form above might be rewritten using the `with-open` macro. Since the\n`Decoder` object implements `AutoCloseable` interface, it handles the `.close`\nmethod which in turn closes the underlying stream.\n\n~~~clojure\n(with-open [d (deed/decoder \"test.deed\")]\n (doseq [item (deed/decode-seq d)]\n (println item)))\n~~~\n\nIn fact, you don't even need to pass the decoder object into `decode-seq`\nbecause it implements the `Iterable` Java interface. Just iterate the decoder:\n\n~~~clojure\n(deed/with-decoder [d \"test.deed\"]\n (mapv inc d))\n;; [2 3 3]\n\n(deed/with-decoder [d \"test.deed\"]\n (doseq [item d]\n (println \"item is\" item)))\n;; item is 1\n;; item is 2\n;; item is 3\n~~~\n\nThe items are read lazily so you won't saturate memory.\n\n### Low-Level API\n\nDeed provides low-level API for conditional or imperative encoding. The `encode`\nfunction writes a value into an instance of the `Encoder` class. You can use it\nin a cycle with some condition:\n\n~~~clojure\n(deed/with-encoder [e \"test.deed\"]\n (doseq [x (range 1 32)]\n (when (even? x)\n (deed/encode e x))))\n~~~\n\nThe `decode` function reads an object from the `Decoder` instance. When the end\nof the stream is met, you'll get a special `EOF` object that you can check using\nthe `eof?` preficate:\n\n~~~clojure\n(deed/with-decoder [d \"test.deed\"]\n (loop [i 0]\n (let [item (deed/decode d)]\n (if (deed/eof? item)\n (println \"EOF\")\n (do\n (println \"item\" i item)\n (recur (inc i)))))))\n\n;; item 0 2\n;; item 1 4\n;; item 2 6\n;; item 3 8\n;; item 4 10\n;; item 5 12\n;; item 6 14\n;; item 7 16\n;; item 8 18\n;; item 9 20\n;; item 10 22\n;; item 11 24\n;; item 12 26\n;; item 13 28\n;; item 14 30\n;; EOF\n~~~\n\nThe low-level API is useful for precise control on encoding and decoding.\n\n### API Options\n\nMost of the functions accept an optional map of parameters. Here is a list of\noptions supported at the moment:\n\n| Name | Default | Meaning |\n|--------------------------|-------------------|---------------------------------------------------------------------------------------------------------------------------|\n| `:deref-timeout-ms` | 5000 | The number of milliseconds to wait when derefing futures. |\n| `:object-chunk-size` | 0xFF | The number of object chunk when encoding uncountable collections (e.g. lazy seqs). |\n| `:byte-chunk-size` | 0xFFFF | The number of byte chunk when encoding input streams. |\n| `:uncountable-max-items` | Integer.MAX_VALUE | The max number of items to process when encoding uncountable collections (e.g. lazy seqs). |\n| `:encode-unsupported?` | true | If true, dump every unsupported object into a string ([see below](#handle-unsupported-types)). Otherwise, throw an error. |\n| `:io-temp-file?` | false | When deciding previously encoded input stream, write its payload into a temp file. |\n| `:save-meta?` | true | Preserve metadata for objects what have it. |\n| `:append?` | false | Write at the end of an existing dump ([see below](#appending-to-a-file)). |\n\n\nThat's unlikely you'll need to change any of these, yet in rare cases they might\nhelp.\n\n## GZipped Streams\n\nDeed has a couple of functions that turn any input or output into\n`GZIPInputStream` and `GZIPOutputStream` instances respectfully. It allows to\ncompress the data on the fly. Here is how you compress:\n\n~~~clojure\n(with-open [out (deed/gzip-output-stream \"dump.deed.gz\")]\n (deed/encode-to [1 2 3] out))\n~~~\n\nAnd decompress:\n\n~~~clojure\n(with-open [in (deed/gzip-input-stream \"dump.deed.gz\")]\n (deed/decode-from in))\n;; [1 2 3]\n~~~\n\nKeep in mind that compression saves disk space but consumes CPU usage.\n\n## Versioning and Backward Compatibility\n\nDeed has a built-in versioning system. Every time you encode something, the\nlibrary emits a leading `Header` object with a version of the protocol. At the\nmoment of writing this, the version is just 1. When decoding, this version\nnumber is read from the header before parsing any other objects.\n\nIf any breaking changes appear in encode/decode logic, two things will take\nplace in the next release:\n\n- the constant `HEADER_VERSION` will be bumped from 1 to 2 so any further\n encoding will have protocol version 2 as well;\n\n- a corresponding encode/decode logic will be wrapped into a `switch...case`\n branch depending on the current version of the protocol.\n\nAt that very moment, Deed has protocol version 1 with no branching, yet there is\na room for extending. Adding new OIDs and types won't change the protocol\nversion.\n\n## Appending to a File\n\nIn rare cases, you'd like to append data to an existing file. This might be done\nin two steps. First, you initiate the `FileOutputStream` object manually and\npass the `true` boolean flag meaning the data is put at the end of a file:\n\n~~~clojure\n(with-open [out (new FileOutputStream file true)]\n ...)\n~~~\n\nSecond, when encoding the data into such an output, specify the `{:append?\ntrue}` option. In this case, Deed won't emit a leading `deed.Header` object:\n\n~~~clojure\n(with-open [out (new FileOutputStream file true)]\n (deed/encode-to {:hello 123} out {:append? true}))\n~~~\n\n## Handle Unsupported Types\n\nBy default, when Deed doesn't know how to encode an object, it turns it into a\nstring using the standard `.toString` method. Than it makes a special\n`Unsupported` object that tracks full class name and the text payload. Let's\npresent it with a custom `deftype` declaration:\n\n~~~clojure\n(deftype MyType [a b c])\n\n(def mt (new MyType :hello \"test\" 42))\n\n(deed/encode-to mt \"test.deed\")\n\n(deed/decode-from \"test.deed\")\n;; #\u003cUnsupported@b918edf: {:content \"demo.MyType@4376ae5c\", :class \"demo.MyType\"}\u003e\n~~~\n\nThe `Unsupported` object can be checked with the `unsupported?` predicate:\n\n~~~clojure\n(def mt-back\n (deed/decode-from \"test.deed\"))\n\n(deed/unsupported? mt-back)\n;; true\n~~~\n\nTo coerce it to Clojure, just `deref` it:\n\n~~~clojure\n@mt-back\n{:content \"demo.MyType@4376ae5c\", :class \"demo.MyType\"}\n~~~\n\nAbove, the `\"demo.MyType@4376ae5c\"` string doesn't say much. This is because the\ndefault `.toString` implementation of `deftype` lacks fields. That's why it's\nalways worth overriding the `.toString` method for custom types:\n\n~~~clojure\n(deftype MyType [a b c]\n Object\n (toString [_]\n (format \"\u003cMyType: %s, %s, %s\u003e\" a b c)))\n\n(def mt (new MyType :hello \"test\" 42))\n\n(deed/encode-to mt \"test.deed\")\n\n(def mt-back\n (deed/decode-from \"test.deed\"))\n\n(str mt-back)\n;; \"Unsupported[className=demo.MyType, content=\u003cMyType: :hello, test, 42\u003e]\"\n\n@mt-back\n;; {:content \"\u003cMyType: :hello, test, 42\u003e\", :class \"demo.MyType\"}\n~~~\n\nNow the content has fields, so at least you can observe them.\n\nWhen the `:encode-unsupported?` boolean option is false, Deed throws an\nexception by facing an unsupported object:\n\n~~~clojure\n(deed/encode-to mt \"test.deed\" {:encode-unsupported? false})\n\n;; Execution error at deed.Err/error (Err.java:14).\n;; Cannot encode object, type: demo.MyType, object: \u003cMyType: :hello, test, 42\u003e\n~~~\n\n## Supported Types\n\nThe table below renders types supported by Deed out from the box:\n\n| OID | TAG | Class | Comment |\n|--------|----------------------|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| 0x0000 | NULL | `null` (`nil`) | |\n| 0x0001 | HEADER | `deed.Header` | A leading object with general info about encoding |\n| 0x0002 | UNSUPPORTED | `deed.Unsupported` | A wrapper for unsupported objects |\n| 0x0003 | META | | Specifies an object with metadata |\n| 0x0004 | INT | `int`, `java.lang.Integer` | |\n| 0x0005 | INT_ZERO | | A special OID for 0 int |\n| 0x0006 | INT_ONE | | A special OID for 1 int |\n| 0x0007 | INT_MINUS_ONE | | A special OID for -1 int |\n| 0x0008 | SHORT | `short`, `java.lang.Short` | |\n| 0x0009 | SHORT_ZERO | | A special OID for 0 short |\n| 0x000A | SHORT_ONE | | A special OID for 1 short |\n| 0x000B | SHORT_MINUS_ONE | | A special OID for -1 short |\n| 0x000C | LONG | `long`, `java.lang.Long` | |\n| 0x000D | LONG_ZERO | | |\n| 0x000E | LONG_ONE | | |\n| 0x000F | LONG_MINUS_ONE | | |\n| 0x0010 | IO_INPUT_STREAM | `java.io.InputStream` | When decoding, the bytes are put into a `ByteArrayInputStream`. It's also possible to put them into a temp file and obtain a `FileInputStream` |\n| 0x0011 | IO_READER | - | Not implemented |\n| 0x0012 | IO_FILE | - | Not implemented |\n| 0x0013 | IO_BYTEBUFFER | `java.nio.ByteBuffer` | |\n| 0x0014 | ARR_BYTE | `byte[]` | |\n| 0x0015 | ARR_INT | `int[]` | |\n| 0x0016 | ARR_SHORT | `short[]` | |\n| 0x0017 | ARR_BOOL | `boolean[]` | |\n| 0x0018 | ARR_FLOAT | `float[]` | |\n| 0x0019 | ARR_DOUBLE | `double[]` | |\n| 0x001A | ARR_OBJ | `Object[]` | |\n| 0x001B | ARR_LONG | `long[]` | |\n| 0x001C | ARR_CHAR | `char[]` | |\n| 0x001D | REGEX | `java.util.regex.Pattern` | |\n| 0x001E | CLJ_SORTED_SET | `clojure.lang.PersistentTreeSet` | A sorted set usually created with `(sorted-set ...)` |\n| 0x001F | CLJ_SORTED_SET_EMPTY | | A special OID for an empty sorted set |\n| 0x0020 | CLJ_SORTED_MAP | `clojure.lang.PersistentTreeMap` | A sorted map usually created with `(sorted-map ...)` |\n| 0x0021 | CLJ_SORTED_MAP_EMPTY | | An empty sorted map |\n| 0x0022 | URI | `java.net.URI` | |\n| 0x0023 | URL | `java.net.URL` | |\n| 0x0024 | EXCEPTION | `java.lang.Exception` | Keeps message, class name, stack trace, cause (recursively encoded), and all the suppressed exceptions |\n| 0x0025 | IO_EXCEPTION | | |\n| 0x0026 | THROWABLE | | |\n| 0x0027 | EX_INFO | | |\n| 0x0028 | EX_NPE | | |\n| 0x0029 | BOOL_TRUE | `boolean`, `java.lang.Boolean` | True value only |\n| 0x002A | BOOL_FALSE | `boolean`, `java.lang.Boolean` | False value only |\n| 0x002B | STRING | `java.lang.String` | Stored as a number of bytes + bytes |\n| 0x002C | STRING_EMPTY | | A special OID indicating an empty string |\n| 0x002D | CHAR | `char`, `java.lang.Character` | |\n| 0x002E | CLJ_VEC | `clojure.lang.APersistentVector` | A standard Clojure vector |\n| 0x002F | CLJ_VEC_EMPTY | | A special OID indicating an empty vector |\n| 0x0030 | CLJ_ATOM | `clojure.lang.Atom` | Gets deref-ed when encoding |\n| 0x0031 | CLJ_REF | `clojure.lang.Ref` | Gets deref-ed when encoding |\n| 0x0032 | FUTURE | `java.util.concurrent.Future` | Deed `.get`s the value using timeout from options. When time is up, an exception is throw. When decoded, it's returned as an instance of `deed.FutureWrapper`: a fake object that mimics a future. |\n| 0x0033 | CLJ_SET | `clojure.lang.APersistentSet` | A standard Clojure immutable set |\n| 0x0034 | CLJ_SET_EMPTY | | A special OID indicating an empty set |\n| 0x0035 | CLJ_LAZY_SEQ | `clojure.lang.LazySeq` | Encode a lazy sequence produced with `map`, `for`, etc. Stored as a collection of chunks like `\u003cchunk-len\u003e\u003citems...\u003e`. When decoding, read until the chunk of zero length is met. |\n| 0x0036 | CLJ_SEQ | Type depends on the origin collection | Becomes a vector when decoding |\n| 0x0037 | CLJ_LIST | `clojure.lang.PersistentList` | |\n| 0x0038 | CLJ_LIST_EMPTY | | A special OID indicating an empty list |\n| 0x0039 | CLJ_QUEUE | `clojure.lang.PersistentQueue` | |\n| 0x003A | CLJ_QUEUE_EMPTY | | A special OID indicating an empty queue |\n| 0x003B | CLJ_MAP | `clojure.lang.APersistentMap` | |\n| 0x003C | CLJ_MAP_EMPTY | `clojure.lang.APersistentMap` | An empty Clojure map |\n| 0x003D | CLJ_MAP_ENTRY | `clojure.lang.MapEntry` | A pair of key and value |\n| 0x003E | CLJ_RECORD | `clojure.lang.IRecord` | An instance of `defrecord` object. When decoding, becomes an ordinary map. To preserve the origin type, use the `handle-record` macro (see below). |\n| 0x003F | CLJ_TR_VEC | `c.l.PersistentVector$TransientVector` | A transient Clojure vector. |\n| 0x0040 | JVM_MAP | `java.util.Map` | A Java map, usually an instance of `HashMap`. |\n| 0x0041 | JVM_MAP_ENTRY | `java.util.Map$Entry` | |\n| 0x0042 | UUID | `java.util.UUID` | |\n| 0x0043 | JVM_LIST | `java.util.List` | When decoding, becomes an instance of `ArrayList`. |\n| 0x0044 | JVM_LIST_EMPTY | `java.util.List` | A stub for an empty list. |\n| 0x0045 | JVM_VECTOR | `java.util.Vector` | |\n| 0x0046 | JVM_VECTOR_EMPTY | | An empty Java vector. |\n| 0x0047 | JVM_ITERABLE | `java.lang.Iterable` | Encoded as uncounted chunked sequence of objects |\n| 0x0048 | JVM_ITERATOR | | When decoding, becomes an instance of `ArrayList`. |\n| 0x0049 | JVM_STREAM | `java.util.stream.Stream` | |\n| 0x004A | CLJ_KEYWORD | `clojure.lang.Keyword` | |\n| 0x004B | CLJ_SYMBOL | `clojure.lang.Symbol` | |\n| 0x004C | UTIL_DATE | `java.util.Date` | |\n| 0x004D | DT_LOCAL_DATE | `java.time.LocalDate` | |\n| 0x004E | DT_LOCAL_TIME | `java.time.LocalTime` | |\n| 0x004F | DT_LOCAL_DATETIME | `java.time.LocalDateTime` | |\n| 0x0050 | DT_OFFSET_DATETIME | `java.time.OffsetDateTime` | |\n| 0x0051 | DT_OFFSET_TIME | `java.time.OffsetTime` | |\n| 0x0052 | DT_DURATION | `java.time.Duration` | |\n| 0x0053 | DT_PERIOD | `java.time.Period` | |\n| 0x0054 | DT_ZONED_DATETIME | `java.time.ZonedDateTime` | |\n| 0x0055 | DT_ZONE_ID | `java.time.ZoneId` | |\n| 0x0056 | DT_INSTANT | `java.time.Instant` | |\n| 0x0057 | SQL_TIMESTAMP | `java.sql.Timestamp` | |\n| 0x0058 | SQL_TIME | `java.sql.Time` | |\n| 0x0059 | SQL_DATE | `java.sql.Date` | |\n| 0x005A | BYTE | `byte`, `java.lang.Byte` | |\n| 0x005B | BYTE_ZERO | | A stub for byte 0 |\n| 0x005C | BYTE_ONE | | A stub for byte 1 |\n| 0x005D | BYTE_MINUS_ONE | | A stub for byte -1 |\n| 0x005E | FLOAT | `float`, `java.lang.Float` | |\n| 0x005F | FLOAT_ZERO | | A stub for float 0 |\n| 0x0060 | FLOAT_ONE | | A stub for float 1 |\n| 0x0061 | FLOAT_MINUS_ONE | | A stub for float -1 |\n| 0x0062 | DOUBLE | `double`, `java.lang.Double` | |\n| 0x0063 | DOUBLE_ZERO | | A stub for double 0 |\n| 0x0064 | DOUBLE_ONE | | A stub for double 1 |\n| 0x0065 | DOUBLE_MINUS_ONE | | A stub for double -1 |\n| 0x0066 | JVM_BIG_DEC | `java.math.BigDecimal` | |\n| 0x0067 | JVM_BIG_INT | `java.math.BigInteger` | |\n| 0x0068 | CLJ_BIG_INT | `clojure.lang.BigInt` | |\n| 0x0069 | CLJ_RATIO | `clojure.lang.Ratio` | |\n| 0x006A | VECTORZ_AVECTOR | `mikera.vectorz.AVector` | See the `deed-vectorz` package |\n\n## Extending Custom Types\n\n### Encode\n\nAlthough Deed handles plenty of types, you can easily have a type that is\nunknown to it. Handling such a type is a matter of two things: extending both\nencoding and decoding logic.\n\nImagine you have a custom `deftype` with three fields:\n\n~~~clojure\n(deftype SomeType [x y z]\n ...)\n~~~\n\n[oids]: deed-core/src/java/deed/OID.java\n\nHere is how you extend the encoding logic. First, declare a custom OID number\nfor it. The number must be `short` (two bytes) meaning the range from -32768\nto 32767. When declaring such an OID, please ensure it doens't overlap with\n[pre-existing OIDs][oids].\n\n~~~clojure\n(def SomeTypeOID 4321)\n~~~\n\nThen extend the `IEncode` protocol with that type:\n\n~~~clojure\n(extend-protocol deed/IEncode\n SomeType\n (-encode [this encoder]\n (deed/writeOID encoder SomeTypeOID) ;; !\n (deed/encode encoder (.-x this))\n (deed/encode encoder (.-y this))\n (deed/encode encoder (.-z this))))\n~~~\n\nOr vice versa: extend the type with the protocol:\n\n~~~clojure\n(extend-type SomeType\n deed/IEncode\n (-encode [this encoder]\n (deed/writeOID encoder SomeTypeOID) ;; !\n (deed/encode encoder (.-x this))\n (deed/encode encoder (.-y this))\n (deed/encode encoder (.-z this))))\n~~~\n\nPay attention that in both cases the first expression must be the `writeOID`\ninvocation. The OID is always put first because decoding logic relies on\nit. Then we encode custom fields `x`, `y`, and `z`.\n\nEncoding might be a bit easier if you extend a type with `IEncode` when\ndeclaring it. In this case, the `-encode` method has direct access to `x`, `y`,\nand `z` as they were local variables.\n\n~~~clojure\n(deftype SomeType [x y z]\n deed/IEncode\n (-encode [this encoder]\n (deed/writeOID encoder SomeTypeOID)\n (deed/encode encoder x)\n (deed/encode encoder y)\n (deed/encode encoder z)))\n~~~\n\nSince `encode` is a general function, it handles any types. You don't bother if\n`x` is a number, or a string, or a keyword, or a nested `SomeType` instance.\n\n### Decode\n\nExtend the decoding counterpart by adding implementation to the `-decode`\nmultimethod. The branching value is the OID you declared:\n\n~~~clojure\n(defmethod deed/-decode SomeTypeOID\n [_ decoder]\n (let [x (deed/decode decoder)\n y (deed/decode decoder)\n z (deed/decode decoder)]\n (new SomeType x y z)))\n~~~\n\nHere, we retrieve `x`, `y`, and `z` fields back and componse a new instance of\n`SomeType`. Let's check it quckly:\n\n~~~clojure\n(def _buf\n (deed/encode-to-bytes (new SomeType 1 2 3)))\n\n(deed/decode-from _buf)\n;; #object[demo.SomeType 0x47e19f78 \"demo.SomeType@47e19f78\"]\n~~~\n\nThe order of writing and reading fields does matter, of course. If you mix them,\nyou'll get a broken object back.\n\n### Macros\n\nThere is a couple of macros that extend protocols and multimethods under the\nhood: `expand-encode` and `expand-decode`. The first one accepts an OID, a type\n(class), and a couple of symbols to bind: the current `Encoder` instance and the\ncurrent value you process. Then there is a body that encodes fields:\n\n~~~clojure\n(deed/expand-encode [MyOID SomeType encoder some-type]\n (deed/encode encoder (.-x some-type))\n (deed/encode encoder (.-y some-type))\n (deed/encode encoder (.-z some-type)))\n~~~\n\nPay attention, you **don't call** `writeOID` inside the body because it's\nalready a part of the macro.\n\nThe `expand-decode` macro accepts an OID and a symbol bound to the current\n`Decoder` instance. Inside, you decode fields and compose an object:\n\n~~~clojure\n(deed/expand-decode [MyOID decoder]\n (let [x (deed/decode decoder)\n y (deed/decode decoder)\n z (deed/decode decoder)]\n (new AnotherType x y z)))\n~~~\n\n### Grouping Fields\n\nAbove, we encoded and decoded fields `x`, `y` and `z` one by one. This is ok for\na time being, but what if one day you need to add one more field? If you add it,\nyou cannot read it from a decoder any longer because it will break the\npipeline. Here is an example:\n\n~~~clojure\n;; before\n(deftype SomeType [x y z]\n ...)\n\n;; after\n(deftype SomeType [x y z a]\n ...)\n~~~\n\nNow you alter the `-decode` multimethod so it retrieves the `a` field as well:\n\n~~~clojure\n(defmethod deed/-decode SomeTypeOID\n [_ decoder]\n (let [x (deed/decode decoder)\n y (deed/decode decoder)\n z (deed/decode decoder)\n a (deed/decode decoder)]\n (new SomeType x y z a)))\n~~~\n\nWhen reading an old dump made **before** adding `a` to `SomeType`, it will lead\nto a mysterious bug. The `a` value belongs to another item that has been written\nafter `SomeType`. It might not even trigger an exception but rather return a\nweird data.\n\nTo solve this, collect fields into a single collection and emit it into the\nencoder. A map is the best candidate:\n\n~~~clojure\n(deftype SomeType [x y z]\n deed/IEncode\n (-encode [this encoder]\n (let [fields {:x x :y y :z z}]\n (deed/writeOID encoder SomeTypeOID)\n (deed/encode encoder fields))))\n~~~\n\nWhen decoding, split that map on fields:\n\n~~~clojure\n(defmethod deed/-decode SomeTypeOID\n [_ decoder]\n (let [{:keys [x y z]}\n (deed/decode decoder)]\n (new SomeType x y z)))\n~~~\n\nNow that you have introduced a new `a` field into the type, just add it into the\n`:keys` vector and pass into the constructor:\n\n~~~clojure\n(defmethod deed/-decode SomeTypeOID\n [_ decoder]\n (let [{:keys [x y z a]}\n (deed/decode decoder)]\n (new SomeType x y z a)))\n~~~\n\nThe `a` field, since missing in the origin map, will be `nil`. You can add any\nkind of default fallback, for example merge the decoded map with defaults:\n\n~~~clojure\n(def Defaults\n {:x 1\n :y \"this\"\n :z true\n :a {:some \"map\"}})\n\n(defmethod deed/-decode SomeTypeOID\n [_ decoder]\n (let [decoded\n (deed/decode decoder)\n\n {:keys [x y z a]}\n (merge Defaults decoded)]\n\n (new SomeType x y z a)))\n~~~\n\nWith this approach, you'll be safe when extending types and reading old dumps.\n\n## Handling Defrecords\n\nBy default, records defined with the `defrecord` macro are read as Clojure\nmaps. This is because they're created at runtime and thus are not known to\nDeed. To preserve the origin type, either you extend a record with the `IEncode`\nprotocol and extend the `-decode` multimethod as described above. Another way is\nto use the `handle-record` macro that does the same. It accepts just two\narguments: a unique OID and the type of a record:\n\n~~~clojure\n(defrecord Bar [x y])\n\n(deed/handle-record 4321 Bar)\n~~~\n\nThe body of the macro is missing as you don't need to pass anything\nelse. Internally, a record is still encoded as a Clojure map but using the OID\nyou passed. When decoding this map, before it gets returned to the user, it's\nwrapped into the `\u003cYourType\u003e/create` invocation which creates a record instance\nfrom a map.\n\n## Contrib\n\nDeed comes with some minor packages that will make your life a bit easier.\n\n### Base64\n\nThe package `deed-base64` brings functions to encode values into a\nbase64-encoded stream, and read them back. This is useful when passing encoded\ndata throughout a text format, for example JSON. The package relies on\n`Base64OutputStream` and `Base64InputStream` classes from the Apacke Commons\nCodec library. As it's a third-party dependency, the functionalty is shipped in\na sub-package.\n\nA quick example:\n\n~~~clojure\n(ns deed-base64-demo\n (:require\n [deed.core :as deed]\n [deed.base64 :as b64]))\n\n(with-open [out (-\u003e \"dump.deed.b64\"\n (b64/base64-output-stream))]\n (deed/encode-to [1 2 3] out))\n~~~\n\nThe `base64-output-stream` function wraps any source and makes an instance of\n`Base64OutputStream`. As you write to it, the data gets silently base64-encoded:\n\n~~~clojure\n(slurp \"dump.deed.b64\")\n\"AAEAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAuAAAAAwAOAAwAAAAAAAAAAgAMAAAAAAAAAAM=\"\n~~~\n\nRead it again by wrapping the file into the `base64-input-stream` function:\n\n~~~clojure\n(with-open [in (-\u003e \"dump.deed.b64\"\n (io/file)\n (b64/base64-input-stream))]\n (deed/decode-from in))\n\n;; [1 2 3]\n~~~\n\nThe `deed.base64` namespace provides a number of shortcuts, namely:\n\n| Function | Comment |\n|---------------------------------|------------------------------------------------------|\n| `encode-to-base64-bytes` | Encode a single value into base64 byte array |\n| `encode-seq-to-base64-bytes` | Encode a sequence of values into a base64 byte array |\n| `encode-to-base64-string` | Encode a single value into a base64 string |\n| `encode-seq-to-base64-string` | Encode a sequence of values into a base64 string |\n| `decode-from-base64-bytes` | Decode a single value from a base64 byte array |\n| `decode-from-base64-string` | Decode a single value from a base64 string |\n| `decode-seq-from-base64-bytes` | Decode a sequence of values from base64 byte array |\n| `decode-seq-from-base64-string` | Decode a sequence of values from base64 string |\n\nTheir signatures are similar to what we've covered so far.\n\n### VectorZ\n\nThe `deed-vectorz` package extends Deed with classes the from the\n[mikera/vectorz][vectorz] library. These vectors are good for fast\ncomputations. A brief demo:\n\n~~~clojure\n(ns demo\n (:require\n [deed.core :as deed]\n [deed.vectorz :as vz])\n (:import\n (mikera.vectorz Vectorz)))\n\n(def vz\n (Vectorz/create (double-array [1.1 2.2 3.3])))\n\n(def dump\n (deed/encode-to-bytes vz))\n\n(deed/decode-from dump)\n;; #object[mikera.vectorz.Vector3 0x5ecb1a9a \"[1.1,2.2,3.3]\"]\n~~~\n\nThe `Vectorz` class is a general factory for other classes like `Vector1`,\n`Vector2` and similar. Deed extends the `AVector` class which is common for all\nof these.\n\n## Binary Format\n\nThe binary payload of Deed is quite simple. It consists from the following\npattern:\n\n~~~\n\u003c2-byte-OID\u003e\u003ccontent\u003e\u003c2-byte-OID\u003e\u003ccontent\u003e...\n~~~\n\nAt the beginning, there is always a `deed.Header` object with general\ninformation about how encoding was made. At the moment, it only tracks the\nversion of the protocol and nothing else. The header has constant size of 30\nbytes where unused bytes are reserved. In there future, there might be more data\nin the header.\n\nThe content depend on the nature of a type. Say, if it's an integer, there are\nalways four bytes. If it's a string, than we have four-byte length of the\nupcoming byte array, and then the array by itself.\n\nCounted collections are encoded whis way too. First, there is a total length of\na collection, and the items encoded one by one. As the items might be of\ndifferent types, they have their own OIDs:\n\n~~~\n\u003c2-byte-vector-id\u003e\u003c4-byte-vector-length\u003e\u003coid-item-1\u003e\u003cpayload-item-1\u003e\u003coid-item-2\u003e\u003cpayload-item-2\u003e\n~~~\n\n[encoder]: deed-core/src/java/deed/Encoder.java\n[decoder]: deed-core/src/java/deed/Decoder.java\n\nThis section, perhaps is not too detailed at the moment, will be extended in the\nfuture. For now, check out the source code: see the [Encoder.java][encoder] and\n[Decoder.java][decoder] files.\n\n## Benchmarks\n\nDeed is a bit faster than Nippy as it's written mostly in Java. The time\ndifference depends on the nature of data being processed, but in general Deed is\nabout 1.3-1.5 times faster. Here are encoding metrics made on two various\nmachines using the Criterium library:\n\n\u003cimg src=\"charts/encoding.svg\" width=75% height=auto\u003e\n\nDecode measurements made for the same data:\n\n\u003cimg src=\"charts/decoding.svg\" width=75% height=auto\u003e\n\n---\n\nIvan Grishaev, 2024\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrishaev%2Fdeed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Figrishaev%2Fdeed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Figrishaev%2Fdeed/lists"}