{"id":32191280,"url":"https://github.com/clojurewerkz/buffy","last_synced_at":"2025-10-22T01:36:27.148Z","repository":{"id":11795405,"uuid":"14339633","full_name":"clojurewerkz/buffy","owner":"clojurewerkz","description":"Buffy The ByteBuffer Slayer, Clojure library for working with binary data.","archived":false,"fork":false,"pushed_at":"2022-06-11T03:09:27.000Z","size":234,"stargazers_count":194,"open_issues_count":2,"forks_count":13,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-10-22T01:36:24.384Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clojurewerkz.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-11-12T17:27:52.000Z","updated_at":"2025-03-13T22:59:07.000Z","dependencies_parsed_at":"2022-08-07T06:16:35.534Z","dependency_job_id":null,"html_url":"https://github.com/clojurewerkz/buffy","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/clojurewerkz/buffy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clojurewerkz%2Fbuffy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clojurewerkz%2Fbuffy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clojurewerkz%2Fbuffy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clojurewerkz%2Fbuffy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clojurewerkz","download_url":"https://codeload.github.com/clojurewerkz/buffy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clojurewerkz%2Fbuffy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280365574,"owners_count":26318385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-21T02:00:06.614Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-22T01:36:23.140Z","updated_at":"2025-10-22T01:36:27.140Z","avatar_url":"https://github.com/clojurewerkz.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Buffy, The Byte Buffer Slayer\n\nBuffy is a Clojure library for working with binary data, writing\ncomplete binary protocol implementations in Clojure, storing complex\ndata structures in an off-heap cache, reading binary files and doing\neverything you would usually do with `ByteBuffer`.\n\n## Main features\n\n  * partial deserialization (read and deserialize parts of a byte buffer)\n  * named access (access parts of your buffer by names)\n  * composing/decomposing from key/value pairs\n  * pretty hexdump\n  * many useful default types that you can combine and extend easily\n\n\n## Project Maturity\n\nBuffy is a young project. The API is fairly stable and the project has reached\n1.0 in December 2014.\n\n\n## Installation\n\n### Artefacts\n\nLatest artifacts are published to\n[Clojars](https://clojars.org/clojurewerkz/buffy) If you are using\nMaven, add the following repository definition to your `pom.xml`:\n\n``` xml\n\u003crepository\u003e\n  \u003cid\u003eclojars.org\u003c/id\u003e\n  \u003curl\u003ehttp://clojars.org/repo\u003c/url\u003e\n\u003c/repository\u003e\n```\n\n### The Most Recent Release\n\nWith Leiningen:\n\n```clj\n[clojurewerkz/buffy \"1.1.0\"]\n```\n\nWith Maven:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003eclojurewerkz\u003c/groupId\u003e\n  \u003cartifactId\u003ebuffy\u003c/artifactId\u003e\n  \u003cversion\u003e1.1.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n## Usage\n\nRequire Buffy's main namespace:\n\n``` clojure\n(ns my.app\n  (:require [clojurewerkz.buffy.core :refer :all]))\n```\n\nBuffy creates buffers from a spec you specify. The spec consists of\none or more fields of known data types, for example:\n\n```clojure\n(spec :my-field-1 (int32-type)\n      :my-field-2 (string-type 10))\n```\n\nThe spec can be a map (e.g. `array-map`) or a vector of vectors.\nAvoid using hash maps, since they are unordered.\n\nBelow is a specification for a buffer containing 2 fields, one 4 bytes\nlong and second one 10:\n\n```\n0            4                         14\n+------------+-------------------------+\n| my-field-1 |         my-field-2      |\n|    (int)   |        (10 string)      |\n+------------+-------------------------+\n```\n\nNow you can use this specification to create a byte buffer:\n\n```clojure\n(compose-buffer (spec :my-field-1 (int32-type) :my-field-2 (string-type 10)))\n;= a byte buffer\n```\n\nNote that they keys (`:my-field-1`, `:my-field-2`) are not part of the\nbyte buffer (not serialized in the data). If you're transferring a\nbyte buffer over a network, the receiving end should be able to\ndeserialize it.\n\n### Accessing Fields In The Payload\n\nUse `get-field` and `set-field` to access individual fields of the\npayload.\n\nHere's an example:\n\n```clojure\n(ns my-binary-project.core\n  (:require [clojurewerkz.buffy.core :refer :all]))\n\n(let [s    (spec :int-field (int32-type)\n                 :string-field (string-type 10))\n      buf  (compose-buffer s)]\n\n  (set-field buf :int-field 101)\n  (get-field buf :int-field)\n  ;; =\u003e 101\n\n  (set-field buf :string-field \"stringie\")\n  (get-field buf :string-field)\n  ;; =\u003e \"stringie\"\n  )\n```\n\n\n### Deserializing complete buffer\n\nYou can also serialize and deserialize a complete buffer:\n\n```clojure\n\n(let [s     (spec :first-field (int32-type)\n                  :second-field (string-type 10)\n                  :third-field (boolean-type))\n      buf  (compose-buffer spec)]\n\n  (compose buf {:first-field 101\n                :second-field \"string\"\n                :third-field true})\n\n  (decompose buf)\n  ;; =\u003e {:third-field true :second-field \"string\" :first-field 101}\n)\n```\n\n\n## Data Types\n\nBuilt-in data types are:\n\n### Primitive types\n\n  * `int32-type`: 32 bit integer\n  * `boolean-type`: boolean (1 byte)\n  * `byte-type`: a single byte\n  * `short-type`: 16 bit integer\n  * `medium-type`: 24 bit integer\n  * `float-type`: 32 bit floating point\n  * `long-type`: 64 bit integer\n\n### Arbitrary-length types\n\n  * `string-type` - arbitrary size `string` (you define the length).\n\nIn order to construct a `string-type`, specify its length:\n\n```clojure\n(string-type 15)\n```\n\n  * `bytes-type` - arbitrary size `byte-array` (you define length yourself)\n\nSame is true for `BytesType`, when constructing it, just pass a number of bytes it should\ncontain:\n\n```clojure\n(bytes-type 25)\n```\n\n### Bit type\n\nBit type is `n` bits long sequence of bits that are turned either on or off, for example,\n\n```clojure\n[true true false false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false]\n```\n\nTranslates to binary\n\n```\n0000 0000 0000 00011\n```\n\nWhich translates to decimal `3`, that is stored in a 4-bits integer field.\n\nThere are some helper functions, such as:\n\n```clojure\n(clojurewerkz.buffy.util/bits-on-at [0 1 2])\n\n[true true true false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false\n false false false false]\n```\n\nOr an inverse of it, `clojurewerkz.buffy.util/bits-off-at`.\n\nAlso, `on-bits-indexes` that returns positions at which bits are set, and\n`off-bits-indexes` that returns positions at which bits are cleared.\n\nIn order to use bit type, you need to give it a 32-items long sequence of\ntruthy or falsy falues:\n\n```clojure\n(let [s (spec :first-field (bit-type 4) ;; Bit field that fills 4 bytes\n              :second-field (string-type 10))\n      buf (compose-buffer s)]\n  (set-field b :first-field [true  true  false false\n                             false false false false\n                             false false false false\n                             false false false false\n                             false false false false\n                             false false false false\n                             false false false false\n                             false false false false]))\n```\n\n### Complex (Composite) Types\n\nComposite types combine multiple primitive types.\n\n`composite-type` produces a slice of a buffer. In the byte representation, no\npaddings or offsets are added. All parts are written to the buffer\nsequentially:\n\nHere's what composite type consisting of `int` and 10 characters long\n`string` would look like:\n\n```clojure\n(composite-type (int32-type) (string-type 10))\n```\n\n`repeated-type` repeats a type one or more times.  Repeated types are\nused when you need to have many fields of the same size:\n\n```clojure\n(repeated-type (string-type 10) 5)\n```\n\nwill produce a type consisting of 5 `strings` of length 10.\n\nIt's possible to combine `repeated-type` and `composite-type`:\n\n```clojure\n(repeated-type (composite-type (int32-type) (string-type 10)) 5)\n```\n\nWhich will construct a type consisting of `int`/`string` chunks\nrepeated 5 times.\n\n `enum-type` produces a mapping between human-readable values and\n their internal binary representation.\n\nConsider a binary protocol where the `STARTUP` verb is encoded as a\nlong value of `0x01` and the `QUERY` verb is encoded as `0x07`:\n\n```clojure\n(enum-type (long-type) {:STARTUP 0x02 :QUERY 0x07})\n```\n\nWith this enum type, it is possible to set a field using `:STARTUP`\nand `:QUERY` keywords:\n\n```clojure\n(set-field buffer :payload-type :STARTUP)\n```\n\nWhen reading a field, its symbolic representation is returned:\n\n```clojure\n(get-field buffer :payload-type)\n;; =\u003e :QUERY\n```\n\n## Buffer types\n\nCurrently, Buffy supports `direct`, `heap` and `wrapped` buffers.\nIn order to create a heap buffer:\n\n```clojure\n(def my-spec (spec :first-field (int32-type)\n                   :second-field (string-type 10)))\n(compose-buffer my-spec :buffer-type :heap)\n```\n\nFor off-heap (direct) buffer:\n\n```clojure\n(def my-spec (spec :first-field (int32-type)\n                   :second-field (string-type 10)))\n(compose-buffer my-spec :buffer-type :direct)\n```\n\nAnd for wrapped buffer (that wraps the given byte array,\n`j.nio.ByteBuffer` or netty `ByteBuf`):\n\n```clojure\n(def my-spec (spec :first-field (int32-type)\n                   :second-field (string-type 10)))\n(compose-buffer my-spec :orig-buffer (java.nio.ByteBuffer/allocate 14))\n```\n\n## Dynamic Frames\n\nIf you're working with sophisticated protocols, more often than not you can't know\nthe buffer size before you construct an entire type. One of the most primitive examples\nis the `netstrings` protocol, that consists of\n\n```clojure\n(short-type) ;; Identifies the length of string\n(string-type 10) ;; Identifies the string itself\n```\n\nProblem with construction of such type lays in the fact that you can't construct a buffer\nbefore you know the value of the string itself. Buffy helps you here, too. This feature\nis called dynamic frame. In order to construct a dynamic frame, you should create an\nencoder and decoder. Let's take a closer look at netstrings protocol implementation:\n\nFirst, encoder:\n\n```clojure\n(frame-encoder [value]\n               ;; Name     ;; Child frame or type      ;; Dynamic value\n               length      (short-type)                (count value)\n               string      (string-type (count value)) value)\n```\n\nHere, in a binding you have a `value`. `length` part of the frame is a `short-type` that\nholds a length of the string, you specify this value through `(count value)`.\n\nNext off, the `string` itself, that is a `string-type` and holds a `value` itself.\n\nDecoder is written in a same manner:\n\n```clojure\n(frame-decoder [buffer offset]\n               length (short-type)\n               string (string-type (read length buffer offset)))\n```\n\nSince values are not decoded by that time just yet, and you may need access to an entire\nbuffer in order to read a certain field's value, you specify only types and have a possibility\nof \"look-behind\", using already constructed types.\n\nSo, `string` type is constructed by reading off the `length` as a first field of the frame.\n\nAn entire frame would look as follows:\n\n```clojure\n(def dynamic-string-payload\n  (dynamic-buffer\n   (frame-type\n    (frame-encoder [value]\n                   length (short-type) (count value)\n                   string (string-type (count value)) value)\n    (frame-decoder [buffer offset]\n                   length (short-type)\n                   string (string-type (read length buffer offset)))\n    second ;; Value Formatter\n    )))\n```\n\n`second` here is just a value formatter. When we read off the value from the buffer, we see the\n`short` as well as `string`, but it's just a helper for correct decomposition, therefore\nwe should discard it and take only the second value, which is a string itself.\n\nIn order to compose/decompose it, you should use `compose` and `decompose` functions:\n\n```clojure\n(compose dynamic-string-payload [\"super-duper-random-string\" \"long-ans-senseless-stringyoyoyo\"])\n```\n\nThis one will return a buffer. Same with `decompose`, that receives dynamic buffer and a value,\nand returns deserialized value.\n\nYou can go ahead and create even more complicated patterns. For example, you can construct\na map of strings (as in Cassandra binary CQL protocol), where the map is specified by\n\n```\n\u003cshort\u003e|(repeated \u003cstring\u003e|\u003cstring\u003e)\n```\n\nWhere each `\u003cstring\u003e` is actually\n\n```\n\u003cshort\u003e|\u003cstring itself\u003e\n```\n\nIt's implementation is a little bit more complex, but still reasonably simple. First, we\ndefine a dynamic string frame in the same manner as we made with `netstrings`:\n\n```clojure\n(def dynamic-string\n  (frame-type\n   (frame-encoder [value]\n                  length (short-type) (count value)\n                  string (string-type (count value))\n                  value)\n   (frame-decoder [buffer offset]\n                  length (short-type)\n                  string (string-type (read length buffer offset)))\n   second))\n```\n\nNext off, key-value pairs. Each one of them is nothing more than a string repeated twice.\n\n```clojure\n(def key-value-pair\n  (composite-frame\n   dynamic-string\n   dynamic-string))\n```\n\nNext is dynamic map, which is a frame type that holds a `length` which is `short-type` and\n`repeated-frame` of `key-value-pairs`:\n\n```clojure\n(def dynamic-map\n  (frame-type\n   (frame-encoder [value]\n                  length (short-type) (count value)\n                  map    (repeated-frame key-value-pair (count value)) value)\n   (frame-decoder [buffer offset]\n                  length (short-type)\n                  map    (repeated-frame key-value-pair (read length buffer offset)))\n   second))\n```\n\nNow, our dynamic map is ready for composition and decomposition:\n\n```clojure\n(let [dynamic-type (dynamic-buffer dynamic-map)]\n  (compose dynamic-type [[[\"key1\" \"value1\"] [\"key1\" \"value1\"] [\"key1\" \"value1\"]]]) ;; Returns a constructred buffer\n\n  (-\u003e dynamic-type\n      (compose [[[\"key1\" \"value1\"] [\"key1\" \"value1\"] [\"key1\" \"value1\"]]])\n      decompose) ;; Decomposes it back to the key-value pairs\n```\n\n## Working With Bits\n\nIn Java, there are no data types for bits, therefore\nwe've added some wrapper functions for existing types, that may\nrepresent your values as series of 1s and 0es. For example, you can\nconvert an integer `101` to it's binary representation:\n\n```clojure\n(to-bit-map (int32-type) 101)\n```\n\nThis will return a bitmap of `0000 0000   0000 0000   0000 0000   0110 0101` (represented as\nvector of `true` and `false`), which is a binary representation of `101`.\n\nSame way, you can convert a bitmap consisting of `true` and `false`\nback to it's actual value with `from-bit-map` function.\n\n## Hex Dump\n\nIt is possible to produce a hex-dump of a buffer created with Buffy\nusing `clojurewerkz.buffy.util/hex-dump`. It will produce the\nfollowing representation:\n\n```\n            +--------------------------------------------------+\n            | 0  1  2  3  4  5  6  7   8  9  a  b  c  d  e  f  |\n +----------+--------------------------------------------------+------------------+\n | 00000000 | 48 69 65 72 20 69 73 74  20 65 69 6e 20 42 65 69 | Hier ist ein Bei |\n | 00000010 | 73 70 69 65 6c 74 65 78  74 2e 20 44 65 72 20 48 | spieltext. Der H |\n | 00000020 | 65 78 64 75 6d 70 20 69  73 74 20 61 75 66 20 64 | exdump ist auf d |\n | 00000030 | 65 72 20 6c 69 6e 6b 65  6e 20 53 65 69 74 65 20 | er linken Seite  |\n | 00000040 | 7a 75 20 73 65 68 65 6e  2e 20 4e 65 75 65 20 5a | zu sehen. Neue Z |\n | 00000050 | 65 69 6c 65 6e 20 6f 64  65 72 20 41 62 73 c3 a4 | eilen oder Abs.. |\n | 00000060 | 74 7a 65 20 73 69 6e 64  20 64 61 6e 6e 20 61 75 | tze sind dann au |\n | 00000070 | 63 68 20 22 5a 65 69 63  68 65 6e 22 20 6d 69 74 | ch \"Zeichen\" mit |\n | 00000080 | 20 65 69 6e 65 6d 20 62  65 73 74 69 6d 6d 74 65 |  einem bestimmte |\n | 00000090 | 6e 20 43 6f 64 65 20 28  30 61 29 00 00 00 00 00 | n Code (0a)..... |\n +----------+--------------------------------------------------+------------------+\n```\n\n\n## Community\n\nTo subscribe for announcements of releases, important changes and so on, please follow [@ClojureWerkz](http://twitter.com/clojurewerkz) on Twitter.\n\n## Supported Clojure Versions\n\nBuffy requires Clojure 1.4+.\n\n## Continuous Integration Status\n\n[![Continuous Integration status](https://secure.travis-ci.org/clojurewerkz/buffy.png)](http://travis-ci.org/clojurewerkz/buffy)\n\n## Buffy Is a ClojureWerkz Project\n\nBuffy is part of the [group of Clojure libraries known as ClojureWerkz](http://clojurewerkz.org), together with\n\n * [Langohr](http://clojurerabbitmq.info)\n * [Elastisch](http://clojureelasticsearch.info)\n * [Cassaforte](http://clojurecassandra.info)\n * [Monger](http://clojuremongodb.info)\n * [Titanium](http://titanium.clojurewerkz.org)\n * [Neocons](http://clojureneo4j.info)\n * [Quartzite](http://clojurequartz.info)\n\nand several others.\n\n\n## Development\n\nBuffy uses [Leiningen 2](http://leiningen.org). Make sure you\nhave it installed and then run tests against supported Clojure\nversions using\n\n    lein all test\n\nThen create a branch and make your changes on it. Once you are done\nwith your changes and all tests pass, submit a pull request on GitHub.\n\n\n\n## License\n\nCopyright (C) 2013-2016 Alex Petrov, Michael S. Klishin and the ClojureWerkz Team.\n\nDouble licensed under the [Eclipse Public License](http://www.eclipse.org/legal/epl-v10.html) (the same as Clojure) or\nthe [Apache Public License 2.0](http://www.apache.org/licenses/LICENSE-2.0.html).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclojurewerkz%2Fbuffy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclojurewerkz%2Fbuffy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclojurewerkz%2Fbuffy/lists"}