{"id":24600799,"url":"https://github.com/conspack/cl-conspack","last_synced_at":"2025-10-05T21:32:09.397Z","repository":{"id":4700513,"uuid":"5847766","full_name":"conspack/cl-conspack","owner":"conspack","description":"Implementation for Common Lisp.","archived":false,"fork":false,"pushed_at":"2023-02-02T15:44:46.000Z","size":95,"stargazers_count":86,"open_issues_count":10,"forks_count":9,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-03-26T18:19:50.543Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Common Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conspack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-09-17T21:38:42.000Z","updated_at":"2024-03-06T17:04:33.000Z","dependencies_parsed_at":"2023-02-17T21:30:20.492Z","dependency_job_id":null,"html_url":"https://github.com/conspack/cl-conspack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conspack%2Fcl-conspack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conspack%2Fcl-conspack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conspack%2Fcl-conspack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conspack%2Fcl-conspack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conspack","download_url":"https://codeload.github.com/conspack/cl-conspack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235448520,"owners_count":18991891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-24T14:01:33.964Z","updated_at":"2025-10-05T21:32:04.102Z","avatar_url":"https://github.com/conspack.png","language":"Common Lisp","readme":"# News\n\n**Recent changes**:\n\n* Decoder now handles more uses of circular reference.\n* Properties now require `WITH-PROPERTIES` if used outside an `ENCODE`\n  or `DECODE`, see below.\n\n# cl-conspack\n\nCONSPACK was inspired by MessagePack, and by the general lack of\nfeatures among prominent serial/wire formats:\n\n* JSON isn't terrible, but can become rather large, and is potentially\n  susceptible to READ exploits (as per the recently-fixed\n  \"10e99999999\" bug in SBCL).\n\n* BSON (binary JSON) doesn't really solve much; though it encodes\n  numbers, it's not particularly smaller or more featureful than JSON.\n\n* MessagePack is small, but lacks significant features; it can\n  essentially encode arrays or maps of numbers, and any interpretation\n  beyond that is up to the receiver.\n\n* Protobufs and Thrift are static.\n\nIt should be noted that, significantly, **none** of these support\nreferences.  Of course, references can be implemented at a higher\nlayer (e.g., JSPON), but this requires implementing an entire\nadditional layer of abstraction and escaping, including rewalking the\nparsed object hierarchy and looking for specific signatures, which can\nbe error-prone and hurts performance.\n\nAdditionally, none of these appear to have much in the way of\nsecurity, and communicating with an untrusted peer is probably not\nrecommended.\n\nCONSPACK, on the other hand, attempts to be a more robust solution:\n\n* Richer set of data types, differentiating between arrays, lists,\n  maps, typed-maps (for encoding classes/structures etc), numbers,\n  strings, symbols, and a few more.\n\n* Very compact representation that can be smaller than MessagePack.\n\n* In-stream references, including optional forward references, which\n  can allow for shared or circular data structures.  Additionally,\n  remote references allow the receiver the flexibility to parse and\n  return its own objects without further passes on the output.\n\n* Security, including byte-counting for (estimated) maximum output\n  size, and the elimination of circular data structures.\n\n* Speed. Using [fast-io](https://github.com/rpav/fast-io) encoding\n  and decoding can be many times faster than alternatives, even\n  *while* tracking references (faster still without!).\n\nSee [SPEC](https://github.com/conspack/cl-conspack/blob/master/doc/SPEC) for\ncomplete details on encoding.\n\n## Usage\n\n`cl-conspack` is simple to use:\n\n```lisp\n(encode '(1 2 3)) ;; =\u003e #(40 4 16 1 16 2 16 3 0)\n\n(decode (encode '(1 2 3))) ;; =\u003e (1 2 3)\n\n;; Smaller if the element-type is known:\n(encode (fast-io:octets-from '(1 2 3)))\n  ;; =\u003e #(36 3 20 1 2 3)\n```\n\n### CLOS and general objects\n\nConspack provides the ability to serialize and deserialize objects of\nany kind.\n\nThe easiest way, for the common case:\n\n```lisp\n(conspack:defencoding my-class\n  slot-1 slot-2 slot-3)\n```\n\nThis expands to the more flexible way, which specializes\n`ENCODE-OBJECT` and `DECODE-OBJECT-INITIALIZE`:\n\n```lisp\n(defmethod conspack:encode-object append\n    ((object my-class) \u0026key \u0026allow-other-keys)\n  (conspack:slots-to-alist (object)\n    slot-1 slot-2 slot-3 ...))\n\n(defmethod conspack:decode-object-initialize progn\n    ((object my-class) class alist \u0026key \u0026allow-other-keys)\n  (declare (ignore class))\n  (alist-to-slots (alist object)\n    slot-1 slot-2 slot-3))\n```\n\n`ENCODE-OBJECT` should specialize on the object and return an alist.\nThe alist returned will be checked for circularity if `tracking-refs`\nis in use.\n\n`DECODE-OBJECT-ALLOCATE` should specialize on `(eql 'class-name)`, and\nproduce an object *based* on the class and alist.\n\n`DECODE-OBJECT-INITIALIZE` should specialize on the object (which has\nbeen produced by `DECODE-OBJECT-ALLOCATE`), and initializes it.\nThis two step process is necessary to handle circularity correctly.\n\nAs you can see, this does not require objects be in any particular\nformat, or that you store any particular slots or values.  It does not\nspecify how you restore an object.\n\nBut for the \"normal\" case, `SLOTS-TO-ALIST` and `ALIST-TO-SLOTS` are\nprovided to build and restore from alists, and `DEFENCODING` can\ndefine all of this in one simple form.\n\n### Circularity and References\n\nCircularity tracking is not on by default, you can enable it for a\nparticular block of `encode`s or `decode`s by using `tracking-refs`:\n\n```lisp\n(tracking-refs ()\n  (decode (encode CIRCULAR-OBJECT)))\n```\n\n\"Remote\" references are application-level references.  You may encode\na reference using an arbitrary object as a descriptor:\n\n```lisp\n(encode (r-ref '((:url . \"http://...\"))))\n```\n\nWhen decoding, you may provide a function to handle these:\n\n```lisp\n(with-remote-refs (lambda (x) (decode-url x))\n  (decode OBJECT))\n```\n\n### Indexes\n\nIf you have a relatively small static set of symbols you will always\nuse for a particular encoding/decoding, you may want to use\n*indexes*.  These allow symbols to be very-tightly-packed: for up to\n15 symbols, a single byte can encode the symbol!  For up to 256, two\nbytes, and so on.\n\nTrivially:\n\n```lisp\n(cpk:with-index (specifier-1 specifier-2 specifier-3)\n  (cpk:encode '(specifier-1 specifier-2 specifier-3)))\n\n;; =\u003e #(40 4 176 177 178 0)\n\n;; Contrast this with:\n\n(cpk:encode '(specifier-1 specifier-2 specifier-3))\n;; #(40 4 130 64 11 83 80 69 67 73 70 73 69 82 45 49 129 64 16 67 79\n;; 77 77 79 78 45 76 73 83 80 45 85 83 69 82 130 64 11 83 80 69 67 73\n;; 70 73 69 82 45 50 129 64 16 67 79 77 77 79 78 45 76 73 83 80 45 85\n;; 83 69 82 130 64 11 83 80 69 67 73 70 73 69 82 45 51 129 64 16 67 79\n;; 77 77 79 78 45 76 73 83 80 45 85 83 69 82 0)\n```\n\n(This is a somewhat excessive example, since long non-keyword symbols\nare used.  Shorter keyword symbols would be relatively shorter, but\nthis is the general case.)\n\nFor more \"realistic\" use, you may *define* an index and refer to it:\n\n```lisp\n(define-index index-name\n  symbol-1 symbol-2 ...)\n\n(with-named-index 'index-name\n  (encode ...))\n```\n\nFor instance, you may define multiple indexes for multiple different\nformat versions, read the version, and use the appropriate index:\n\n```lisp\n(define-index version-1 ...)\n(define-index version-2 ...)\n\n(let ((version (decode-stream s)))\n  (with-named-index version\n    ;; Decode the rest of the stream appropriately.  You may want to\n    ;; do more checking on VERSION if security is required...\n    (decode-stream s)))\n```\n\nNote that using `tracking-refs` will *also* help encode symbols\nefficiently, but not *quite* as efficiently:\n\n* The full string for the symbol (and if necessary, package), will be\n  encoded at least once, when first encountered\n* Refs are tracked in-order, and may lead to longer tags than a\n  comparable index would use\n\nHowever, `tracking-refs` is a perfectly suitable option, especially if\nflexibility is desired, since all symbol information is encoded, and\nnothing special is needed for decoding.\n\n### Properties\n\n(Properties now require a `WITH-PROPERTIES` block in some\ncircumstances, see below.)\n\nProperties are a way to specify additional information about an object\nthat may be useful at decode-time.  For instance, while hash tables\nare supported as maps, there are no bits to specify the `:test`\nparameter, so decoding a hash table of strings would produce a useless\nobject.  In this case, the `:test` property is set when encoding and\nchecked when decoding hash tables.\n\nYou may specify arbitrary properties for arbitrary objects; the only\nrestriction is the objects must test by `EQ`.\n\n```lisp\n(conspack:with-properties ()\n  (let ((object (make-instance ...)))\n    (setf (property object :foo) 'bar)\n    (property object :foo))) ;; =\u003e BAR\n```\n\nThis sets the `:foo` property to the symbol `bar`, and it is encoded\nalong with the object.  Note this will increase the object size, by\nthe amount required to store a map of symbols-to-values.\n\nWhen decoding, you can access properties about an object via\n`*current-properties*`:\n\n```lisp\n(defmethod decode-object-initialize (...)\n  (let ((prop (getf *current-properties* NAME)))\n    ...))\n```\n\nYou may remove them with `remove-property` or `remove-properties`.\n\n**Properties are now only available within a `WITH-PROPERTIES`\nblock.** This has a number of benefits, including some thread safety,\nand ensuring properties don't stick around forever.\n\n`ENCODE` and `DECODE` have **implicit** `WITH-PROPERTIES` blocks: you\ndon't need to specify `WITH-PROPERTIES` if you use properties inside\n`ENCODE-OBJECT`, `DECODE-OBJECT`, or encode and decode any objects\nthat have implicit properties.  You only need this if you wish to\naccess properties *outside* of the encode or decode (e.g.,\npreassigning properties to be encoded).\n\n### Allocation Limits and Security\n\nConspack provides some level of \"security\" by *approximately* limiting\nthe amount of bytes allocated when reading objects.\n\nBy default, because format sizes are prespecified statically, it's\npossible to specify extremely large allocations for e.g. arrays with\nonly a few bytes.  Obviously, this is not suitable for untrusted\nconspack data.\n\nThe solution is simply to cap allocations:\n\n```lisp\n(with-conspack-security (:max-bytes 200000)\n  (decode ...))\n```\n\nSince actual allocation sizes are rather difficult to get in most\nlisps, this *approximates* the allocation based on how big each object\nmight be, e.g.:\n\n* `pointer-size * array-size`\n* `string-length`\n* `number-size`\n* etc.\n\nEach object header is tallied against the limit just prior to its\ndecoding; if the object would exceed the allowed bytes, decoding halts\nwith an error.\n\nFurther options may be added in the future.\n\n### Interning\n\nBy default, Conspack does not intern symbols, on the assumption that\nthe producer and consumer have agreed on what symbols to use\nbeforehand. If the decoder finds a symbol that has not already been\ninterned, it will ignore the symbol's package and make an uninterned\nsymbol instead.\n\nThe `with-interning` macro can be used if the decoder should instead\nintern symbols:\n\n```lisp\n(with-interning ()\n  (decode ...))\n```\n\nInterning symbols from untrusted data could lead to denial-of-service\nattacks via interning long-lived symbols in memory, so be careful.\n\n## Explaining\n\nSince conspack is a binary format, it's rather difficult for humans to\nread just looking at the stream of bytes.  Thus an `EXPLAIN` feature\nis provided.  This is mostly useful for debugging the format; however\nit may be of interest otherwise and certainly may be helpful when\ncreating other implementations.\n\nFor instance:\n\n```lisp\n(explain (encode '(1 2 3)))\n\n;; =\u003e\n((:LIST 4\n  ((:NUMBER :INT8 1) (:NUMBER :INT8 2) (:NUMBER :INT8 3) (:BOOLEAN NIL)))\n END-OF-FILE)\n```\n","funding_links":[],"categories":["Recently Updated","Online editors ##"],"sub_categories":["[Who Wants to Be a Millionare](https://www.boardgamecapital.com/who-wants-to-be-a-millionaire-rules.htm)","Third-party APIs"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconspack%2Fcl-conspack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconspack%2Fcl-conspack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconspack%2Fcl-conspack/lists"}