{"id":16894203,"url":"https://github.com/jsuereth/sauerkraut","last_synced_at":"2025-03-17T06:31:51.809Z","repository":{"id":50763842,"uuid":"240250097","full_name":"jsuereth/sauerkraut","owner":"jsuereth","description":"A reimagined scala-pickling in the Scala 3 world","archived":false,"fork":false,"pushed_at":"2023-05-20T15:14:22.000Z","size":331,"stargazers_count":73,"open_issues_count":0,"forks_count":2,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-16T09:11:49.744Z","etag":null,"topics":["picklers","scala","serialization-framework"],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jsuereth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-13T12:00:53.000Z","updated_at":"2025-03-06T11:29:39.000Z","dependencies_parsed_at":"2024-10-27T12:13:06.400Z","dependency_job_id":"5a98775d-d264-497f-8fa8-a24daaae68fb","html_url":"https://github.com/jsuereth/sauerkraut","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsuereth%2Fsauerkraut","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsuereth%2Fsauerkraut/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsuereth%2Fsauerkraut/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsuereth%2Fsauerkraut/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jsuereth","download_url":"https://codeload.github.com/jsuereth/sauerkraut/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243986147,"owners_count":20379272,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["picklers","scala","serialization-framework"],"created_at":"2024-10-13T17:17:59.426Z","updated_at":"2025-03-17T06:31:50.497Z","avatar_url":"https://github.com/jsuereth.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sauerkraut\n\nThe library for those cabbage lovers out there who want\nto send data over the wire.\n\nA revitalization of [Pickling](http://github.com/scala/pickling) in the\n[Scala 3](https://dotty.epfl.ch/docs/index.html) world.\n\n## Usage\n\nWhen defining over-the-wire messages, do this:\n\n```scala\nimport sauerkraut.core.{Buildable,Writer,given}\ncase class MyMessage(field: String, data: Int)\n  derives Buildable, Writer\n```\n\nThen, when you need to serialize, pick a format and go:\n\n```scala\nimport format.json.{Json,given}\nimport sauerkraut.{pickle,read,write}\n\nval out = StringWriter()\npickle(Json).to(out).write(MyMessage(\"test\", 1))\nprintln(out.toString())\n\nval msg = pickle(Json).from(out.toString()).read[MyMessage]\n```\n\n# Current Formats\n\nHere's a feature matrix for each format:\n\n| Format | Reader | Writer | All Types | Evolution Friendly | Notes                                    |\n| ------ | ------ | ------ | --------- | ------------------ | ---------------------------------------- |\n| Json   | Yes    | Yes    | Yes       | Yes                | Uses Jawn for parsing                    |\n| Protos | Yes    | Yes    | Yes       | Yes                | Binary format evolution friendly format  |\n| NBT    | Yes    | Yes    | Yes       |                    | For the kids.                            |\n| XML    | Yes    | Yes    | Yes       |                    | Inefficient prototype.                   |\n| Pretty | No     | Yes    | No        |                    | For pretty-printing strings              |\n\nSee [Compliance](compliance/README.md) for more details on what this means.\n\n## Json\nEveryone's favorite non-YAML web data transfer format!   This uses Jawn under the covers for parsing, but\ncan write Json without any dependencies.\n\n\nExample:\n```scala\nimport sauerkraut.{pickle,read,write}\nimport sauerkraut.core.{Buildable,Writer, given}\nimport sauerkraut.format.json.Json\n\ncase class MyWebData(value: Int, someStuff: Array[String])\n    derives Buildable, Writer\n\ndef read(in: java.io.InputStream): MyWebData =\n  pickle(Json).from(in).read[MyWebData]\ndef write(out: java.io.OutputStream): Unit = \n  pickle(Json).to(out).write(MyWebData(1214, Array(\"this\", \"is\", \"a\", \"test\")))\n```\n\nsbt build:\n```scala\nlibraryDependencies += \"com.jsuereth.sauerkraut\" %% \"json\" % \"\u003cversion\u003e\"\n```\n\nSee [json project](json/README.md) for more information.\n\n## Protos\nA new encoding for protocol buffers within Scala!  This supports a subset of all possible protocol buffer messages\nbut allows full definition of the message format within your Scala code.\n\nExample:\n```scala\nimport sauerkraut.{pickle,write,read, Field}\nimport sauerkraut.core.{Writer, Buildable, given}\nimport sauerkraut.format.pb.{Proto,,given}\n\n\ncase class MyMessageData(value: Int @Field(3), someStuff: Array[String] @Field(2))\n    derives Writer, Buildable\n\ndef write(out: java.io.OutputStream): Unit = \n  pickle(Proto).to(out).write(MyMessageData(1214, Array(\"this\", \"is\", \"a\", \"test\")))\n```\n\nThis example serializes to the equivalent of the following protocol buffer message:\n\n```proto\nmessage MyMessageData {\n  int32 value = 3;\n  repeated string someStuff = 2;\n}\n```\n\n\nsbt build:\n```scala\nlibraryDependencies += \"com.jsuereth.sauerkraut\" %% \"pb\" % \"\u003cversion\u003e\"\n```\n\nSee [pb project](pb/README.md) for more information.\n\n\n# NBT\nNamed-Binary-Tags, a format popularized by Minecraft.\n\nExample:\n```scala\nimport sauerkraut.{pickle,read,write}\nimport sauerkraut.core.{Buildable,Writer, given}\nimport sauerkraut.format.nbt.Nbt\n\ncase class MyGameData(value: Int, someStuff: Array[String])\n    derives Buildable, Writer\n\ndef read(in: java.io.InputStream): MyGameData =\n  pickle(Nbt).from(in).read[MyGameData]\ndef write(out: java.io.OutputStream): Unit = \n  pickle(Nbt).to(out).write(MyGameData(1214, Array(\"this\", \"is\", \"a\", \"test\")))\n```\n\nsbt build:\n```scala\nlibraryDependencies += \"com.jsuereth.sauerkraut\" %% \"nbt\" % \"\u003cversion\u003e\"\n```\n\n\nSee [nbt project](nbt/README.md) for more information.\n\n\n# XML\nEveryone's favorite markup language for data transfer!\n\nExample:\n```scala\nimport sauerkraut.{pickle,read,write}\nimport sauerkraut.core.{Buildable,Writer, given}\nimport sauerkraut.format.xml.{Xml, given}\n\ncase class MySlowWebData(value: Int, someStuff: Array[String])\n    derives Buildable, Writer\n\ndef read(in: java.io.InputStream): MySlowWebData =\n  pickle(Xml).from(in).read[MySlowWebData]\ndef write(out: java.io.Writer): Unit = \n  pickle(Xml).to(out).write(MySlowWebData(1214, Array(\"this\", \"is\", \"a\", \"test\")))\n```\n\nsbt build:\n```scala\nlibraryDependencies += \"com.jsuereth.sauerkraut\" %% \"xml\" % \"\u003cversion\u003e\"\n```\n\n\nSee [xml project](xml/README.md) for more information.\n\n\n# Pretty\nA format that is solely used to pretty-print object contents to strings.  This does not have\na [PickleReader] only a [PickleWriter].\n\nExample:\n```scala\nimport sauerkraut._, sauerkraut.core.{Writer,given}\ncase class MyAwesomeData(theBest: Int, theCoolest: String) derives Writer\n\nscala\u003e MyAwesomeData(1, \"The Greatest\").prettyPrint\nval res0: String = Struct(rs$line$2.MyAwesomeData) {\n  theBest: 1\n  theCoolest: The Greatest\n}\n```\n\n\n# Design\n\nWe split Serialization into three layers:\n\n1. The `source` layer.  It is expected these are some kind of stream.\n2. The `Format` layer.  This is responsible for reading a raw source and converting into\n   the component types used in the `Shape` layer.  See `PickleReader` and `PickleWriter`.\n3. The `Shape` layer.  This is responsible for turning Primitives, Structs, Choices and Collections\n   into component types.\n\nIt's the circle of data:\n```\n   Source   =\u003e     format    =\u003e  shape =\u003e memory =\u003e  shape  =\u003e   format    =\u003e   Destination        \n\n[PickleData] =\u003e PickleReader =\u003e Builder[T] =\u003e T =\u003e Writer[T] =\u003e PickleWriter =\u003e [PickleData]\n```\n\nThis, hopefully, means we can reuse a lot of logic betwen various formats with light loss to efficiency.\n\n*Note:  This library is not measuring performance yet.*\n\n### Shape layer\nThe Shape layer is responsible for extracting Scala types into known shapes that can be used for\nserialization.  These shapes, current, are `Collection`, `Structure` and `Primitive`.   Custom\nshapes can be created in terms of these three shapes.\n\nThe Shape layer defines these three classes:\n- `sauerkraut.core.Writer[T]`:\n  Can translate a value into write* calls of Primitive, Structure or Collection.\n- `sauerkraut.core.Builder[T]`:  \n  Can accept an incomiing stream of collections/structures/primitives and build a value of T from them.\n- `sauerkraut.core.Buildable[T]`:\n  Can provide a `Builder[T]` when asked.\n\n### Format layer\nThe format layer is responsible for mapping sauerkraut shapes (`Collection`, `Structure`, `Primitive`, `Choice`) into\nthe underlying format.  Not all shapes in sauerkraut will map exactly to underlying formats, and so each\nformat may need to adjust/tweak incoming data as appropriate.\n\nThe format layer has these primary classes:\n\n- `sauerkraut.format.PickleReader`:  Can load data and push it into a Builder of type T\n- `sauerkraut.format.PickleWriter`:  Accepts pushed structures/collections/primitives and places it into a Pickle\n\n### Source Layer\nThe `source` layer is allowed to be any type that a format wishes to support.   Inputs and outputs are\nprovided to the API via these two classes:\n\n- `sauerkraut.format.PickleReaderSupport[Input, Format]`:\n  A given of this instance will allow the `PickleReader` to be constructed from a type of input.\n- `sauerkraut.format.PickleWriterSupport[Output,Format]`:\n  A given of this instance will allow `PickleWriter` to be constructed from a type of output.\n\nThis layer is designed to support any type of input and output, not just an in-memory store (like a Json Ast) or\na streaming input.  Formats can define what types of input/output (or execution environment) they allow.\n\n## Writing a new format.\n\nNew formats are expected to provide the \"format\" + \"source\" layer implementations they require.\n\nTODO - a bit more here.\n\n\n# Differences from Scala Pickling\n\nThere are a few major differences from the old [scala pickling project](http://github.com/scala/pickling).\n\n- The core library is built for 100% static code generation.   While we think that dynamic (i.e. runtime-reflection-based)\n  pickling could be built using this library, it is a non-goal.\n  - Users are expected to rely on typeclass derivation to generate Reader/Writers, rather than using macros\n  - The supported types that can be pickled are limited to the same supported by typeclass derivation or that\n    can have hand-written `Writer[_]`/`Builder[_]` instances.\n- Readers are no longer driven by the Scala type.  Instead we use a new `Buildable[A]`/`Builder[A}` design\n  to allow each `PickleReader` to push value into a `Builder[A]` that will then construct the scala class.\n- There have been no runtime performance optimisations around codegen.   Those will come as we test the\n  limits of Scala 3 / Dotty.\n- Format implementations are separate libraries.\n- The `PickleWriter` contract has been split into several types to avoid misuse.  This places a heavier amount\n  of lambdas in play, but may be offsite with optimisations in modern versions of Scala/JVM.\n- The name is more German.\n\n\n# Benchmarking\n\nBenchmarking is still being built-out, and is pending the final design on Choice/Sum-Types within the Format/Shape layer.\n\nYou can see benchmark results via: ` benchmarks/jmh:run -rf csv`.\n\nLatest status/analysis can be found in the [benchmarks directory](benchmarks/latest-results.md).\n\n## Benchmarking TODOs\n\n- [X] Basic comparison of all formats\n- [X] Size-of-Pickle measurement\n- [ ] Well-thought out dataset for reading/writing\n- [X] Isolated read vs. write testing\n- [ ] Comparison against other frameworks.\n  - [X] Protos vs. protocol buffer java implementation\n  - [ ] Json Reading vs. raw JAWN to AST (measure overhead)\n  - [ ] Jackson\n  - [X] Kryo\n  - [ ] Thrift\n  - [ ] Circe\n  - [X] uPickle\n- [ ] Automatic well-formatted graph dump in Markdown of results.\n\n\n# Thanks\n\nThanks to everyone who contributed to the original pickling library for inspiration, with a few callouts.\n\n- Heather Miller + Philipp Haller for the original idea, innovation and motivation for Scala.\n- Havoc Pennington + Eugene Yokota for helping define what's important when pickling a protocol and evolving that protocol.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsuereth%2Fsauerkraut","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsuereth%2Fsauerkraut","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsuereth%2Fsauerkraut/lists"}