{"id":13753124,"url":"https://github.com/maximveksler/awesome-serialization","last_synced_at":"2025-05-09T20:34:48.043Z","repository":{"id":45253742,"uuid":"163650590","full_name":"maximveksler/awesome-serialization","owner":"maximveksler","description":"Data formats useful for API, Big Data, ML, Graph \u0026 co","archived":false,"fork":false,"pushed_at":"2024-12-29T11:06:25.000Z","size":50,"stargazers_count":43,"open_issues_count":0,"forks_count":4,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-26T02:08:01.304Z","etag":null,"topics":["awesome-list","big-data","data-science","serialization-formats"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maximveksler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"contributing.md","funding":null,"license":"LICENSE","code_of_conduct":"code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-31T07:59:20.000Z","updated_at":"2025-02-25T01:03:40.000Z","dependencies_parsed_at":"2023-02-12T18:15:28.082Z","dependency_job_id":null,"html_url":"https://github.com/maximveksler/awesome-serialization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximveksler%2Fawesome-serialization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximveksler%2Fawesome-serialization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximveksler%2Fawesome-serialization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maximveksler%2Fawesome-serialization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maximveksler","download_url":"https://codeload.github.com/maximveksler/awesome-serialization/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253133137,"owners_count":21859110,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome-list","big-data","data-science","serialization-formats"],"created_at":"2024-08-03T09:01:16.633Z","updated_at":"2025-05-09T20:34:48.028Z","avatar_url":"https://github.com/maximveksler.png","language":null,"funding_links":[],"categories":["Others","Related Awesome Lists","Other Lists"],"sub_categories":["Research Papers","TeX Lists"],"readme":"# awesome-serialization [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)\n\n\u003e So many formats, so much wow. *See [wesm](https://github.com/18F/data-act-pilot/issues/161#issuecomment-139812420) comment before making up your mind.*\n\n## Contents\n\n- [API](#api)\n- [RPC](#rpc)\n- [Big Data](#big-data)\n- [Scientific](#scientific)\n- [Machine Learning](#machine-learning)\n- [Graph](#graph)\n- [Object](#object)\n- [Workflow](#workflow)\n- [Programming](#programming)\n- [Streaming](#streaming)\n- [Security-Focused](#security-focused)\n- [Academic](#academic)\n\n### **API**\n\nSerialization suitable for API RPC networked services.\n\n- [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) - Comma Separated Values. Textual.\n- [JSON](https://www.json.org) - Lightweight document data-interchange format. Textual.\n- [JSONL](https://jsonlines.org) - Schemeless \"multiple JSON documents in 1 file\" container data format. Textual.\n- [JSON5](https://json5.org/) - JSON with added support for comments and relaxed syntax. Textual.\n- [Thrift](http://thrift.apache.org) - Scalable code generation, schema evolution binary format. Binary.\n- [Protocol Buffers](https://github.com/protocolbuffers/protobuf) - Google's data interchange format. Binary.\n- [Message Pack](https://msgpack.org) - Efficient JSON-like binary serialization format. Binary.\n- [bson](http://bsonspec.org) - Binary schemeless JSON encoding. Binary.\n- [XML](https://www.w3.org/XML/) - Extensible Markup Language. Genuinely Horrible. Textual.\n- [Plist](https://en.wikipedia.org/wiki/Property_list) - Property List representation. Apple. Textual.\n- [YAML](https://yaml.org) - Indentation-based data serialization standard. Textual.\n- [TOML](https://github.com/toml-lang/toml) - Tom's Obvious, Minimal Language. Textual.\n- [CBOR](https://cbor.io) - Concise Binary Object Representation. Schema-free. Binary.\n- [ION](https://amzn.github.io/ion-docs/) - Amazon's advanced JSON-compatible serialization. Textual/Binary.\n\n### **RPC**\n\n- [gRPC](https://grpc.io) - A high-performance, open source universal RPC framework. Binary, ISO Layer 7.\n- [RSocket](https://rsocket.io) - Application protocol providing Reactive Streams semantics. Binary, ISO Layer 5 (or 6).\n- [Cap’n Proto](https://capnproto.org/) - High-performance, schema-based data interchange format. Binary.\n- [FlatBuffers](https://google.github.io/flatbuffers/) - Suitable for zero-copy deserialization. Binary.\n\n### **Big Data**\n\nSerialization suitable for big data at rest systems, from Hadoop family of solutions.\n\n- [Parquet](https://parquet.apache.org) - Columnar storage for Hadoop workloads. Binary.\n- [FlatBuffers](https://google.github.io/flatbuffers/) - Protocol Buffers suitable for larger datasets. Binary.\n- [ORC](https://orc.apache.org) - Columnar storage for Hadoop workloads. Binary.\n- [Avro](https://avro.apache.org) - Scheme embedded, dynamic rich data structures. Textual/Binary.\n- [Ion](https://amzn.github.io/ion-docs/) - Row storage with skip scan parsing. Structured, schema embedded. Amazon. Textual/Binary.\n- [Arrow](https://arrow.apache.org) - Cross-language columnar data format optimized for analytics workloads. Binary.\n- [Delta Lake](https://delta.io/) - Transactional storage layer for big data workflows. Binary.\n- [Iceberg](https://iceberg.apache.org/) - Open table format for large datasets. Binary.\n\n### **Scientific**\n\nLarge-scale sparse arrays used in physical, mathematics, and statistics research.\n\n- [HDF5®](https://www.hdfgroup.org) - n-dimensional datasets, complex objects, with schema. Efficient I/O. Binary.\n- [npy](https://numpy.org/devdocs/reference/generated/numpy.lib.format.html) - Numpy arrays, cell sparse metadata. Binary.\n- [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) - Self-describing, machine-independent data format for scientific data. Binary.\n- [Zarr](https://zarr.readthedocs.io/) - Scalable storage of n-dimensional arrays. Binary.\n- [ASDF](https://asdf-standard.readthedocs.io/) - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.\n\n### **Machine Learning**\n\nSerialization of deep learning networks and weights.\n\n- [SavedModel](https://www.tensorflow.org/guide/saved_model) - TensorFlow package, weights, graph, executable code. Binary.\n- [CoreML](https://developer.apple.com/documentation/coreml) - Apple's on-device ML model format. Binary.\n- [ONNX](https://onnx.ai) - Open Neural Network Exchange. Interoperability focused. Binary.\n- [MLIR](https://mlir.llvm.org/) - Intermediate representation for machine learning computations. Textual/Binary.\n- [TorchScript](https://pytorch.org/docs/stable/jit.html) - Serialization for PyTorch models. Binary.\n\n\u003c!-- trunk-ignore(markdownlint/MD001) --\u003e\n###### Deprecated in Machine Learning\n\n- [GraphDef](https://www.tensorflow.org/guide/extend/model_files) - TensorFlow graphs. Binary.\n- [PMML](https://dmg.org/pmml/v4-4/GeneralStructure.html) - Predictive Model Markup Language for exchanging ML models.\n\n### **Graph**\n\nSerialization formats for representing graph-oriented data structures.\n\n- [json-ld](https://json-ld.org) - JSON for Linking Data. Textual.\n- [Turtle](https://www.w3.org/TR/turtle/) - Terse RDF Triple Language. Textual.\n- [GraphML](http://graphml.graphdrawing.org/) - XML-based graph serialization format. Textual.\n- [DOT](https://www.graphviz.org/doc/info/lang.html) - Graph description language, developed as a part of the Graphviz project. Textual.\n- [ParquetGraph](https://github.com/graphfoundry/parquet-graph) - Integration of Parquet with graph data structures. Binary.\n- [GraphSON](https://tinkerpop.apache.org/docs/current/dev/io/#graphson) - JSON-based graph serialization. Textual.\n\n\u003c!-- trunk-ignore(markdownlint/MD001) --\u003e\n###### Deprecated in Graph\n\n- [GraphML](http://graphml.graphdrawing.org/) - XML-based graph serialization format. Textual.- [GML](https://en.wikipedia.org/wiki/Graph_Modelling_Language) - Graph Modeling Language. Hierarchical ASCII-based. Textual.\n\n### **Workflow**\n\n- [common-workflow-language](https://github.com/common-workflow-language) - Specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments.\n- [Apache Airflow DAGs](https://airflow.apache.org) - Python-based Directed Acyclic Graphs for workflows.\n- [WDL](https://openwdl.org/) - Workflow Description Language for genomics and scientific workflows.\n- [Relational Algebra and Datalog for Graphs](https://www.coursera.org/lecture/data-manipulation/relational-algebra-and-datalog-for-graphs-U8zVV) - Coursera course on graph data manipulation.\n- [Cromwell](https://cromwell.readthedocs.io/) - Scientific workflow management, compatible with WDL and CWL.\n- [Nextflow](https://www.nextflow.io/) - Scalable and reproducible scientific workflows.\n\n### **Programming**\n\nLanguage-native serialization formats for in-transit data (aka memory-based \"live\" objects).\n\n#### **JavaScript**\n\n- [BSON.js](https://github.com/mongodb/js-bson) - BSON serializer. Binary.\n- [avsc](https://github.com/mtth/avsc) - JavaScript implementation of Apache Avro. Textual.\n\n#### **Dart**\n\n- [Dart Object Serialization](https://github.com/flutter/packages/tree/main/packages/standard_message_codec) - RAM to Disk serialization. Dart-specific. Binary.\n\n#### **Python**\n\n- [pickle](https://docs.python.org/3/library/pickle.html) - RAM to Disk serialization. Binary.\n- [msgpack-python](https://github.com/msgpack/msgpack-python) - MessagePack serializer implementation for Python.\n- [srsly](https://github.com/explosion/srsly) - Modern high-performance serialization utilities for Python.\n\n#### **Swift**\n\n- [MessagePack.swift](https://github.com/a2/MessagePack.swift) - Swift MessagePack Serializer.\n\n#### **Java**\n\n- [Java Object Serialization](https://docs.oracle.com/javase/8/docs/technotes/guides/serialization/index.html) - RAM to Disk serialization. Binary.\n\n#### **Rust**\n\n- [Serde](https://serde.rs/) - Rust's serialization framework for multiple formats like JSON, CBOR, and MessagePack.\n- [bincode](https://github.com/bincode-org/bincode) - High-performance binary serialization for Rust.\n\n#### **Go**\n\n- [GOB](https://pkg.go.dev/encoding/gob) - Go's built-in serialization format for arbitrary data structures. Binary.\n\n### **Streaming**\n\nSerialization formats optimized for real-time streaming data.\n\n- [Kafka Streams](https://kafka.apache.org/documentation/streams/) - Real-time stream processing framework with built-in serialization.\n- [Protobuf-Lite](https://github.com/protocolbuffers/protobuf-javalite) - Lightweight Protocol Buffers for constrained environments.\n\n### **Security-Focused**\n\nSerialization formats designed with security and robustness in mind.\n\n- [safetensors](https://github.com/huggingface/safetensors) - Safe serialization of tensors for machine learning. Binary.\n- [Sealed Object Serialization](https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/encrypt) - Encrypted serialization for web data. Textual/Binary.\n\n### **Academic**\n\nResearch papers discussing types, category theory, benchmarks \u0026 co.\n\n- [Type theory](https://en.wikipedia.org/wiki/Type_theory) - Studies types, which informally are attributes that objects can possess.\n- [Category theory](https://en.wikipedia.org/wiki/Category_theory) - General theory of functions. Axiomatic foundation for mathematics, as an alternative to set theory.\n- [Graph Compression Techniques](https://arxiv.org/abs/2006.03684) - Research on optimizing graph serialization.\n- [Efficient Serialization in Distributed Systems](https://arxiv.org/abs/2201.02357) - Study of efficient serialization techniques for scalability.\n\n## Contribute\n\nContributions welcome! Read the [contribution guidelines](contributing.md) first.\n\n## License\n\n[![CC0](http://mirrors.creativecommons.org/presskit/buttons/88x31/svg/cc-zero.svg)](http://creativecommons.org/publicdomain/zero/1.0)\n\nTo the extent possible under law, Maxim Veksler has waived all copyright and\nrelated or neighboring rights to this work.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaximveksler%2Fawesome-serialization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaximveksler%2Fawesome-serialization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaximveksler%2Fawesome-serialization/lists"}