awesome-serialization

Curated list of data serialization formats — API, ML, Agentic AI, Big Data, Configuration, and beyond
https://github.com/maximveksler/awesome-serialization

Last synced: 1 day ago
JSON representation

Academic
- Java
  - Type theory - studies types, which informally are attributes that objects can possess.
  - Category theory - General theory of functions. Axiomatic foundation for mathematics, as an alternative to set theory.
Language specific
- Dart
  - Dart Object Serialization - Ram to Disk serialization. Dart specific. Binary.
- Java
  - Java Object Serialization - Ram to Disk serialization. Binary.
- Python
  - pickle - Ram to Disk serialization. Binary.
  - msgpack-python - MessagePack serializer implementation for Python.
  - srsly - Modern high-performance serialization utilities for Python.
- Swift
  - MessagePack.swift - Swift MessagePack Serializer.
Machine Learning
- safetensors - Simple, safe way to store and distribute tensors
Scientific
- HDF5® - n-dimensional datasets, complex objects, with schema. Efficient I/O. Binary.
Uncategorized
- Uncategorized
  - CSV - Comma Separated Values. Textual.
  - XML - Extensible Markup Language. Genuinely Horrible. Textual.
  - Plist - Property List representation. Apple. Textual.
  - npy - Numpy arrays, cell sparse metadata. Binary.
  - SavedModel - TensorFlow package, weights, graph, executable code. Binary.
  - Turtle - Terse RDF Triple Language. Textual.
  - Protocol Buffers - Google's data interchange format. Binary.
  - TOML - Tom's Obvious, Minimal Language. Textual.
  - Ion - Row storage with skip scan parsing. Structured, schema embedded. Amazon. Textual/Binary.
  - JSON5 - JSON with added support for comments and relaxed syntax. Textual.
  - Cap’n Proto - High-performance, schema-based data interchange format. Binary.
  - Delta Lake - Transactional storage layer for big data workflows. Binary.
  - Iceberg - Open table format for large datasets. Binary.
  - NetCDF - Self-describing, machine-independent data format for scientific data. Binary.
  - ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
  - CoreML - Apple's on-device ML model format. Binary.
  - MLIR - Intermediate representation for machine learning computations. Textual/Binary.
  - TorchScript - Serialization for PyTorch models. Binary.
  - PMML - Predictive Model Markup Language for exchanging ML models.
  - ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
  - ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
  - ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
  - GraphML - XML-based graph serialization format. Textual.
  - DOT - Graph description language, developed as a part of the Graphviz project. Textual.
  - GraphSON - JSON-based graph serialization. Textual.
  - WDL - Workflow Description Language for genomics and scientific workflows.
  - Cromwell - Scientific workflow management, compatible with WDL and CWL.
  - Nextflow - Scalable and reproducible scientific workflows.
  - BSON.js - BSON serializer. Binary.
  - avsc - JavaScript implementation of Apache Avro. Textual.
  - Serde - Rust's serialization framework for multiple formats like JSON, CBOR, and MessagePack.
  - bincode - High-performance binary serialization for Rust.
  - GOB - Go's built-in serialization format for arbitrary data structures. Binary.
  - Kafka Streams - Real-time stream processing framework with built-in serialization.
  - Sealed Object Serialization - Encrypted serialization for web data. Textual/Binary.
  - Graph Compression Techniques - Research on optimizing graph serialization.
  - Efficient Serialization in Distributed Systems - Study of efficient serialization techniques for scalability.
  - NetCDF - Self-describing, machine-independent data format for scientific data. Binary.
  - ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
  - Smile - Binary JSON format from FasterXML/Jackson. Used in Elasticsearch. Binary.
  - JSON Schema - Schema vocabulary for annotating and validating JSON documents. Textual.
  - Connect - Modern RPC framework compatible with gRPC, with HTTP/1.1, JSON, and browser support. Binary/Textual.
  - CloudEvents - CNCF specification for describing event data in a common way. Textual/Binary.
  - AsyncAPI - OpenAPI equivalent for event-driven and message-driven architectures. Textual.
  - TOON - Token-Oriented Object Notation. Compact, schema-aware JSON alternative achieving 30–60% token savings for LLM prompts. Textual.
  - Markdown - Lightweight markup widely used as the native "language" of LLM input/output. Highly token-efficient vs HTML/XML. Textual.
  - MCP - Model Context Protocol. Anthropic's open standard for connecting LLM agents to tools and data sources. JSON-RPC based. Textual.
  - A2A - Agent2Agent Protocol. Google's open protocol for agent-to-agent communication and interoperability. JSON based. Textual.
  - BAML - Boundary AI Markup Language. Domain-specific language for defining LLM function signatures with type-safe structured output. Textual.
  - MLX format - Apple's ML framework format, safetensors-based. Optimized for Apple Silicon. Binary.
  - CUE - Constraint-based configuration language with validation, templating, and JSON superset. Google. Textual.
  - Pkl - Programmable, typed configuration language. Apple. Textual.
  - KCL - Kusion Configuration Language for cloud-native config and policy. Textual.
  - PASETO - Platform-Agnostic Security Tokens. Secure alternative to JWT. Textual.
  - COSE - CBOR Object Signing and Encryption. Used in WebAuthn/FIDO2. Binary.
  - GML - Graph Modeling Language. Hierarchical ASCII-based. Textual.
  - Protobuf-Lite - Lightweight Protocol Buffers for constrained environments.
  - JSON - Lightweight document data-interchange format. Textual.
  - JSONL - Schemeless "multiple JSON documents in 1 file" container data format. Textual.
  - Thrift - Scalable code generation, schema evolution binary format. Binary.
  - Message Pack - Efficient JSON-like binary serialization format. Binary.
  - bson - Binary schemeless JSON encoding. Binary.
  - CBOR - Concise Binary Object Representation. Schema-free. Binary.
  - RSocket - Application protocol providing Reactive Streams semantics. Binary, ISO Layer 5 (or 6).
  - GraphDef - TensorFlow graphs. Binary.
  - ORC - Columnar storage for Hadoop workloads. Binary.
  - Avro - Scheme embedded, dynamic rich data structures. Textual/Binary.
  - Arrow - Cross-language columnar data format optimized for analytics workloads. Binary.
  - HDF5® - n-dimensional datasets, complex objects, with schema. Efficient I/O. Binary.
  - Zarr - Scalable storage of n-dimensional arrays. Binary.
  - json-ld - JSON for Linking Data. Textual.
  - ParquetGraph - Integration of Parquet with graph data structures. Binary.
  - Apache Airflow DAGs - Python-based Directed Acyclic Graphs for workflows.
  - Relational Algebra and Datalog for Graphs - Coursera course on graph data manipulation.
  - Cromwell - Scientific workflow management, compatible with WDL and CWL.
Workflow
- common-workflow-language - Specification for describing analysis workflows and tools in a way that makes them portable and scalable across a variety of software and hardware environments.
- Relational Algebra and Datalog for Graphs - Coursera course on graph data manipulation.

Programming Languages

TypeScript 3 Python 3 JavaScript 2 Rust 2 C++ 2 Swift 1

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

awesome-serialization

Academic

Java

Language specific

Dart

Java

Python

Swift

Machine Learning

Scientific

Uncategorized

Uncategorized

Workflow