Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

awesome-serialization

Data formats useful for API, Big Data, ML, Graph & co
https://github.com/maximveksler/awesome-serialization

Last synced: 2 days ago
JSON representation

  • Uncategorized

    • Uncategorized

      • ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
      • Plist - Property List representation. Apple. Textual.
      • npy - Numpy arrays, cell sparse metadata. Binary.
      • SavedModel - TensorFlow package, weights, graph, executable code. Binary.
      • Turtle - Terse RDF Triple Language. Textual.
      • TOML - Tom's Obvious, Minimal Language. Textual.
      • JSON - Lightweight document data-interchange format. Textual.
      • JSONL - Schemeless "multiple JSON documents in 1 file" container data format. Textual.
      • Thrift - Scalable code generation, schema evolution binary format. Binary.
      • Message Pack - Efficient JSON-like binary serialization format. Binary.
      • bson - Binary schemeless JSON encoding. Binary.
      • CBOR - Concise Binary Object Representation. Schema-free. Binary.
      • RSocket - Application protocol providing Reactive Streams semantics. Binary, ISO Layer 5 (or 6).
      • Arrow - Cross-language columnar data format optimized for analytics workloads. Binary.
      • HDF5® - n-dimensional datasets, complex objects, with schema. Efficient I/O. Binary.
      • Zarr - Scalable storage of n-dimensional arrays. Binary.
      • ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
      • GraphDef - TensorFlow graphs. Binary.
      • json-ld - JSON for Linking Data. Textual.
      • ParquetGraph - Integration of Parquet with graph data structures. Binary.
      • Relational Algebra and Datalog for Graphs - Coursera course on graph data manipulation.
      • Cromwell - Scientific workflow management, compatible with WDL and CWL.
      • Protobuf-Lite - Lightweight Protocol Buffers for constrained environments.
      • NetCDF - Self-describing, machine-independent data format for scientific data. Binary.
      • CSV - Comma Separated Values. Textual.
      • JSON5 - JSON with added support for comments and relaxed syntax. Textual.
      • Protocol Buffers - Google's data interchange format. Binary.
      • XML - Extensible Markup Language. Genuinely Horrible. Textual.
      • Cap’n Proto - High-performance, schema-based data interchange format. Binary.
      • MLIR - Intermediate representation for machine learning computations. Textual/Binary.
      • TorchScript - Serialization for PyTorch models. Binary.
      • Delta Lake - Transactional storage layer for big data workflows. Binary.
      • Iceberg - Open table format for large datasets. Binary.
      • NetCDF - Self-describing, machine-independent data format for scientific data. Binary.
      • ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
      • CoreML - Apple's on-device ML model format. Binary.
      • PMML - Predictive Model Markup Language for exchanging ML models.
      • ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
      • ASDF - Advanced Scientific Data Format for astronomy and beyond. Binary/Textual.
      • Ion - Row storage with skip scan parsing. Structured, schema embedded. Amazon. Textual/Binary.
      • GraphML - XML-based graph serialization format. Textual.
      • DOT - Graph description language, developed as a part of the Graphviz project. Textual.
      • GraphSON - JSON-based graph serialization. Textual.
      • WDL - Workflow Description Language for genomics and scientific workflows.
      • Cromwell - Scientific workflow management, compatible with WDL and CWL.
      • Nextflow - Scalable and reproducible scientific workflows.
      • BSON.js - BSON serializer. Binary.
      • avsc - JavaScript implementation of Apache Avro. Textual.
      • Serde - Rust's serialization framework for multiple formats like JSON, CBOR, and MessagePack.
      • bincode - High-performance binary serialization for Rust.
      • GOB - Go's built-in serialization format for arbitrary data structures. Binary.
      • Kafka Streams - Real-time stream processing framework with built-in serialization.
      • Sealed Object Serialization - Encrypted serialization for web data. Textual/Binary.
      • Graph Compression Techniques - Research on optimizing graph serialization.
      • Efficient Serialization in Distributed Systems - Study of efficient serialization techniques for scalability.

  • Scientific

    • HDF5® - n-dimensional datasets, complex objects, with schema. Efficient I/O. Binary.

  • Workflow

  • Language specific

  • Academic

    • Java

      • Type theory - studies types, which informally are attributes that objects can possess.
      • Category theory - General theory of functions. Axiomatic foundation for mathematics, as an alternative to set theory.

  • Machine Learning

    • safetensors - Simple, safe way to store and distribute tensors
Programming Languages
Python 3 JavaScript 2 Swift 1 TypeScript 1 Rust 1 C++ 1
Categories
Uncategorized 55 Language specific 6 Academic 2 Workflow 2 Machine Learning 1 License 1 Scientific 1
Sub Categories
Uncategorized 55 Python 3 Java 3 Swift 1 Dart 1
Keywords
serialization 4 encoding 2 msgpack 2 python 2 marshalling 1 protobuf 1 protobuf-runtime 1 protoc 1 protocol-buffers 1 protocol-compiler 1 rpc 1 binary 1 rust 1 bson 1 bson-library 1 mongodb 1 node-js 1 nodejs 1 json 1 pickle 1 python-2 1 python-3 1 ujson 1 yaml 1 avro 1 big-data 1 binary-format 1 javascript 1 schema-evolution 1 typescript 1 messagepack 1 swift 1