{"id":15296132,"url":"https://github.com/arangodb/velocypack","last_synced_at":"2025-05-15T19:05:46.656Z","repository":{"id":2149276,"uuid":"44518776","full_name":"arangodb/velocypack","owner":"arangodb","description":"A fast and compact format for serialization and storage","archived":false,"fork":false,"pushed_at":"2025-02-18T09:56:30.000Z","size":6462,"stargazers_count":424,"open_issues_count":31,"forks_count":43,"subscribers_count":62,"default_branch":"main","last_synced_at":"2025-05-15T19:05:05.825Z","etag":null,"topics":["arangodb","cplusplus","json","performance","serialization","velocypack","vpack"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arangodb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-10-19T07:42:50.000Z","updated_at":"2025-03-16T14:52:55.000Z","dependencies_parsed_at":"2023-02-14T00:16:28.216Z","dependency_job_id":"e4bfe312-d224-4433-8294-35c4972aa887","html_url":"https://github.com/arangodb/velocypack","commit_stats":{"total_commits":1181,"total_committers":31,"mean_commits":"38.096774193548384","dds":"0.41320914479254867","last_synced_commit":"373d0e2c5b930886b99a8451c9687f91684968f7"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arangodb%2Fvelocypack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arangodb%2Fvelocypack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arangodb%2Fvelocypack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arangodb%2Fvelocypack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arangodb","download_url":"https://codeload.github.com/arangodb/velocypack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254404357,"owners_count":22065641,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["arangodb","cplusplus","json","performance","serialization","velocypack","vpack"],"created_at":"2024-09-30T18:09:30.609Z","updated_at":"2025-05-15T19:05:46.635Z","avatar_url":"https://github.com/arangodb.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"VelocyPack (VPack) - a fast and compact format for serialization and storage\n============================================================================\n\nGitHub Action: ![Build](https://github.com/arangodb/velocypack/actions/workflows/github-ci.yml/badge.svg) Coveralls: [![Coverage Status](https://coveralls.io/repos/arangodb/velocypack/badge.svg?branch=main\u0026service=github)](https://coveralls.io/github/arangodb/velocypack?branch=main)\n\nMotivation\n----------\n\nThese days, JSON (JavaScript Object Notation, see\n[ECMA-404](http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf))\nis used in many cases where data has to be exchanged.\nLots of protocols between different services use it, databases store\nJSON (document stores naturally, but others increasingly as well). It\nis popular, because it is simple, human-readable, and yet surprisingly\nversatile, despite its limitations.\n\nAt the same time there is a plethora of alternatives ranging from XML\nover Universal Binary JSON, MongoDB's BSON, MessagePack, BJSON (binary\nJSON), Apache Thrift till Google's protocol buffers and ArangoDB's\nshaped JSON.\n\nWhen looking into this, we were surprised to find that none of these\nformats manages to combine compactness, platform independence, fast\naccess to sub-objects and rapid conversion from and to JSON.\n\nWe have invented VPack because we need a binary format that\n\n  - is self-contained and schemaless \n  - is compact\n  - is largely platform independent (see [Portability](VelocyPack.md#Portability))\n  - covers all of JSON plus dates, integers, binary data and arbitrary\n    precision numbers\n  - can be used in a database kernel to access sub-documents for\n    example for indexes, so it must be possible to access sub-documents\n    (array and object members) efficiently\n  - can be transferred to JSON and from JSON rapidly\n  - avoids too many memory allocations\n  - gives flexibility to assemble objects, such that sub-objects reside\n    in the database in an unchanged way\n  - allows to use an external table for frequently used attribute names\n  - quickly allows to read off the type and length of a given object\n    from its first byte(s)\n\nAll this gives us the possibility to use *the same byte sequence of\ndata* for **transport**, **storage** and (read-only) **work**. Using a\nsingle data format not only eliminates a lot of conversions but can \nalso reduce runtime memory usage, as data does only need a single \nin-memory representation.\n\nThe other popular formats we looked at have all some deficiency with\nrespect to the above list. To name but a few:\n\n  - JSON itself lacks some data types (dates and binary data) and does\n    not provide quick sub-value access without parsing. Parsing JSON is\n    also quite a challenge performance-wise\n  - XML is not compact and is not good with binary data, it also lacks\n    quick sub-value access\n  - BSON gets quite a lot right with respect to data types, but is \n    seriously lacking w.r.t. sub-value access. Furthermore, it is not\n    very compact and quite wasteful space-wise when storing array values\n  - Apache Thrift and Google's Protocol Buffers are not schemaless and \n    self-contained. Their transport format is a serialization that is\n    not good for rapid sub-value access\n  - MessagePack is probably the closest to our shopping list. It has\n    has decent data types and is quite compact. However, we found that \n    one can do better in terms of compactness for some cases. More\n    important for us, MessagePack provides no quick sub-value access\n  - Our own shaped JSON (used in ArangoDB as internal storage format)\n    has very quick sub-value access, but the shape data is kept outside\n    the actual data, so the shaped values are not self-contained.\n    Furthermore, we have run into scalability issues on multi-core\n    because of the shared data structures used for interpretation of\n    the values\n\nAny new data format must be backed by C++ classes to allow\n\n  - easy and fast parsing from JSON\n  - easy and convenient buildup without too many memory allocations\n  - fast access to data and its sub-objects (for arrays and objects)\n  - flexible memory management\n  - fast dumping to JSON\n\nThe VelocyPack format is an attempt to achieve all this.\n\nThis repository contains a C++ library for building, manipulating and\nserializing VPack data. It is the *reference implementation for the \nVelocyPack format*. The library is written in C++20 so it should compile \non many up-to-date systems.\n\nThe VelocyPack format and library are used extensively in the \n[ArangoDB database](https://www.arangodb.com/).\n\n\nSpecification\n-------------\n\nSee the file [VelocyPack.md](VelocyPack.md) for a detailed description of\nthe VPack format.\n\n\nPerformance\n-----------\n\nSee the file [Performance.md](Performance.md) for a thorough comparison\nto other formats like JSON itself, MessagePack and BSON. We look at file\nsizes as well as parsing and conversion performance.\n\n\nBuilding the VPack library\n--------------------------\n\nThe VPack library can be built on Linux, MacOS and Windows. It will likely\ncompile and work on other platforms for which a recent version of *cmake* and\na working C++20-enabled compiler are available.\n\nSee the file [Install.md](Install.md) for compilation and installation\ninstructions.\n\n\nUsing the VPack library\n-----------------------\n\nPlease consult the file [examples/API.md](examples/API.md) for usage examples, \nand the file [examples/Embedding.md](examples/Embedding.md) for information\nabout how to embed the library into client applications.\n\n\nTesting and validating with fuzzer\n----------------------------------\n\nThe fuzzer tool can be used to generate random VPack or JSON structures and\nvalidate them. The tool can be run with multiple iterations, parallelism, and \na seed can be provided for the random generation.\nPlease consult the file [tools/README.md](tools/README.md) for usage \ninformation.\n\n\nContributing\n------------\n\nWe welcome bug fixes and patches from 3rd party contributors!\n\nPlease follow the guidelines in [CONTRIBUTING.md](.github/CONTRIBUTING.md)\nif you want to contribute to VelocyPack. Have a look for the tag _help wanted_\nin the issue tracker!\n\nWe also provide a golang version of VPack in the \n[go-velocypack repository](https://github.com/arangodb/go-velocypack) and a\nJava version in the [java-velocypack](https://github.com/arangodb/java-velocypack).\n\nAdditionally, there is a third party VPack implementation for \n[PHP](https://github.com/martin-schilling/php-velocypack).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farangodb%2Fvelocypack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farangodb%2Fvelocypack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farangodb%2Fvelocypack/lists"}