Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/status-im/nim-json-serialization
Flexible JSON serialization not relying on run-time type information
https://github.com/status-im/nim-json-serialization
Last synced: about 2 months ago
JSON representation
Flexible JSON serialization not relying on run-time type information
- Host: GitHub
- URL: https://github.com/status-im/nim-json-serialization
- Owner: status-im
- License: apache-2.0
- Created: 2018-11-11T09:46:35.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2024-07-27T00:27:13.000Z (about 2 months ago)
- Last Synced: 2024-07-27T09:51:25.535Z (about 2 months ago)
- Language: Nim
- Size: 256 KB
- Stars: 46
- Watchers: 20
- Forks: 9
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHEv2
Awesome Lists containing this project
- awesome-nim - json-serialization - Flexible JSON serialization not relying on run-time type information. (Data / Serialization)
README
# nim-json-serialization
[](https://opensource.org/licenses/Apache-2.0)
[](https://opensource.org/licenses/MIT)
Flexible JSON serialization does not rely on run-time type information.
## Overview
nim-json-serialization offers rich features on top of [nim-serialization](https://github.com/status-im/nim-serialization)
framework. The following is available but not an exhaustive list of features:
- Decode into Nim data types efficiently without an intermediate token.
- Able to parse full spec of JSON including the notorious JSON number.
- Support stdlib/JsonNode out of the box.
- While stdlib/JsonNode does not support the full spec of the Json number, we offer an alternative `JsonValueRef`.
- Skipping Json value is an efficient process, no token is generated at all and at the same time, the grammar is checked.
- Skipping is also free from custom serializer interference.
- An entire Json value can be parsed into a valid Json document string. This string document can be parsed again without losing any information.
- Custom serialization is easy and safe to implement with the help of many built-in parsers.
- Nonstandard features are put behind flags. You can choose which features to switch on or off.
- Because the intended usage of this library will be in a security-demanding application, we make sure malicious inputs will not crash
this library through fuzz tests.
- The user also can tweak certain limits of the lexer/parser behavior using the configuration object.
- `createJsonFlavor` is a powerful way to prevent cross contamination between different subsystem using different custom serializar on the same type.
## Spec compliance
nim-json-serialization implements [RFC8259](https://datatracker.ietf.org/doc/html/rfc8259)
JSON spec and pass these test suites:
- [JSONTestSuite](https://github.com/nst/JSONTestSuite)
## Switchable features
Many of these switchable features are widely used features in various projects but are not standard JSON features.
But you can access them using the flags:
- **allowUnknownFields[=off]**: enable unknown fields to be skipped instead of throwing an error.
- **requireAllFields[=off]**: if one of the required fields is missing, the serializer will throw an error.
- **escapeHex[=off]**: JSON doesn't support `\xHH` escape sequence, but it is a common thing in many languages.
- **relaxedEscape[=off]**: only '0x00'..'0x1F' can be prepended by escape char `\\`, turn this on and you can escape any char.
- **portableInt[=off]**: set the limit of integer to `-2**53 + 1` and `+2**53 - 1`.
- **trailingComma[=on]**: allow the presence of a trailing comma after the last object member or array element.
- **allowComments[=on]**: JSOn standard doesn't mention about comments. Turn this on to parse both C style comments of `//..EOL` and `/* .. */`.
- **leadingFraction[=on]**: something like `.123` is not a valid JSON number, but its widespread usage sometimes creeps into Json documents.
- **integerPositiveSign[=on]**: `+123` is also not a valid JSON number, but since `-123` is a valid JSON number, why not parse it safely?
## Safety features
You can modify these default configurations to suit your needs.
- **nestedDepthLimit: 512**: maximum depth of the nested structure, they are a combination of objects and arrays depth(0=disable).
- **arrayElementsLimit: 0**: maximum number of allowed array elements(0=disable).
- **objectMembersLimit: 0**: maximum number of key-value pairs in an object(0=disable).
- **integerDigitsLimit: 128**: limit the maximum digits of the integer part of JSON number.
- **fractionDigitsLimit: 128**: limit the maximum digits of faction part of JSON number.
- **exponentDigitsLimit: 32**: limit the maximum digits of the exponent part of JSON number.
- **stringLengthLimit: 0**: limit the maximum bytes of string(0=disable).
## Special types
- **JsonString**: Use this type if you want to parse a Json value to a valid Json document contained in a string.
- **JsonVoid**: Use this type to skip a valid Json value.
- **JsonNumber**: Use this to parse a valid Json number including the fraction and exponent part.
- Please note that this type is a generic, it support `uint64` and `string` as generic param.
- The generic param will define the integer and exponent part as `uint64` or `string`.
- If the generic param is `uint64`, overflow can happen, or max digit limit will apply.
- If the generic param is `string`, the max digit limit will apply.
- The fraction part is always a string to keep the leading zero of the fractional number.
- **JsonValueRef**: Use this type to parse any valid Json value into something like stdlib/JsonNode.
- `JsonValueRef` is using `JsonNumber` instead of `int` or `float` like stdlib/JsonNode.
## Flavor
While flags and limits are runtime configuration, flavor is a powerful compile time mechanism to prevent
cross contamination between different custom serializer operated the same type. For example,
`json-rpc` subsystem dan `json-rest` subsystem maybe have different custom serializer for the same `UInt256`.
Json-Flavor will make sure, the compiler picks the right serializer for the right subsystem.
You can use `useDefaultSerializationIn` to add serializers of a flavor to a specific type.
```Nim
# These are the parameters you can pass to `createJsonFlavor` to create a new flavor.
FlavorName: untyped
mimeTypeValue = "application/json"
automaticObjectSerialization = false
requireAllFields = true
omitOptionalFields = true
allowUnknownFields = true
skipNullFields = false
```
```Nim
type
OptionalFields = object
one: Opt[string]
two: Option[int]
createJsonFlavor OptJson
OptionalFields.useDefaultSerializationIn OptJson
```
`omitOptionalFields` is used by the Writer to ignore fields with null value.
`skipNullFields` is used by the Reader to ignore fields with null value.
## Decoder example
```nim
type
NimServer = object
name: string
port: int
MixedServer = object
name: JsonValueRef
port: int
StringServer = object
name: JsonString
port: JsonString
# decode into native Nim
var nim_native = Json.decode(rawJson, NimServer)
# decode into mixed Nim + JsonValueRef
var nim_mixed = Json.decode(rawJson, MixedServer)
# decode any value into string
var nim_string = Json.decode(rawJson, StringServer)
# decode any valid JSON
var json_value = Json.decode(rawJson, JsonValueRef)
```
## Load and save
```Nim
var server = Json.loadFile("filename.json", Server)
var server_string = Json.loadFile("filename.json", JsonString)
Json.saveFile("filename.json", server)
```
## Objects
Decoding an object can be achieved via the `parseObject` template.
To parse the value, you can use one of the helper functions or use `readValue`.
`readObject` and `readObjectFields` iterators are also handy when creating a custom object parser.
```Nim
proc readValue*(r: var JsonReader, table: var Table[string, int]) =
parseObject(r, key):
table[key] = r.parseInt(int)
```
## Sets and list-like
Similar to `Object`, sets and list or array-like data structures can be parsed using
`parseArray` template. It comes in two variations, indexed and non-indexed.
Built-in `readValue` for regular `seq` and `array` is implemented for you.
No built-in `readValue` for `set` or `set-like` is provided, you must overload it yourself depending on your need.
```nim
type
HoldArray = object
data: array[3, int]
HoldSeq = object
data: seq[int]
WelderFlag = enum
TIG
MIG
MMA
Welder = object
flags: set[WelderFlag]
proc readValue*(r: var JsonReader, value: var HoldArray) =
# parseArray with index, `i` can be any valid identifier
r.parseArray(i):
value.data[i] = r.parseInt(int)
proc readValue*(r: var JsonReader, value: var HoldSeq) =
# parseArray without index
r.parseArray:
let lastPos = value.data.len
value.data.setLen(lastPos + 1)
readValue(r, value.data[lastPos])
proc readValue*(r: var JsonReader, value: var Welder) =
# populating set also okay
r.parseArray:
value.flags.incl r.parseInt(int).WelderFlag
```
## Custom iterators
Using these custom iterators, you can have access to sub-token elements.
```Nim
customIntValueIt(r: var JsonReader; body: untyped)
customNumberValueIt(r: var JsonReader; body: untyped)
customStringValueIt(r: var JsonReader; limit: untyped; body: untyped)
customStringValueIt(r: var JsonReader; body: untyped)
```
## Convenience iterators
```Nim
readArray(r: var JsonReader, ElemType: typedesc): ElemType
readObjectFields(r: var JsonReader, KeyType: type): KeyType
readObjectFields(r: var JsonReader): string
readObject(r: var JsonReader, KeyType: type, ValueType: type): (KeyType, ValueType)
```
## Helper procs
When crafting a custom serializer, use these parsers, they are safe and intuitive.
Avoid using the lexer directly.
```Nim
tokKind(r: var JsonReader): JsonValueKind
parseString(r: var JsonReader, limit: int): string
parseString(r: var JsonReader): string
parseBool(r: var JsonReader): bool
parseNull(r: var JsonReader)
parseNumber(r: var JsonReader, T: type): JsonNumber[T: string or uint64]
parseNumber(r: var JsonReader, val: var JsonNumber)
toInt(r: var JsonReader, val: JsonNumber, T: type SomeInteger, portable: bool): T
parseInt(r: var JsonReader, T: type SomeInteger, portable: bool = false): T
toFloat(r: var JsonReader, val: JsonNumber, T: type SomeFloat): T
parseFloat(r: var JsonReader, T: type SomeFloat): T
parseAsString(r: var JsonReader, val: var string)
parseAsString(r: var JsonReader): JsonString
parseValue(r: var JsonReader, T: type): JsonValueRef[T: string or uint64]
parseValue(r: var JsonReader, val: var JsonValueRef)
parseArray(r: var JsonReader; body: untyped)
parseArray(r: var JsonReader; idx: untyped; body: untyped)
parseObject(r: var JsonReader, key: untyped, body: untyped)
parseObjectWithoutSkip(r: var JsonReader, key: untyped, body: untyped)
parseObjectSkipNullFields(r: var JsonReader, key: untyped, body: untyped)
parseObjectCustomKey(r: var JsonReader, keyAction: untyped, body: untyped)
parseJsonNode(r: var JsonReader): JsonNode
skipSingleJsValue(r: var JsonReader)
readRecordValue[T](r: var JsonReader, value: var T)
```
## Helper procs of JsonWriter
```Nim
beginRecord(w: var JsonWriter, T: type)
beginRecord(w: var JsonWriter)
endRecord(w: var JsonWriter)
writeObject(w: var JsonWriter, T: type)
writeObject(w: var JsonWriter)
writeFieldName(w: var JsonWriter, name: string)
writeField(w: var JsonWriter, name: string, value: auto)
iterator stepwiseArrayCreation[C](w: var JsonWriter, collection: C): auto
writeIterable(w: var JsonWriter, collection: auto)
writeArray[T](w: var JsonWriter, elements: openArray[T])
writeNumber[F,T](w: var JsonWriter[F], value: JsonNumber[T])
writeJsonValueRef[F,T](w: var JsonWriter[F], value: JsonValueRef[T])
```
## Enums
```Nim
type
Fruit = enum
Apple = "Apple"
Banana = "Banana"
Drawer = enum
One
Two
Number = enum
Three = 3
Four = 4
Mixed = enum
Six = 6
Seven = "Seven"
```
nim-json-serialization automatically detect which representation an enum should be parsed.
The detection occurs when parse json literal and from the enum declaration itself.
'Fruit' expect string literal. 'Drawer' or 'Number' expect numeric literal.
'Mixed' is disallowed. If the json literal does not match the expected enum style,
exception will be raised. But you can configure individual enum type with:
```Nim
configureJsonDeserialization(
T: type[enum], allowNumericRepr: static[bool] = false,
stringNormalizer: static[proc(s: string): string] = strictNormalize)
# example:
Mixed.configureJsonDeserialization(allowNumericRepr = true) # only at top level
```
When encode an enum, user is also given flexibility to configure at Flavor level
or for individual enum type.
```Nim
type
EnumRepresentation* = enum
EnumAsString
EnumAsNumber
EnumAsStringifiedNumber
# examples:
# Flavor level
Json.flavorEnumRep(EnumAsString) # default flavor, can be called from non top level
Flavor.flavorEnumRep(EnumAsNumber) # custom flavor, can be called from non top level
# individual enum type no matter what flavor
Fruit.configureJsonSerialization(EnumAsNumber) # only at top level
# individual enum type of specific flavor
MyJson.flavorEnumRep(Drawer, EnumAsString) # only at top level
```
## License
Licensed and distributed under either of
* MIT license: [LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT
or
* Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0)
at your option. These files may not be copied, modified, or distributed except according to those terms.