Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pwwang/toml-bench
Which toml package to use in python?
https://github.com/pwwang/toml-bench
benchmarking toml toml-parser toml-parsing toml-validation
Last synced: 3 days ago
JSON representation
Which toml package to use in python?
- Host: GitHub
- URL: https://github.com/pwwang/toml-bench
- Owner: pwwang
- License: mit
- Created: 2022-05-06T23:22:48.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-06-25T19:40:36.000Z (6 months ago)
- Last Synced: 2024-12-10T05:42:11.123Z (13 days ago)
- Topics: benchmarking, toml, toml-parser, toml-parsing, toml-validation
- Language: Python
- Homepage:
- Size: 131 KB
- Stars: 51
- Watchers: 4
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# toml-bench
[![deps][1]][2]
Which toml package to use in python?
See also: [toml-lang](https://toml.io/en/) and [PEP 680](https://www.python.org/dev/peps/pep-0680/)
## Report
### Version
The verions of the packages tested in this report.
| |Version|
|-|-----------------------|
|toml|0.10.2|
|tomli/tomli_w|2.0.1; **tomli_w**: 1.0.0|
|tomlkit|0.12.5|
|rtoml|0.11.0|
|qtoml|0.3.1|
|tomllib|(Python 3.12.2)|### Dumping `None` value
How the package dumps `None` value in python
Literally `.dumps(None)`
| |Dumped value or error|
|-|-----------------------|
|toml|'NoneType' object is not iterable|
|tomli/tomli_w|'NoneType' object has no attribute 'items'|
|tomlkit|Expecting Mapping or TOML Container, given|
|rtoml|"null"
---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
"@None"|
|qtoml|'NoneType' object has no attribute 'items'|
|tomllib|module 'tomllib' has no attribute 'dumps'|### Dumping key-`None` pair
How the package dumps key-value pair with value `None`
Literally `.dumps({"key": None})`
| |Dumped value or error|
|-|-----------------------|
|toml||
|tomli/tomli_w|Object of type is not TOML serializable|
|tomlkit|Invalid type |
|rtoml|key = "null"
---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
key = "@None"
|
|qtoml|TOML cannot encode None|
|tomllib|module 'tomllib' has no attribute 'dumps'|### Dumping list with `None` value
How the package dumps a list with `None` value in it.
Literally `.dumps({"key": [1, 2, 3, None, 5]})`
| |Dumped value or error|
|-|-----------------------|
|toml|key = [ 1, 2, 3, "None", 5,]
|
|tomli/tomli_w|Object of type is not TOML serializable|
|tomlkit|Invalid type |
|rtoml|key = [1, 2, 3, "null", 5]
---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
key = [1, 2, 3, "@None", 5]
|
|qtoml|bad type '' for dump_value|
|tomllib|module 'tomllib' has no attribute 'dumps'|### Loading `None`-like values
How the package loads `None`-like value in string
Literally `.loads('v1 = "null" v2 = "None"')`
| |Loaded as|
|-|-----------------------|
|toml|{'v1': 'null', 'v2': 'None'}|
|tomli/tomli_w|module 'tomli_w' has no attribute 'loads'|
|tomlkit|{'v1': 'null', 'v2': 'None'}|
|rtoml|{'v1': 'null', 'v2': 'None'}
---
rtoml v0.11+ supports loading custom None values:
`rtoml.loads(data, none_value='None')`:
{'v1': 'null', 'v2': None}
`rtoml.loads(data, none_value='null')`:
{'v1': None, 'v2': 'None'}|
|qtoml|{'v1': 'null', 'v2': 'None'}|
|tomllib|{'v1': 'null', 'v2': 'None'}|### Dumping a heterogenous array
How the package dumps a python dictionary with a heterogenous array.
Literally `.dumps({"v": [1, 1.2, True, "string"]})`
| |Dumped value or error|
|-|-----------------------|
|toml|v = \[ 1, 1.2, true, "string",\]
|
|tomli/tomli_w|v = \[
1,
1.2,
true,
"string",
\]
|
|tomlkit|v = \[1, 1.2, true, "string"\]
|
|rtoml|v = \[1, 1.2, true, "string"\]
|
|qtoml|v = \[1, 1.2, true, 'string'\]
|
|tomllib|Dumping not supported|### Loading a heterogenous array
How the package loads a toml string with a heterogenous array.
Literally `.loads('v = [1, 1.2, True, "string"]')`
| |Loaded as|
|-|-----------------------|
|toml|Not a homogeneous array (line 2 column 1 char 1)|
|tomli/tomli_w|`{'v': [1, 1.2, True, 'string']}`|
|tomlkit|`{'v': [1, 1.2, True, 'string']}`|
|rtoml|`{'v': [1, 1.2, True, 'string']}`|
|qtoml|`{'v': [1, 1.2, True, 'string']}`|
|tomllib|`{'v': [1, 1.2, True, 'string']}`|### Dumping a nested array
How the package dumps a python dictionary with a nested array.
Literally `.dumps({"v": [[1], [1, 2]]})`
| |Dumped value or error|
|-|-----------------------|
|toml|v = \[ \[ 1,\], \[ 1, 2,\],\]
|
|tomli/tomli_w|v = \[
\[
1,
\],
\[
1,
2,
\],
\]
|
|tomlkit|v = \[\[1\], \[1, 2\]\]
|
|rtoml|v = \[\[1\], \[1, 2\]\]
|
|qtoml|v = \[\[1\], \[1, 2\]\]
|
|tomllib|Dumping not supported|### Loading a nested array
How the package loads a toml string with a nested array.
Literally `.loads('v = [[1], [1, 2]]')`
| |Loaded as|
|-|-----------------------|
|toml|`{'v': [[1], [1, 2]]}`|
|tomli/tomli_w|`{'v': [[1], [1, 2]]}`|
|tomlkit|`{'v': [[1], [1, 2]]}`|
|rtoml|`{'v': [[1], [1, 2]]}`|
|qtoml|`{'v': [[1], [1, 2]]}`|
|tomllib|`{'v': [[1], [1, 2]]}`|### Dumping keeps order of keys?
Whether the package preserves the order of the keys while dumps
a python dictionary.Thus, whether `.dumps({"c": 1, "a": 2, "b": 3})` yields a string
like `c = 1\na = 2\nb = 3\n`.| |Order kept?|
|-|-----------------------|
|toml|Kept|
|tomli/tomli_w|Kept|
|tomlkit|Kept|
|rtoml|Kept|
|qtoml|Kept|
|tomllib|Dumping not supported|### Loading keeps order of keys?
Whether the package preserves the order of the keys
while loads a TOML string.Thus, whether `.loads('c = 1\na = 2\nb = 3\n')` yields
a dictionary with keys in the order of `['c', 'a', 'b']`.| |Order kept?|
|-|-----------------------|
|toml|Kept|
|tomli/tomli_w|Kept|
|tomlkit|Kept|
|rtoml|Kept|
|qtoml|Kept|
|tomllib|Kept|### Dumping unicode
How the package dumps Unicode in python
Literally, `.dumps({"你好": "世界"})`
| |Dumped value|
|-|-----------------------|
|toml|"你好" = "世界"
|
|tomli/tomli_w|"你好" = "世界"
|
|tomlkit|"你好" = "世界"
|
|rtoml|"你好" = "世界"
|
|qtoml|'你好' = '世界'
|
|tomllib|Dumping not supported|### Loaded unicode
How the package loads a file with unicode.
The file was created by:
```python
## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
f.write('"你好" = "世界"\n')## Use `.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
loaded = .load(f)
```| |Loaded as|
|-|-----------------------|
|toml|{'你好': '世界'}|
|tomli/tomli_w|File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`
When loaded with `rb`:
{'你好': '世界'}|
|tomlkit|{'你好': '世界'}|
|rtoml|{'你好': '世界'}|
|qtoml|{'你好': '世界'}|
|tomllib|File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`
When loaded with `rb`:
{'你好': '世界'}|### Compliance with valid tests in toml-test
Test the compliance with the standard test suites for valid toml files
here:> https://github.com/BurntSushi/toml-test/
The tests come up with a JSON counterpart that can be used to valid whether
loading the toml file yields the same result as the JSON counterpart.| |Result (toml-test v1.5.0)|
|-|-----------------------|
|toml|[spec/array-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/array-0.toml) Not a homogeneous array (line 8 column 1 char 261)
[spec/keys-4.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/keys-4.toml) Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57)
[spec/local-time-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/local-time-0.toml) Parsed as unexpected data.
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46)
[datetime/local-time.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/local-time.toml) Parsed as unexpected data.
[datetime/datetime.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/datetime.toml) Parsed as unexpected data.
[comment/tricky.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/tricky.toml) Parsed as unexpected data.
[key/dotted-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-1.toml) Parsed as unexpected data.
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Found invalid character in key name: ''. Try quoting the key name. (line 5 column 2 char 67)
[key/dotted-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-2.toml) Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166)
[key/quoted-unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/quoted-unicode.toml) Duplicate keys! (line 3 column 1 char 19)
[key/dotted-empty.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-empty.toml) Duplicate keys! (line 2 column 1 char 17)
[key/escapes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/escapes.toml) Parsed as unexpected data.
[table/empty-name.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/table/empty-name.toml) Can't have a keygroup with an empty name (line 1 column 1 char 0)
[string/raw-multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/raw-multiline.toml) Unbalanced quotes (line 20 column 50 char 532)
[string/ends-in-whitespace-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/ends-in-whitespace-escape.toml) Reserved escape sequence used (line 6 column 1 char 28)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Reserved escape sequence used (line 3 column 1 char 35)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Reserved escape sequence used (line 1 column 1 char 0)
[string/multiline-quotes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/multiline-quotes.toml) Unterminated string found. Reached end of file. (line 27 column 1 char 664)
[float/zero.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/float/zero.toml) Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)
[array/mixed-int-string.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-string.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/nested-double.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/nested-double.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/string-with-comma-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/string-with-comma-2.toml) string index out of range
[array/mixed-int-float.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-float.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/mixed-string-table.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-string-table.toml) list index out of range
[array/mixed-int-array.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-array.toml) Not a homogeneous array (line 1 column 1 char 0)
[inline-table/multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/multiline.toml) Invalid inline table value encountered (line 1 column 1 char 0)
[inline-table/key-dotted-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/key-dotted-1.toml) Parsed as unexpected data.
[inline-table/key-dotted-5.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/key-dotted-5.toml) Not a homogeneous array (line 2 column 1 char 20)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Key name found without value. Reached end of line. (line 5 column 2 char 98)
*157/187 (83.96%) passed*|
|tomli/tomli_w|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Expected newline or end of document after a statement (at line 2, column 23)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Invalid statement (at line 3, column 1)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Unescaped '\' in a string (at line 3, column 22)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Unescaped '\' in a string (at line 1, column 10)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Invalid initial character for a key part (at line 3, column 21)
*182/187 (97.33%) passed*|
|tomlkit|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Invalid number at line 2 col 25
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Empty key at line 3 col 0
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Invalid character 'x' in string at line 3 col 20
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Invalid character 'e' in string at line 1 col 8
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Empty key at line 3 col 20
*182/187 (97.33%) passed*|
|rtoml|[spec/table-9.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/table-9.toml) duplicate key: `apple` for key `fruit` at line 8 column 1
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) expected a colon, found a newline at line 2 column 26
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) unexpected character found: `\u{20ac}` at line 3 column 1
[table/array-within-dotted.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/table/array-within-dotted.toml) duplicate key: `apple` for key `fruit` at line 4 column 1
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) invalid escape character in string: `x` at line 3 column 21
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) invalid escape character in string: `e` at line 1 column 9
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) expected a table key, found a newline at line 3 column 21
*180/187 (96.26%) passed*|
|qtoml|[spec/string-4.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/string-4.toml) Didn't find expected newline (line 7, column 62)
[spec/string-7.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/string-7.toml) Didn't find expected newline (line 7, column 50)
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) can't parse type (line 2, column 20)
[datetime/milliseconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/milliseconds.toml) Didn't find expected newline (line 2, column 27)
[datetime/datetime.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/datetime.toml) Didn't find expected newline (line 4, column 18)
[comment/after-literal-no-ws.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/after-literal-no-ws.toml) can't parse type (line 1, column 4)
[comment/tricky.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/tricky.toml) can't parse type (line 11, column 7)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) '€' cannot begin key (line 3, column 0)
[string/raw-multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/raw-multiline.toml) Didn't find expected newline (line 22, column 3)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) \x not a valid escape (line 3, column 43)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) \e not a valid escape (line 1, column 33)
[string/multiline-quotes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/multiline-quotes.toml) Didn't find expected newline (line 4, column 26)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) ' ' cannot begin key (line 3, column 20)
*174/187 (93.05%) passed*|
|tomllib|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Expected newline or end of document after a statement (at line 2, column 23)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Invalid statement (at line 3, column 1)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Unescaped '\' in a string (at line 3, column 22)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Unescaped '\' in a string (at line 1, column 10)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Invalid initial character for a key part (at line 3, column 21)
*182/187 (97.33%) passed*|### Compliance with invalid tests in toml-test
Test the compliance with the standard test suites for invalid toml files
here:> https://github.com/BurntSushi/toml-test/
- `Not OK`: The toml file is parsed without error, but expected to fail.
- `OK`: All files are failed to parse, as expected. Showing the last
parsing error.| |Result (toml-test v1.5.0)|
|-|-----------------------|
|toml|Not OK: [integer/double-sign-plus.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/double-sign-plus.toml) incorrectly parsed.
Not OK: [integer/us-after-bin.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-bin.toml) incorrectly parsed.
Not OK: [integer/double-sign-nex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/double-sign-nex.toml) incorrectly parsed.
Not OK: [integer/us-after-hex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-hex.toml) incorrectly parsed.
Not OK: [integer/us-after-oct.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-oct.toml) incorrectly parsed.
Not OK: [spec/inline-table-2-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/inline-table-2-0.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-hour.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-hour.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/string-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/string-del.toml) incorrectly parsed.
Not OK: *63 more items incorrectly parsed.*
*298/371 (80.32%) passed*|
|tomli/tomli_w|OK: [inline-table/linebreak-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/linebreak-1.toml) Unclosed inline table (at line 3, column 18)
*371/371 (100%) passed*|
|tomlkit|Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: [control/rawmulti-cd.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/rawmulti-cd.toml) incorrectly parsed.
Not OK: [control/bare-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/bare-cr.toml) incorrectly parsed.
Not OK: [table/append-with-dotted-keys-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-with-dotted-keys-1.toml) incorrectly parsed.
Not OK: [table/overwrite-array-in-parent.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/overwrite-array-in-parent.toml) incorrectly parsed.
Not OK: [table/append-to-array-with-dotted-keys.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-to-array-with-dotted-keys.toml) incorrectly parsed.
Not OK: [table/append-with-dotted-keys-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-with-dotted-keys-2.toml) incorrectly parsed.
Not OK: [array/extend-defined-aot.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/array/extend-defined-aot.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-09.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/overwrite-09.toml) incorrectly parsed.
*361/371 (97.30%) passed*|
|rtoml|Not OK: [integer/positive-hex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-hex.toml) incorrectly parsed.
Not OK: [integer/positive-bin.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-bin.toml) incorrectly parsed.
Not OK: [integer/positive-oct.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-oct.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-hour.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-hour.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: [control/rawmulti-cd.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/rawmulti-cd.toml) incorrectly parsed.
Not OK: [control/bare-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/bare-cr.toml) incorrectly parsed.
*361/371 (97.30%) passed*|
|qtoml|Not OK: [spec/inline-table-2-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/inline-table-2-0.toml) incorrectly parsed.
Not OK: [spec/table-9-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/table-9-1.toml) incorrectly parsed.
Not OK: [spec/table-9-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/table-9-0.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/comment-lf.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-lf.toml) incorrectly parsed.
Not OK: [control/comment-null.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-null.toml) incorrectly parsed.
Not OK: [control/comment-ff.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-ff.toml) incorrectly parsed.
Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: *14 more items incorrectly parsed.*
*347/371 (93.53%) passed*|
|tomllib|OK: [inline-table/linebreak-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/linebreak-1.toml) Unclosed inline table (at line 3, column 18)
*371/371 (100%) passed*|### Compliance with valid tests in python tomllib test data
Test the compliance with python tomllib test data (since python 3.11)
for valid toml files here:> https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid
The tests come up with a JSON counterpart that can be used to valid whether
loading the toml file yields the same result as the JSON counterpart.| |Result (cpython tag 3.12.4)|
|-|-----------------------|
|toml|[apostrophes-in-literal-string.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/apostrophes-in-literal-string.toml) Unbalanced quotes (line 1 column 50 char 49)
[five-quotes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/five-quotes.toml) Unterminated string found. Reached end of file. (line 7 column 1 char 97)
[dates-and-times/datetimes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/dates-and-times/datetimes.toml) Parsed as unexpected data.
[multiline-basic-str/ends-in-whitespace-escape.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/multiline-basic-str/ends-in-whitespace-escape.toml) Reserved escape sequence used (line 6 column 1 char 28)
*8/12 (66.67%) passed*|
|tomli/tomli_w|OK, *12/12 (100%) passed*|
|tomlkit|OK, *12/12 (100%) passed*|
|rtoml|OK, *12/12 (100%) passed*|
|qtoml|[apostrophes-in-literal-string.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/apostrophes-in-literal-string.toml) Didn't find expected newline (line 3, column 3)
[five-quotes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/five-quotes.toml) Didn't find expected newline (line 3, column 3)
[dates-and-times/datetimes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/dates-and-times/datetimes.toml) Didn't find expected newline (line 1, column 19)
*9/12 (75.00%) passed*|
|tomllib|OK, *12/12 (100%) passed*|### Compliance with invalid tests in python tomllib test data
Test the compliance with python tomllib test data (since python 3.11)
for invalid toml files here:> https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid
- `Not OK`: The toml file is parsed without error, but expected to fail.
- `OK`: All files are failed to parse, as expected. Showing the last
parsing error.| |Result (cpython tag 3.12.4)|
|-|-----------------------|
|toml|Not OK: [invalid-comment-char.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/invalid-comment-char.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [array/unclosed-empty.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/unclosed-empty.toml) incorrectly parsed.
Not OK: [array/file-end-after-val.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/file-end-after-val.toml) incorrectly parsed.
Not OK: [array/unclosed-after-item.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/unclosed-after-item.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-value-in-inner-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-value-in-inner-table.toml) incorrectly parsed.
Not OK: [inline-table/unclosed-empty.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/unclosed-empty.toml) incorrectly parsed.
*41/50 (82.00%) passed*|
|tomli/tomli_w|OK: [inline-table/overwrite-implicitly.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-implicitly.toml) Cannot overwrite a value (at line 1, column 21)
*50/50 (100%) passed*|
|tomlkit|Not OK: [array-of-tables/overwrite-array-in-parent.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array-of-tables/overwrite-array-in-parent.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-aot.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-aot.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [inline-table/override-val-in-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/override-val-in-table.toml) incorrectly parsed.
*44/50 (88.00%) passed*|
|rtoml|Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
*49/50 (98.00%) passed*|
|qtoml|Not OK: [non-scalar-escaped.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/non-scalar-escaped.toml) incorrectly parsed.
Not OK: [invalid-comment-char.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/invalid-comment-char.toml) incorrectly parsed.
Not OK: [table/redefine-2.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/table/redefine-2.toml) incorrectly parsed.
Not OK: [table/redefine-1.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/table/redefine-1.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-value-in-inner-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-value-in-inner-table.toml) incorrectly parsed.
Not OK: [inline-table/override-val-with-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/override-val-with-table.toml) incorrectly parsed.
*41/50 (82.00%) passed*|
|tomllib|OK: [inline-table/overwrite-implicitly.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-implicitly.toml) Cannot overwrite a value (at line 1, column 21)
*50/50 (100%) passed*|### Running speed with data provided by `rtoml`
Test the speed of loading and dumping the loaded using data
provided by `rtoml`> https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml
| |Loading speed|Dumping speed|
|-|-|-|
|toml|Excluded (heterogeneous arrays not supported)|Excluded (heterogeneous arrays not supported)|
|tomli/tomli_w|2.14s (5000 iterations)|0.73s (5000 iterations)|
|tomlkit|39.78s (5000 iterations)|0.98s (5000 iterations)|
|rtoml|0.37s (5000 iterations)|0.08s (5000 iterations)|
|qtoml|4.99s (5000 iterations)|1.87s (5000 iterations)|
|tomllib|2.04s (5000 iterations)|Dumping not supported|### Running speed with data provided by `tomli`
Test the speed of loading and dumping the loaded using data
provided by `tomli`> https://github.com/hukkin/tomli/raw/master/benchmark/data.toml
| |Loading speed|Dumping speed|
|-|-|-|
|toml|Excluded (heterogeneous arrays not supported)|Excluded (heterogeneous arrays not supported)|
|tomli/tomli_w|1.41s (5000 iterations)|0.46s (5000 iterations)|
|tomlkit|24.55s (5000 iterations)|0.51s (5000 iterations)|
|rtoml|0.32s (5000 iterations)|0.16s (5000 iterations)|
|qtoml|3.63s (5000 iterations)|1.25s (5000 iterations)|
|tomllib|1.44s (5000 iterations)|Dumping not supported|## Other reports
- [Tests with `toml-test` v1.4.0](./reports/with_toml-test_v1.4.0.md)
- [Tests with `toml-test` v1.3.0](./reports/with_toml-test_v1.3.0.md)
- [Tests with `toml-test` v1.2.0](./reports/with_toml-test_v1.2.0.md)## Run your own report
### Install
```shell
pip install -U toml-bench
```### Generate your own report
```shell
toml-bench
```#### Use a different data directory than the default one
```shell
toml-bench --datadir /tmp/toml-bench
```#### Write the report to a markdown file
```shell
toml-bench --report ./README.md
```#### Test with a different version of compliance set (`BurntSushi/toml-test`)
```shell
toml-bench --comver 1.0.0
```#### Use a different number of iterations in speed tests
```shell
toml-bench --iter 5000
```#### Test with different versions of packages
```shell
git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench
```[1]: https://img.shields.io/librariesio/release/pypi/toml-bench?style=flat-square
[2]: https://libraries.io/github/pwwang/toml-bench#repository_dependencies