Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/pwwang/toml-bench

Which toml package to use in python?
https://github.com/pwwang/toml-bench
benchmarking toml toml-parser toml-parsing toml-validation
Last synced: about 2 months ago
JSON representation
Which toml package to use in python?
Host: GitHub
URL: https://github.com/pwwang/toml-bench
Owner: pwwang
License: mit
Created: 2022-05-06T23:22:48.000Z (over 2 years ago)
Default Branch: master
Last Pushed: 2024-06-25T19:40:36.000Z (3 months ago)
Last Synced: 2024-07-05T13:23:30.492Z (2 months ago)
Topics: benchmarking, toml, toml-parser, toml-parsing, toml-validation
Language: Python
Homepage:
Size: 131 KB
Stars: 50
Watchers: 4
Forks: 1
Open Issues: 2
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

        # toml-bench

[![deps][1]][2]

Which toml package to use in python?

See also: [toml-lang](https://toml.io/en/) and [PEP 680](https://www.python.org/dev/peps/pep-0680/)

## Report

### Version

The verions of the packages tested in this report.

| |Version|

|-|-----------------------|

|toml|0.10.2|

|tomli/tomli_w|2.0.1; **tomli_w**: 1.0.0|

|tomlkit|0.12.5|

|rtoml|0.11.0|

|qtoml|0.3.1|

|tomllib|(Python 3.12.2)|

### Dumping `None` value

How the package dumps `None` value in python

Literally `.dumps(None)`

| |Dumped value or error|

|-|-----------------------|

|toml|'NoneType' object is not iterable|

|tomli/tomli_w|'NoneType' object has no attribute 'items'|

|tomlkit|Expecting Mapping or TOML Container,  given|

|rtoml|"null"
---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
"@None"|

|qtoml|'NoneType' object has no attribute 'items'|

|tomllib|module 'tomllib' has no attribute 'dumps'|

### Dumping key-`None` pair

How the package dumps key-value pair with value `None`

Literally `.dumps({"key": None})`

| |Dumped value or error|

|-|-----------------------|

|toml||

|tomli/tomli_w|Object of type  is not TOML serializable|

|tomlkit|Invalid type |

|rtoml|key = "null"

---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
key = "@None"
|

|qtoml|TOML cannot encode None|

|tomllib|module 'tomllib' has no attribute 'dumps'|

### Dumping list with `None` value

How the package dumps a list with `None` value in it.

Literally `.dumps({"key": [1, 2, 3, None, 5]})`

| |Dumped value or error|

|-|-----------------------|

|toml|key = [ 1, 2, 3, "None", 5,]
|

|tomli/tomli_w|Object of type  is not TOML serializable|

|tomlkit|Invalid type |

|rtoml|key = [1, 2, 3, "null", 5]

---
rtoml v0.11+ supports dumping None to a desired string:
`rtoml.dumps(data, none_value='@None')`:
key = [1, 2, 3, "@None", 5]
|

|qtoml|bad type '' for dump_value|

|tomllib|module 'tomllib' has no attribute 'dumps'|

### Loading `None`-like values

How the package loads `None`-like value in string

Literally `.loads('v1 = "null" v2 = "None"')`

| |Loaded as|

|-|-----------------------|

|toml|{'v1': 'null', 'v2': 'None'}|

|tomli/tomli_w|module 'tomli_w' has no attribute 'loads'|

|tomlkit|{'v1': 'null', 'v2': 'None'}|

|rtoml|{'v1': 'null', 'v2': 'None'}
---
rtoml v0.11+ supports loading custom None values:
`rtoml.loads(data, none_value='None')`:
{'v1': 'null', 'v2': None}
`rtoml.loads(data, none_value='null')`:
{'v1': None, 'v2': 'None'}|

|qtoml|{'v1': 'null', 'v2': 'None'}|

|tomllib|{'v1': 'null', 'v2': 'None'}|

### Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally `.dumps({"v": [1, 1.2, True, "string"]})`

| |Dumped value or error|

|-|-----------------------|

|toml|v = \[ 1, 1.2, true, "string",\]
|

|tomli/tomli_w|v = \[
    1,
    1.2,
    true,
    "string",
\]
|

|tomlkit|v = \[1, 1.2, true, "string"\]
|

|rtoml|v = \[1, 1.2, true, "string"\]
|

|qtoml|v = \[1, 1.2, true, 'string'\]
|

|tomllib|Dumping not supported|

### Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally `.loads('v = [1, 1.2, True, "string"]')`

| |Loaded as|

|-|-----------------------|

|toml|Not a homogeneous array (line 2 column 1 char 1)|

|tomli/tomli_w|`{'v': [1, 1.2, True, 'string']}`|

|tomlkit|`{'v': [1, 1.2, True, 'string']}`|

|rtoml|`{'v': [1, 1.2, True, 'string']}`|

|qtoml|`{'v': [1, 1.2, True, 'string']}`|

|tomllib|`{'v': [1, 1.2, True, 'string']}`|

### Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally `.dumps({"v": [[1], [1, 2]]})`

| |Dumped value or error|

|-|-----------------------|

|toml|v = \[ \[ 1,\], \[ 1, 2,\],\]
|

|tomli/tomli_w|v = \[
    \[
        1,
    \],
    \[
        1,
        2,
    \],
\]
|

|tomlkit|v = \[\[1\], \[1, 2\]\]
|

|rtoml|v = \[\[1\], \[1, 2\]\]
|

|qtoml|v = \[\[1\], \[1, 2\]\]
|

|tomllib|Dumping not supported|

### Loading a nested array

How the package loads a toml string with a nested array.

Literally `.loads('v = [[1], [1, 2]]')`

| |Loaded as|

|-|-----------------------|

|toml|`{'v': [[1], [1, 2]]}`|

|tomli/tomli_w|`{'v': [[1], [1, 2]]}`|

|tomlkit|`{'v': [[1], [1, 2]]}`|

|rtoml|`{'v': [[1], [1, 2]]}`|

|qtoml|`{'v': [[1], [1, 2]]}`|

|tomllib|`{'v': [[1], [1, 2]]}`|

### Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps

a python dictionary.

Thus, whether `.dumps({"c": 1, "a": 2, "b": 3})` yields a string

like `c = 1\na = 2\nb = 3\n`.

| |Order kept?|

|-|-----------------------|

|toml|Kept|

|tomli/tomli_w|Kept|

|tomlkit|Kept|

|rtoml|Kept|

|qtoml|Kept|

|tomllib|Dumping not supported|

### Loading keeps order of keys?

Whether the package preserves the order of the keys

while loads a TOML string.

Thus, whether `.loads('c = 1\na = 2\nb = 3\n')` yields

a dictionary with keys in the order of `['c', 'a', 'b']`.

| |Order kept?|

|-|-----------------------|

|toml|Kept|

|tomli/tomli_w|Kept|

|tomlkit|Kept|

|rtoml|Kept|

|qtoml|Kept|

|tomllib|Kept|

### Dumping unicode

How the package dumps Unicode in python

Literally, `.dumps({"你好": "世界"})`

| |Dumped value|

|-|-----------------------|

|toml|"你好" = "世界"
|

|tomli/tomli_w|"你好" = "世界"
|

|tomlkit|"你好" = "世界"
|

|rtoml|"你好" = "世界"
|

|qtoml|'你好' = '世界'
|

|tomllib|Dumping not supported|

### Loaded unicode

How the package loads a file with unicode.

The file was created by:

```python

## Create a file with unicode content

with open(self.datafile, "w", encoding="utf-8") as f:

    f.write('"你好" = "世界"\n')

## Use `.load()` to load the file

with open(self.datafile, "r", encoding="utf-8") as f:

    loaded = .load(f)

```

| |Loaded as|

|-|-----------------------|

|toml|{'你好': '世界'}|

|tomli/tomli_w|File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`
When loaded with `rb`:
{'你好': '世界'}|

|tomlkit|{'你好': '世界'}|

|rtoml|{'你好': '世界'}|

|qtoml|{'你好': '世界'}|

|tomllib|File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')`
When loaded with `rb`:
{'你好': '世界'}|

### Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files

here:

> https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether

loading the toml file yields the same result as the JSON counterpart.

| |Result (toml-test v1.5.0)|

|-|-----------------------|

|toml|[spec/array-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/array-0.toml) Not a homogeneous array (line 8 column 1 char 261)
[spec/keys-4.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/keys-4.toml) Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57)
[spec/local-time-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/local-time-0.toml) Parsed as unexpected data.
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46)
[datetime/local-time.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/local-time.toml) Parsed as unexpected data.
[datetime/datetime.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/datetime.toml) Parsed as unexpected data.
[comment/tricky.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/tricky.toml) Parsed as unexpected data.
[key/dotted-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-1.toml) Parsed as unexpected data.
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Found invalid character in key name: '‍'. Try quoting the key name. (line 5 column 2 char 67)
[key/dotted-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-2.toml) Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166)
[key/quoted-unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/quoted-unicode.toml) Duplicate keys! (line 3 column 1 char 19)
[key/dotted-empty.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/dotted-empty.toml) Duplicate keys! (line 2 column 1 char 17)
[key/escapes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/escapes.toml) Parsed as unexpected data.
[table/empty-name.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/table/empty-name.toml) Can't have a keygroup with an empty name (line 1 column 1 char 0)
[string/raw-multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/raw-multiline.toml) Unbalanced quotes (line 20 column 50 char 532)
[string/ends-in-whitespace-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/ends-in-whitespace-escape.toml) Reserved escape sequence used (line 6 column 1 char 28)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Reserved escape sequence used (line 3 column 1 char 35)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Reserved escape sequence used (line 1 column 1 char 0)
[string/multiline-quotes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/multiline-quotes.toml) Unterminated string found. Reached end of file. (line 27 column 1 char 664)
[float/zero.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/float/zero.toml) Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)
[array/mixed-int-string.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-string.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/nested-double.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/nested-double.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/string-with-comma-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/string-with-comma-2.toml) string index out of range
[array/mixed-int-float.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-float.toml) Not a homogeneous array (line 1 column 1 char 0)
[array/mixed-string-table.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-string-table.toml) list index out of range
[array/mixed-int-array.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/array/mixed-int-array.toml) Not a homogeneous array (line 1 column 1 char 0)
[inline-table/multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/multiline.toml) Invalid inline table value encountered (line 1 column 1 char 0)
[inline-table/key-dotted-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/key-dotted-1.toml) Parsed as unexpected data.
[inline-table/key-dotted-5.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/key-dotted-5.toml) Not a homogeneous array (line 2 column 1 char 20)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Key name found without value. Reached end of line. (line 5 column 2 char 98)
*157/187 (83.96%) passed*|

|tomli/tomli_w|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Expected newline or end of document after a statement (at line 2, column 23)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Invalid statement (at line 3, column 1)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Unescaped '\' in a string (at line 3, column 22)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Unescaped '\' in a string (at line 1, column 10)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Invalid initial character for a key part (at line 3, column 21)
*182/187 (97.33%) passed*|

|tomlkit|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Invalid number at line 2 col 25
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Empty key at line 3 col 0
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Invalid character 'x' in string at line 3 col 20
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Invalid character 'e' in string at line 1 col 8
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Empty key at line 3 col 20
*182/187 (97.33%) passed*|

|rtoml|[spec/table-9.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/table-9.toml) duplicate key: `apple` for key `fruit` at line 8 column 1
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) expected a colon, found a newline at line 2 column 26
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) unexpected character found: `\u{20ac}` at line 3 column 1
[table/array-within-dotted.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/table/array-within-dotted.toml) duplicate key: `apple` for key `fruit` at line 4 column 1
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) invalid escape character in string: `x` at line 3 column 21
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) invalid escape character in string: `e` at line 1 column 9
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) expected a table key, found a newline at line 3 column 21
*180/187 (96.26%) passed*|

|qtoml|[spec/string-4.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/string-4.toml) Didn't find expected newline (line 7, column 62)
[spec/string-7.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/spec/string-7.toml) Didn't find expected newline (line 7, column 50)
[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) can't parse type (line 2, column 20)
[datetime/milliseconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/milliseconds.toml) Didn't find expected newline (line 2, column 27)
[datetime/datetime.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/datetime.toml) Didn't find expected newline (line 4, column 18)
[comment/after-literal-no-ws.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/after-literal-no-ws.toml) can't parse type (line 1, column 4)
[comment/tricky.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/comment/tricky.toml) can't parse type (line 11, column 7)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) '€' cannot begin key (line 3, column 0)
[string/raw-multiline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/raw-multiline.toml) Didn't find expected newline (line 22, column 3)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) \x not a valid escape (line 3, column 43)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) \e not a valid escape (line 1, column 33)
[string/multiline-quotes.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/multiline-quotes.toml) Didn't find expected newline (line 4, column 26)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) ' ' cannot begin key (line 3, column 20)
*174/187 (93.05%) passed*|

|tomllib|[datetime/no-seconds.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/datetime/no-seconds.toml) Expected newline or end of document after a statement (at line 2, column 23)
[key/unicode.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/key/unicode.toml) Invalid statement (at line 3, column 1)
[string/hex-escape.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/hex-escape.toml) Unescaped '\' in a string (at line 3, column 22)
[string/escape-esc.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/string/escape-esc.toml) Unescaped '\' in a string (at line 1, column 10)
[inline-table/newline.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//valid/inline-table/newline.toml) Invalid initial character for a key part (at line 3, column 21)
*182/187 (97.33%) passed*|

### Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files

here:

> https://github.com/BurntSushi/toml-test/

- `Not OK`: The toml file is parsed without error, but expected to fail.

- `OK`: All files are failed to parse, as expected. Showing the last

parsing error.

| |Result (toml-test v1.5.0)|

|-|-----------------------|

|toml|Not OK: [integer/double-sign-plus.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/double-sign-plus.toml) incorrectly parsed.
Not OK: [integer/us-after-bin.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-bin.toml) incorrectly parsed.
Not OK: [integer/double-sign-nex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/double-sign-nex.toml) incorrectly parsed.
Not OK: [integer/us-after-hex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-hex.toml) incorrectly parsed.
Not OK: [integer/us-after-oct.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/us-after-oct.toml) incorrectly parsed.
Not OK: [spec/inline-table-2-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/inline-table-2-0.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-hour.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-hour.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/string-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/string-del.toml) incorrectly parsed.
Not OK: *63 more items incorrectly parsed.*
*298/371 (80.32%) passed*|

|tomli/tomli_w|OK: [inline-table/linebreak-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/linebreak-1.toml) Unclosed inline table (at line 3, column 18)
 *371/371 (100%) passed*|

|tomlkit|Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: [control/rawmulti-cd.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/rawmulti-cd.toml) incorrectly parsed.
Not OK: [control/bare-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/bare-cr.toml) incorrectly parsed.
Not OK: [table/append-with-dotted-keys-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-with-dotted-keys-1.toml) incorrectly parsed.
Not OK: [table/overwrite-array-in-parent.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/overwrite-array-in-parent.toml) incorrectly parsed.
Not OK: [table/append-to-array-with-dotted-keys.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-to-array-with-dotted-keys.toml) incorrectly parsed.
Not OK: [table/append-with-dotted-keys-2.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/table/append-with-dotted-keys-2.toml) incorrectly parsed.
Not OK: [array/extend-defined-aot.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/array/extend-defined-aot.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-09.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/overwrite-09.toml) incorrectly parsed.
*361/371 (97.30%) passed*|

|rtoml|Not OK: [integer/positive-hex.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-hex.toml) incorrectly parsed.
Not OK: [integer/positive-bin.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-bin.toml) incorrectly parsed.
Not OK: [integer/positive-oct.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/integer/positive-oct.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-hour.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-hour.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: [control/rawmulti-cd.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/rawmulti-cd.toml) incorrectly parsed.
Not OK: [control/bare-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/bare-cr.toml) incorrectly parsed.
*361/371 (97.30%) passed*|

|qtoml|Not OK: [spec/inline-table-2-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/inline-table-2-0.toml) incorrectly parsed.
Not OK: [spec/table-9-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/table-9-1.toml) incorrectly parsed.
Not OK: [spec/table-9-0.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/spec/table-9-0.toml) incorrectly parsed.
Not OK: [datetime/offset-overflow-minute.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/datetime/offset-overflow-minute.toml) incorrectly parsed.
Not OK: [control/comment-del.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-del.toml) incorrectly parsed.
Not OK: [control/comment-lf.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-lf.toml) incorrectly parsed.
Not OK: [control/comment-null.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-null.toml) incorrectly parsed.
Not OK: [control/comment-ff.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-ff.toml) incorrectly parsed.
Not OK: [control/comment-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/comment-cr.toml) incorrectly parsed.
Not OK: [control/multi-cr.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/control/multi-cr.toml) incorrectly parsed.
Not OK: *14 more items incorrectly parsed.*
*347/371 (93.53%) passed*|

|tomllib|OK: [inline-table/linebreak-1.toml](https://github.com/BurntSushi/toml-test/blob/v1.5.0/tests//invalid/inline-table/linebreak-1.toml) Unclosed inline table (at line 3, column 18)
 *371/371 (100%) passed*|

### Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11)

for valid toml files here:

> https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether

loading the toml file yields the same result as the JSON counterpart.

| |Result (cpython tag 3.12.4)|

|-|-----------------------|

|toml|[apostrophes-in-literal-string.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/apostrophes-in-literal-string.toml) Unbalanced quotes (line 1 column 50 char 49)
[five-quotes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/five-quotes.toml) Unterminated string found. Reached end of file. (line 7 column 1 char 97)
[dates-and-times/datetimes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/dates-and-times/datetimes.toml) Parsed as unexpected data.
[multiline-basic-str/ends-in-whitespace-escape.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/multiline-basic-str/ends-in-whitespace-escape.toml) Reserved escape sequence used (line 6 column 1 char 28)
*8/12 (66.67%) passed*|

|tomli/tomli_w|OK, *12/12 (100%) passed*|

|tomlkit|OK, *12/12 (100%) passed*|

|rtoml|OK, *12/12 (100%) passed*|

|qtoml|[apostrophes-in-literal-string.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/apostrophes-in-literal-string.toml) Didn't find expected newline (line 3, column 3)
[five-quotes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/five-quotes.toml) Didn't find expected newline (line 3, column 3)
[dates-and-times/datetimes.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//valid/dates-and-times/datetimes.toml) Didn't find expected newline (line 1, column 19)
*9/12 (75.00%) passed*|

|tomllib|OK, *12/12 (100%) passed*|

### Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11)

for invalid toml files here:

> https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

- `Not OK`: The toml file is parsed without error, but expected to fail.

- `OK`: All files are failed to parse, as expected. Showing the last

parsing error.

| |Result (cpython tag 3.12.4)|

|-|-----------------------|

|toml|Not OK: [invalid-comment-char.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/invalid-comment-char.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [array/unclosed-empty.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/unclosed-empty.toml) incorrectly parsed.
Not OK: [array/file-end-after-val.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/file-end-after-val.toml) incorrectly parsed.
Not OK: [array/unclosed-after-item.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array/unclosed-after-item.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-value-in-inner-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-value-in-inner-table.toml) incorrectly parsed.
Not OK: [inline-table/unclosed-empty.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/unclosed-empty.toml) incorrectly parsed.
*41/50 (82.00%) passed*|

|tomli/tomli_w|OK: [inline-table/overwrite-implicitly.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-implicitly.toml) Cannot overwrite a value (at line 1, column 21)
 *50/50 (100%) passed*|

|tomlkit|Not OK: [array-of-tables/overwrite-array-in-parent.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/array-of-tables/overwrite-array-in-parent.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-aot.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-aot.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [inline-table/override-val-in-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/override-val-in-table.toml) incorrectly parsed.
*44/50 (88.00%) passed*|

|rtoml|Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
*49/50 (98.00%) passed*|

|qtoml|Not OK: [non-scalar-escaped.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/non-scalar-escaped.toml) incorrectly parsed.
Not OK: [invalid-comment-char.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/invalid-comment-char.toml) incorrectly parsed.
Not OK: [table/redefine-2.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/table/redefine-2.toml) incorrectly parsed.
Not OK: [table/redefine-1.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/table/redefine-1.toml) incorrectly parsed.
Not OK: [multiline-basic-str/carriage-return.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/multiline-basic-str/carriage-return.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table.toml) incorrectly parsed.
Not OK: [dotted-keys/extend-defined-table-with-subtable.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/dotted-keys/extend-defined-table-with-subtable.toml) incorrectly parsed.
Not OK: [inline-table/overwrite-value-in-inner-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-value-in-inner-table.toml) incorrectly parsed.
Not OK: [inline-table/override-val-with-table.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/override-val-with-table.toml) incorrectly parsed.
*41/50 (82.00%) passed*|

|tomllib|OK: [inline-table/overwrite-implicitly.toml](https://github.com/python/cpython/tree/v3.12.4/Lib/test/test_tomllib/data//invalid/inline-table/overwrite-implicitly.toml) Cannot overwrite a value (at line 1, column 21)
 *50/50 (100%) passed*|

### Running speed with data provided by `rtoml`

Test the speed of loading and dumping the loaded using data

provided by `rtoml`

> https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

| |Loading speed|Dumping speed|

|-|-|-|

|toml|Excluded (heterogeneous arrays not supported)|Excluded (heterogeneous arrays not supported)|

|tomli/tomli_w|2.14s (5000 iterations)|0.73s (5000 iterations)|

|tomlkit|39.78s (5000 iterations)|0.98s (5000 iterations)|

|rtoml|0.37s (5000 iterations)|0.08s (5000 iterations)|

|qtoml|4.99s (5000 iterations)|1.87s (5000 iterations)|

|tomllib|2.04s (5000 iterations)|Dumping not supported|

### Running speed with data provided by `tomli`

Test the speed of loading and dumping the loaded using data

provided by `tomli`

> https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

| |Loading speed|Dumping speed|

|-|-|-|

|toml|Excluded (heterogeneous arrays not supported)|Excluded (heterogeneous arrays not supported)|

|tomli/tomli_w|1.41s (5000 iterations)|0.46s (5000 iterations)|

|tomlkit|24.55s (5000 iterations)|0.51s (5000 iterations)|

|rtoml|0.32s (5000 iterations)|0.16s (5000 iterations)|

|qtoml|3.63s (5000 iterations)|1.25s (5000 iterations)|

|tomllib|1.44s (5000 iterations)|Dumping not supported|

## Other reports

- [Tests with `toml-test` v1.4.0](./reports/with_toml-test_v1.4.0.md)

- [Tests with `toml-test` v1.3.0](./reports/with_toml-test_v1.3.0.md)

- [Tests with `toml-test` v1.2.0](./reports/with_toml-test_v1.2.0.md)

## Run your own report

### Install

```shell

pip install -U toml-bench

```

### Generate your own report

```shell

toml-bench

```

#### Use a different data directory than the default one

```shell

toml-bench --datadir /tmp/toml-bench

```

#### Write the report to a markdown file

```shell

toml-bench --report ./README.md

```

#### Test with a different version of compliance set (`BurntSushi/toml-test`)

```shell

toml-bench --comver 1.0.0

```

#### Use a different number of iterations in speed tests

```shell

toml-bench --iter 5000

```

#### Test with different versions of packages

```shell

git clone https://github.com/pwwang/toml-bench.git

cd toml-bench

# See https://python-poetry.org/docs/cli/#add

# for how to specify a version constraint

poetry add "tomli=2.0.0"

poetry update

poetry install

poetry run toml-bench

```

[1]: https://img.shields.io/librariesio/release/pypi/toml-bench?style=flat-square

[2]: https://libraries.io/github/pwwang/toml-bench#repository_dependencies