https://github.com/ifak/fences
Generate samples for various schemas like json schema and regex
https://github.com/ifak/fences
grammar json regex schema testing verification xml
Last synced: over 1 year ago
JSON representation
Generate samples for various schemas like json schema and regex
- Host: GitHub
- URL: https://github.com/ifak/fences
- Owner: ifak
- License: mit
- Created: 2023-02-09T22:51:25.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-13T15:50:43.000Z (about 2 years ago)
- Last Synced: 2024-04-14T05:02:02.801Z (about 2 years ago)
- Topics: grammar, json, regex, schema, testing, verification, xml
- Language: Python
- Homepage:
- Size: 209 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Fences
[](https://github.com/ifak/fences/actions/workflows/check.yml)
Fences is a python tool which lets you create test data based on schemas.
For this, it generates a set of *valid samples* which fullfil your schema.
Additionally, it generates a set of *invalid samples* which intentionally violate your schema.
You can then feed these samples into your software to test.
If your software is implemented correctly, it must accept all valid samples and reject all invalid ones.
Unlike other similar tools, fences generate samples systematically instead of randomly.
This way, the valid / invalid samples systematically cover all boundaries of your input schema (like placing *fences*, hence the name).
## Installation
Use pip to install Fences:
```
python -m pip install fences
```
Fences is a self contained library without any external dependencies.
It uses [Lark](https://github.com/lark-parser/lark) for regex parsing, but in the standalone version where a python file is generated from the grammar beforehand (Mozilla Public License, v. 2.0).
## Usage
### Regular Expressions
Generate samples for regular expressions:
```python
from fences import parse_regex
graph = parse_regex("a?(c+)b{3,7}")
for i in graph.generate_paths():
sample = graph.execute(i.path)
print("Valid:" if i.is_valid else "Invalid:")
print(sample)
```
Output
```
Valid:
cbbb
Valid:
acccbbbbbbb
```
### JSON Schema
Generate samples for json schema:
```python
from fences import parse_json_schema
import json
graph = parse_json_schema({
'properties': {
'foo': {
'type': 'string'
},
'bar': {
'type': 'boolean'
}
}
})
for i in graph.generate_paths():
sample = graph.execute(i.path)
print("Valid:" if i.is_valid else "Invalid:")
print(json.dumps(sample, indent=4))
```
Output
```json
Valid:
{
"foo": "",
"bar": true
}
Valid:
{}
Valid:
{
"foo": "",
"bar": false
}
Valid:
""
Valid:
[
"string"
]
Valid:
[
42
]
Valid:
[
null
]
Valid:
[
true
]
Valid:
[
false
]
Valid:
[
{}
]
Valid:
[
[]
]
Valid:
true
Valid:
false
Valid:
0
Valid:
null
Invalid:
{
"foo": 42
}
Invalid:
{
"foo": null
}
Invalid:
{
"foo": true,
"bar": true
}
Invalid:
{
"foo": false
}
Invalid:
{
"foo": {},
"bar": true
}
Invalid:
{
"foo": []
}
Invalid:
{
"bar": "string"
}
Invalid:
{
"bar": 42
}
Invalid:
{
"bar": null
}
Invalid:
{
"bar": {}
}
Invalid:
{
"bar": []
}
```
### XML Schema
Generate samples for XML schema:
```python
from fences import parse_xml_schema
from xml.etree import ElementTree
from xml.dom import minidom
et = ElementTree.fromstring("""
""")
graph = parse_xml_schema(et)
for i in graph.generate_paths():
sample = graph.execute(i.path)
s = ElementTree.tostring(sample.getroot())
print("Valid:" if i.is_valid else "Invalid:")
print(minidom.parseString(s).toprettyxml(indent=" "))
```
Output
```xml
Valid:
Valid:
foo
foo
foo
780
Valid:
x
x
x
780
Invalid:
foo
foo
foo
-10
Invalid:
x
x
x
foo
Invalid:
foo
foo
foo
780
Invalid:
x
x
x
780
```
### Grammar
Generate samples for a grammar:
```python
from fences.grammar.types import NonTerminal, CharacterRange
from fences import parse_grammar
number = NonTerminal("number")
integer = NonTerminal("integer")
fraction = NonTerminal("fraction")
exponent = NonTerminal("exponent")
digit = NonTerminal("digit")
digits = NonTerminal("digits")
one_to_nine = NonTerminal("one_to_nine")
sign = NonTerminal("sign")
grammar = {
number: integer + fraction + exponent,
integer: digit
| one_to_nine + digits
| '-' + digit
| '-' + one_to_nine + digits,
digit: '0'
| one_to_nine,
digits: digit*(1, None),
one_to_nine: CharacterRange('1', '9'),
fraction: ""
| "." + digits,
exponent: ""
| 'E' + sign + digits
| "e" + sign + digits,
sign: ["", "+", "-"]
}
graph = parse_grammar(grammar, number)
for i in graph.generate_paths():
sample = graph.execute(i.path)
print(sample)
```
Output
```
0
91.0901E0901
-0e+9
-10901.0
9E-0109
```
### OpenAPI (Swagger)
You can use Fences to parse an OpenAPI specification and generate a set of sample requests:
```python
from fences.open_api.generate import generate_all, SampleCache
from fences.open_api.open_api import OpenApi
description = {
'info': {
'title': 'Video API'
},
'paths': {
'/videos': {
'get': {
'operationId': 'listVideos',
'parameters': [{
'name': 'type',
'in': 'query',
'schema': {
'enum': ['public', 'private']
}
}, {
'name': 'title',
'in': 'query',
'schema': {
'type': 'string',
'minLength': 3
}
}],
'responses': {}
}
},
'/videos/{videoId}': {
'patch': {
'operationId': 'updateVideo',
'parameters': [
{
'name': 'videoId',
'in': 'path',
'schema': {
'type': 'number'
}
}
],
'requestBody': {
'content': {
'application/json': {
'schema': {
'type': 'object',
'properties': {
'title': {
'type': 'string',
'minLength': 10
}
},
'required': ['title']
}
}
}
},
'responses': {}
}
}
}
}
open_api = OpenApi.from_dict(description)
sample_cache = SampleCache()
for operation in open_api.operations.values():
graph = generate_all(operation, sample_cache)
for i in graph.generate_paths():
request = graph.execute(i.path)
request.dump()
```
Output
```
GET /videos
GET /videos?type=public&title=xxx
GET /videos?type=private
GET /videos?type=%23%23%23%23%23%23%23%23&title=xxx
PATCH /videos/0
BODY: {"title": "xxxxxxxxxx"}
PATCH /videos/0
PATCH /videos/0
BODY: {"title": 42}
PATCH /videos/0
BODY: {"title": null}
PATCH /videos/0
BODY: {"title": true}
PATCH /videos/0
BODY: {"title": false}
PATCH /videos/0
BODY: {"title": {}}
PATCH /videos/0
BODY: {"title": []}
PATCH /videos/0
BODY: {}
PATCH /videos/0
BODY: "string"
PATCH /videos/0
BODY: 42
PATCH /videos/0
PATCH /videos/0
BODY: true
PATCH /videos/0
BODY: false
PATCH /videos/0
BODY: []
```
You can execute the generated tests using the `request.execute()` method.
Please note, that you need to install the `requests` library for this.
## Real-World Examples
Find some real-world examples in the `examples` folder.
## Limitations
General:
Fences does not check if your schema is syntactically correct.
Fences is designed to be as permissive as possible when parsing a schema but will complain if there is an aspect it does not understand.
For XML:
Python's default XML implementation `xml.etree.ElementTree` has a very poor support for namespaces (https://docs.python.org/3/library/xml.etree.elementtree.html#parsing-xml-with-namespaces).
This might lead to problems when using the `targetNamespace` attribute in your XML schema.
For Grammars:
Fences currently does not generate invalid samples for grammars.
For OpenAPI:
The test cases generated by Fences are purely syntactic. They do not check for semantics, e.g. if retrieving a deleted resource returns 404.