{"id":13671451,"url":"https://github.com/elliotchance/sqltest","last_synced_at":"2025-10-26T19:03:19.091Z","repository":{"id":45981914,"uuid":"84424258","full_name":"elliotchance/sqltest","owner":"elliotchance","description":"📝 A comprehensive suite of SQL tests for testing the conformance of databases.","archived":false,"fork":false,"pushed_at":"2024-05-29T05:53:25.000Z","size":249,"stargazers_count":48,"open_issues_count":2,"forks_count":17,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-07T19:41:57.911Z","etag":null,"topics":["sql","testing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elliotchance.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-09T09:36:51.000Z","updated_at":"2024-10-31T06:24:50.000Z","dependencies_parsed_at":"2022-07-18T12:48:01.338Z","dependency_job_id":null,"html_url":"https://github.com/elliotchance/sqltest","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotchance%2Fsqltest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotchance%2Fsqltest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotchance%2Fsqltest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elliotchance%2Fsqltest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elliotchance","download_url":"https://codeload.github.com/elliotchance/sqltest/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224722145,"owners_count":17358670,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["sql","testing"],"created_at":"2024-08-02T09:01:10.027Z","updated_at":"2025-10-26T19:03:19.006Z","avatar_url":"https://github.com/elliotchance.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"The goal of this project is to develop a comprehensive suite of SQL tests, based\non the each of the SQL standards to be able to test to conformance of individual\nSQL databases and engines.\n\n[**View the results here**](https://elliotchance.github.io/sqltest/)\n\nThe [latest SQL standard](https://www.iso.org/standard/63556.html) is **not\nfree** and the licence does not allow all or parts of it to be published. Older\nversions are either out of licence, or they don't mind, here is the\n[SQL-92 standard](https://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt).\n\n\nHow It Works\n============\n\nThere's a lot to explain, so here is a quick overview:\n\n1. Extract all of the BNF from the SQL standard PDF document.\n2. We use this syntax defintion with `bnf.py` to produce comprehensive tests.\n3. Each of the standards features are made up of one or more of these templates\n   that produce many SQL tests, *automagically*.\n4. The tests are run against a database and a pretty HTML report is produced.\n\nIn a nutshell, let's look at the feature [E011-02](https://github.com/elliotchance/sqltest/blob/master/standards/2016/E/E011-02.yml). When reading\nthe top secret SQL standard document we come up with 3 base tests:\n\n```yml\nsql: CREATE TABLE TN ( A \u003capproximate numeric type\u003e )\noverride:\n  precision: 2\n---\nsql: SELECT \u003csign\u003e 7.8\n---\nsql: SELECT { \u003csigned numeric literal\u003e | \u003cunsigned numeric literal\u003e }\noverride:\n  digit: 2\n```\n\nWhen running the suite these automatically expand into\n[70 individual tests](https://github.com/elliotchance/sqltest/blob/master/standards/2016/E/E011-02.tests.yml)\nthat are executed against the actual database to produce the final report.\n\n\nProgress\n========\n\nAlmost all of the mandatory features of the 2016 SQL standard have had tests\nwritten for it, but there is a lot more work to be done.\n\n\nIn More Detail\n==============\n\nThe SQL 2016, Part 2 is a 1,732 page PDF document. The document contains many\nindividual definitions of syntax described in\n[Backus–Naur form (BNF)](https://en.wikipedia.org/wiki/Backus–Naur_form). All of\nthese BNF rules are extracted from the PDF into a single file called\n[bnf.txt](https://github.com/elliotchance/sqltest/blob/master/standards/2016/bnf.txt).\n\nWe can use the BNF syntax *backwards* to generate combinations of valid SQL. A\ncustom tool -\n[bnf.py](https://github.com/elliotchance/sqltest/blob/master/bnf.py) has been\ndeveloped for this reason. It contains a few cool features but its main job is\nto output SQL from the BNF file.\n\nFor example:\n\n```bash\npython bnf.py standards/2016/bnf.txt --paths 'A \u003ccomp op\u003e { B | 5 }'\n```\n\nProduces:\n\n```\nA \u003c 5\nA \u003c B\nA \u003c= 5\nA \u003c= B\nA \u003c\u003e 5\nA \u003c\u003e B\nA = 5\nA = B\nA \u003e 5\nA \u003e B\nA \u003e= 5\nA \u003e= B\n```\n\nThis becomes especially useful when there is complex nesting of rules, we can\nsee the rule defintions for `\u003csigned numeric literal\u003e` and\n`\u003cunsigned numeric literal\u003e` by using the command:\n\n```bash\npython bnf.py standards/2016/bnf.txt --rule 'signed numeric literal' 'unsigned numeric literal' --subrules\n```\n\nWhich produces:\n\n```bnf\n\u003capproximate numeric literal\u003e ::=\n    \u003cmantissa\u003e E \u003cexponent\u003e\n\n\u003cdigit\u003e ::=\n    0\n  | 1\n  | 2\n  | 3\n  | 4\n  | 5\n  | 6\n  | 7\n  | 8\n  | 9\n\n\u003cexact numeric literal\u003e ::=\n    \u003cunsigned integer\u003e [ \u003cperiod\u003e [ \u003cunsigned integer\u003e ] ]\n  | \u003cperiod\u003e \u003cunsigned integer\u003e\n\n\u003cexponent\u003e ::=\n    \u003csigned integer\u003e\n\n\u003cmantissa\u003e ::=\n    \u003cexact numeric literal\u003e\n\n\u003cminus sign\u003e ::=\n    -\n\n\u003cperiod\u003e ::=\n    .\n\n\u003cplus sign\u003e ::=\n    +\n\n\u003csign\u003e ::=\n    \u003cplus sign\u003e\n  | \u003cminus sign\u003e\n\n\u003csigned integer\u003e ::=\n    [ \u003csign\u003e ] \u003cunsigned integer\u003e\n\n\u003csigned numeric literal\u003e ::=\n    [ \u003csign\u003e ] \u003cunsigned numeric literal\u003e\n\n\u003cunsigned integer\u003e ::=\n    \u003cdigit\u003e...\n\n\u003cunsigned numeric literal\u003e ::=\n    \u003cexact numeric literal\u003e\n  | \u003capproximate numeric literal\u003e\n\n\u003capproximate numeric literal\u003e ::=\n    \u003cmantissa\u003e E \u003cexponent\u003e\n\n\u003cdigit\u003e ::=\n    0\n  | 1\n  | 2\n  | 3\n  | 4\n  | 5\n  | 6\n  | 7\n  | 8\n  | 9\n\n\u003cexact numeric literal\u003e ::=\n    \u003cunsigned integer\u003e [ \u003cperiod\u003e [ \u003cunsigned integer\u003e ] ]\n  | \u003cperiod\u003e \u003cunsigned integer\u003e\n\n\u003cexponent\u003e ::=\n    \u003csigned integer\u003e\n\n\u003cmantissa\u003e ::=\n    \u003cexact numeric literal\u003e\n\n\u003cminus sign\u003e ::=\n    -\n\n\u003cperiod\u003e ::=\n    .\n\n\u003cplus sign\u003e ::=\n    +\n\n\u003csign\u003e ::=\n    \u003cplus sign\u003e\n  | \u003cminus sign\u003e\n\n\u003csigned integer\u003e ::=\n    [ \u003csign\u003e ] \u003cunsigned integer\u003e\n\n\u003cunsigned integer\u003e ::=\n    \u003cdigit\u003e...\n\n\u003cunsigned numeric literal\u003e ::=\n    \u003cexact numeric literal\u003e\n  | \u003capproximate numeric literal\u003e\n```\n\nTrying to generate a comprehansive set of tests from these rules manually would\nbe very difficult (and this is a very simple example). It's easy with `bnf.py`:\n\n```bash\npython bnf.py standards/2016/bnf.txt --paths 'SELECT { \u003csigned numeric literal\u003e | \u003cunsigned numeric literal\u003e }' --override 'digit=2'\n```\n\nProduces:\n\n```\nSELECT +.2\nSELECT +.2E+2\nSELECT +.2E-2\nSELECT +.2E2\nSELECT +2\nSELECT +2.\nSELECT +2.2\nSELECT +2.2E+2\nSELECT +2.2E-2\nSELECT +2.2E2\nSELECT +2.E+2\nSELECT +2.E-2\nSELECT +2.E2\nSELECT +2E+2\nSELECT +2E-2\nSELECT +2E2\nSELECT -.2\nSELECT -.2E+2\nSELECT -.2E-2\nSELECT -.2E2\nSELECT -2\nSELECT -2.\nSELECT -2.2\nSELECT -2.2E+2\nSELECT -2.2E-2\nSELECT -2.2E2\nSELECT -2.E+2\nSELECT -2.E-2\nSELECT -2.E2\nSELECT -2E+2\nSELECT -2E-2\nSELECT -2E2\nSELECT .2\nSELECT .2\nSELECT .2E+2\nSELECT .2E+2\nSELECT .2E-2\nSELECT .2E-2\nSELECT .2E2\nSELECT .2E2\nSELECT 2\nSELECT 2\nSELECT 2.\nSELECT 2.\nSELECT 2.2\nSELECT 2.2\nSELECT 2.2E+2\nSELECT 2.2E+2\nSELECT 2.2E-2\nSELECT 2.2E-2\nSELECT 2.2E2\nSELECT 2.2E2\nSELECT 2.E+2\nSELECT 2.E+2\nSELECT 2.E-2\nSELECT 2.E-2\nSELECT 2.E2\nSELECT 2.E2\nSELECT 2E+2\nSELECT 2E+2\nSELECT 2E-2\nSELECT 2E-2\nSELECT 2E2\nSELECT 2E2\n```\n\nThe `override` is important, it allows a rule (in this case `\u003cdigit\u003e`) to have a\nfixed value. Without this option we would generate *many* more cases as it would\ndo a combination of every number. Which is not important for our testing.\n\n`override` also becomes critical for rules that are recurrsive to prevent it\nfrom trying to produce an infinite amount of results.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felliotchance%2Fsqltest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felliotchance%2Fsqltest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felliotchance%2Fsqltest/lists"}