{"id":16681973,"url":"https://github.com/stevebuik/stateful-generative-tests","last_synced_at":"2025-04-09T23:12:04.989Z","repository":{"id":83074358,"uuid":"104813291","full_name":"stevebuik/stateful-generative-tests","owner":"stevebuik","description":"auto-generated tests for Clojure WebApps","archived":false,"fork":false,"pushed_at":"2017-09-26T06:35:55.000Z","size":25,"stargazers_count":11,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-09T23:12:00.461Z","etag":null,"topics":["clojure","clojure-spec","generative-testing","testing"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stevebuik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-09-25T23:47:46.000Z","updated_at":"2019-09-22T21:02:22.000Z","dependencies_parsed_at":"2023-03-22T04:35:47.123Z","dependency_job_id":null,"html_url":"https://github.com/stevebuik/stateful-generative-tests","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevebuik%2Fstateful-generative-tests","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevebuik%2Fstateful-generative-tests/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevebuik%2Fstateful-generative-tests/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stevebuik%2Fstateful-generative-tests/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stevebuik","download_url":"https://codeload.github.com/stevebuik/stateful-generative-tests/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248125589,"owners_count":21051770,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","clojure-spec","generative-testing","testing"],"created_at":"2024-10-12T14:05:49.534Z","updated_at":"2025-04-09T23:12:04.974Z","avatar_url":"https://github.com/stevebuik.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stateful Generative testing using Spec Models\n\nIllustrates how to use Clojure Spec to test a webapp that stores data.\n\nThis was used as a talk at Clojure Sydney Meetup on Sep 26th, 2017.\n\n## TL;DR\n\nClone and run tests using `lein test`\n\nLook at [web-crud.clj](https://github.com/stevebuik/stateful-generative-tests/blob/master/src/stateful_testing/web_crud.clj) and use the comments at the bottom to start the ring server to see the UI\n\nLook at [web-crud-tests.clj](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/web_crud_tests.clj) to see how to use generated commands to test Add/Delete in the webapp.\n\nThis test combines Kerodon, Clojure Spec and custom generators to generate valid sequences of *commands* for a web-app.\n\n## The Long Version.....\n\nGenerative testing reduces the need for example-based unit tests.\nClojure Spec takes this further by automatically providing generators to test clojure functions.\n\n**Question**: the generated data is stateless. Webapps are stateful. How can these two ideas be combined?\n\n**Answer**: search the interwebs....and find...\n\n**Stateful Generator Libraries**\n\nhttps://github.com/jstepien/states\n\nhttps://github.com/czan/stateful-check\n\n**Blog Posts**\n\n[Verifying FSMs using test.check by Guillermo Winkler](http://blog.guillermowinkler.com/blog/2015/04/12/verifying-state-machine-behavior-using-test-dot-check/)\n\n**Videos**\n\n[Customising Generators by Stu Halloway](https://www.youtube.com/watch?v=WoFkhE92fqc)\n\n[Teleport Testing by Antonio Montiero \u0026 Mike Kaplinskiy](https://www.youtube.com/watch?v=qijWBPYkRAQ)\n\nThanks to all these people for sharing such valuable work. It inspired this presentation.\n\nThe blog post provides a great explanation and sample code for stateful testing.\nIt could even be written as portable (cljc) Clojure - sweet! It would be great to see tests running in this readme.\n\nIn the blog post, the section on shrinking and Rose Trees is really interesting.\n\nStu's video demonstrates the idea of generator models to make Spec generators smarter.\nMaybe using the code from the blog as a spec model could work? Let's try.\n\n### To the REPL....\n\n(follow the links and/or run the tests in your IDE)\n\n[Experiment #1](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/states_lib_tests.clj)\n: run the sample code for the *states* library.\n\nWorks well but is not portable Clojure. Leaving this path alone for now.\n\n[Experiment #2](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/fsm_tests.clj)\n: run the FSM sample code from the blog post\n\n* observe see the two phases:\n    * cmd-seq is the generation phase\n    * prop/for-all is the application phase\n* exec fn is used in the generation phase to maintain the state.\nthis means that the generation state system is different from the state of the system under test. Could having two state implementations be a source of bugs as complexity grows?\n* the test invariant in this example is not a good example\n* Clojure Spec is not used anywhere (because the blog was written before Spec)\n\n[Experiment #3](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/fsm_tests2.clj)\n: changed the FSM sample to test a set (like the *states* test) instead of vector\n\n* added a :clear-cmd for emptying the set\n* still have different code for gen vs application phase\n* test.check invariant more like a real world example\n* deftest ensures that *true* is the result since test.check puts exceptions in the :result\n\n[Experiment #4](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/fsm_tests3.clj)\n: changed the FSM sample to use same state mgmt fn for gen and application phase\n\n* easier to read, DRY code\n* still not using Spec\n\n[Experiment #5](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/fsm_tests4.clj)\n: changed the FSM sample to use a Spec for the commands\n\n* play with the spec by running the code in the comments. compare the stateless vs the stateful generated commands\n* using a spec for the *apply-commands* fn which means that prop/for-all invariants are no longer required.\nthis is the driver fn for the generative tests.\n* uncomment the two *pprint* lines to see what was tested\n\n### Testing a Web-app instead of a Set\n\nLoad the [web-crud.clj](https://github.com/stevebuik/stateful-generative-tests/blob/master/src/stateful_testing/web_crud.clj)\nfile and run the two expressions in the comment at the bottom, then browse `http://localhost:8080/list`\nand play with the app to understand it\n\nLoad the [web-crud-tests.clj](https://github.com/stevebuik/stateful-generative-tests/blob/master/test/stateful_testing/web_crud_tests.clj)\nfile and run:\n\n1. the expressions in comments\n2. the example-based unit test\n3. the generative test\n4. try breaking it by changing the default id in the add/exec fn\n\nand observe....\n\n* Kerodon is awesome. Like a fast Selenium\n* Add commands don't include an :id since the webapp generates the id\n* Use a multi-spec since now commands have different keys\n* Using a spec'd driver fn, like in Experiment #5\n* The :ret spec for the driver fn is a map, allowing N assertions with a clear path to false values\n* It's fast! Even running 50 generated command sequences is sub-second.\n\nOriginally I used the web-app for the generation and the application phase.\nThis did not work because each generated command sequence retained state from previous sequences.\nThe solution was to go back to two systems for state, one for each phase.\n\nWhen this test is run, the number of assertions is high. This is because every CRUD operation asserts that status = 200 etc when the command is applied.\nThis is the power of generative tests, many combinations generated, applied and asserted.\n\n### Conclusions\n\nAlthough there are two good libraries for stateful testing, I prefer the blog posts solution because the generated commands are pure data (no fns as values).\nThis makes them easier to read, easier to send over a wire for remote invocation and the code could easily be portable (cljc).\n\nThe combination of Kerodon and generated commands is a Selenium killer. Happy days!\nThat said, there is no browser so Selenium is better if you are seeking cross-browser testing.\n\nThis testing technique replaces test.check with Specs and test.check underneath.\nThe amount of code is approx the same but, with Specs, you also have command DSL that can be used for other purposes\ne.g.\n* runtime request validation\n* remote command(s) execution\n\nThese are powerful benefits so testing this way is a valuable investment.\n\n### Future\n\n* A server endpoint could accept an EDN sequence of commands and run them as a live test i.e. generative Selenium\n* Single Page Apps are stateful and can be tested the same way (see the Teleport video for more)\n* Convert Set tests to portable Clojure and run using Klipse\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevebuik%2Fstateful-generative-tests","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstevebuik%2Fstateful-generative-tests","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstevebuik%2Fstateful-generative-tests/lists"}