{"id":19901081,"url":"https://github.com/theupdateframework/tuf-conformance","last_synced_at":"2025-05-02T23:31:23.792Z","repository":{"id":177559946,"uuid":"660499950","full_name":"theupdateframework/tuf-conformance","owner":"theupdateframework","description":"TUF client conformance test suite","archived":false,"fork":false,"pushed_at":"2025-04-22T08:15:37.000Z","size":6880,"stargazers_count":8,"open_issues_count":16,"forks_count":5,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-22T09:28:50.826Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theupdateframework.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-06-30T06:37:02.000Z","updated_at":"2025-04-14T07:09:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"a4edcd5a-62b3-4578-b351-a35d0de8ab1d","html_url":"https://github.com/theupdateframework/tuf-conformance","commit_stats":null,"previous_names":["jku/tuf-conformance","theupdateframework/tuf-conformance"],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theupdateframework%2Ftuf-conformance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theupdateframework%2Ftuf-conformance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theupdateframework%2Ftuf-conformance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theupdateframework%2Ftuf-conformance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theupdateframework","download_url":"https://codeload.github.com/theupdateframework/tuf-conformance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252122262,"owners_count":21698305,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T20:14:03.935Z","updated_at":"2025-05-02T23:31:23.779Z","avatar_url":"https://github.com/theupdateframework.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TUF conformance test suite\n\nThis is a conformance test suite for [TUF](https://theupdateframework.io/) clients. The goal of is to help TUF client developers\n  1. Measure the clients conformance with the [TUF specification](https://theupdateframework.github.io/specification/latest/)\n  2. Achieve better practical compatibility with other implementations\n  3. Collaborate on tests with other client developers\n\n| TUF clients known to pass the test suite |\n| --- |\n| [![python-tuf](https://img.shields.io/github/actions/workflow/status/theupdateframework/python-tuf/conformance.yml?branch=develop\u0026label=python-tuf)](https://github.com/theupdateframework/python-tuf/actions/workflows/conformance.yml?query=branch%3Adevelop) |\n| [![tuf-js](https://img.shields.io/github/actions/workflow/status/theupdateframework/tuf-js/conformance.yml?branch=main\u0026label=tuf-js)](https://github.com/theupdateframework/tuf-js/actions/workflows/conformance.yml?query=branch%3Amain) |\n| [![sigstore-ruby](https://img.shields.io/github/check-runs/sigstore/sigstore-ruby/main?nameFilter=TUF%20Ruby%203.4%20%2F%20ubuntu-latest\u0026label=sigstore-ruby)](https://github.com/sigstore/sigstore-ruby/actions/workflows/ci.yml?query=branch%3Amain) |\n| [![sigstore-java](https://img.shields.io/github/actions/workflow/status/sigstore/sigstore-java/tuf-conformance.yml?branch=main\u0026label=sigstore-java)](https://github.com/sigstore/sigstore-java/actions/workflows/conformance.yml?query=branch%3Amain) |\n\n\u003c!-- This list is based on tuf-conformance dependency graph\n(https://github.com/theupdateframework/tuf-conformance/network/dependents?package_id=UGFja2FnZS01MTQwNzI2MzMz): updates to the list are welcome.\n--\u003e\n\n_Note that a \"passing\" status may include tests that are marked as \"expected to fail\": check the clients `.xfail` file for details._\n\n## Usage\n\nThe conformance test suite provides a GitHub action that can be used to test a TUF client.\nThere are two required steps:\n\n1. Include an executable in the client project that implements the client-under-test\n   [CLI protocol](CLIENT-CLI.md).\n2. Use the `theupdateframework/tuf-conformance` action in your test workflow:\n    ```yaml\n    jobs:\n      tuf-conformance:\n        runs-on: ubuntu-latest\n        steps:\n          - uses: actions/checkout@v4\n\n          # insert possible client compilation/installation steps here\n\n          - uses: theupdateframework/tuf-conformance@v2\n            with:\n              entrypoint: path/to/my/test/executable\n    ```\n\n### Expected failures\n\nThe test suite contains tests that are not strictly specification requirements (such as\ntests for specific keytype or hash algorithm support). Clients can mark tests as \"expected failures\"\nif they do not intend to support this specific feature.\n\nThe tests that are expected to fail can be listed in `\u003centrypoint\u003e.xfails` file. In the previous\nworkflow example the xfails file would be `path/to/my/test/executable.xfails`\n\nthe `.xfails` file may contain singular tests, test groups or parameterized tests from a test group\n```\n# Comments are ok on separate lines\ntest-name\ntest-group\ntest-group[parameter]\n```\n\n## Development\n\nThis repository contains two client-under-test CLI protocol implementations\nto enable easy development and testing. The test suite depends on various\npython modules which will be installed by the make commands into a virtual environment.\nThe suite also depends on `faketime` tool which needs to be available.\n\n```bash\n# run test suite against both included clients or just one of them:\nmake test-all\nmake test-python-tuf\nmake test-go-tuf\n```\n\nInvoking the test suite manually enables more features like running a single test only\nand using a client-under-test CLI installed outside of the test suite:\n\n```bash\nmake dev\n./env/bin/pytest tuf_conformance \\\n    --entrypoint path/to/my/client-under-test/cli \\\n    -k test_unsigned_metadata\n```\n\nlinters can also be invoked with make:\n```bash\n# Run linters\nmake lint\n# Fix issues that can be automatically fixed\nmake fix\n```\n\n### Writing new tests\n\nDeveloping tests for tuf-conformance requires a basic understanding of how TUF metadata works: The\nremaining details are documented in the [Test Development doc](DEVELOPMENT.md).\n\n### Releasing\n\nChecklist for making a new tuf-conformance release\n* Review changes since last release, decide on version number\n  * If the client-under-test CLI has changed or client workflows are otherwise likely to break, bump major version\n  * otherwise if new tests were added, bump minor version\n  * otherwise bump patch version\n* Create and merge PR with README changes if needed (the workflow example contains the major version number)\n* tag the new version from a commit in main branch. Example when releasing v1.1.1:\n  ```\n      VERSION=\"v1.1.1\"\n\n      git tag --sign $VERSION -m \"$VERSION\"\n      git push origin $VERSION\n      # now rewrite the major tag: yes rewriting tags is awful, it's also what GitHub recommends...\n      MAJOR=\"${VERSION%%.*}\"\n      git tag --delete $MAJOR\n      git push --delete origin $MAJOR\n      git tag --sign $MAJOR -m \"$VERSION\"\n      git push origin $MAJOR\n  ```\n* Create GitHub release in the web UI: The release notes will be shown to action users in the dependabot update and\n  must mention all breaking changes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheupdateframework%2Ftuf-conformance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftheupdateframework%2Ftuf-conformance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftheupdateframework%2Ftuf-conformance/lists"}