{"id":20684613,"url":"https://github.com/gpac/compliancewarden","last_synced_at":"2025-04-22T13:24:33.777Z","repository":{"id":46023033,"uuid":"211533156","full_name":"gpac/ComplianceWarden","owner":"gpac","description":"A pluggable compliance checker (ISOBMFF, HEIF/MIAF/AVIF, AV1 HDR10+)","archived":false,"fork":false,"pushed_at":"2025-04-17T16:18:45.000Z","size":1110,"stargazers_count":18,"open_issues_count":22,"forks_count":8,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-18T06:52:47.531Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://gpac.github.io/ComplianceWarden-wasm/","language":"Assembly","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpac.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-28T16:57:05.000Z","updated_at":"2025-04-17T16:18:51.000Z","dependencies_parsed_at":"2023-10-02T20:43:24.448Z","dependency_job_id":"c12f0cd5-e002-4c95-b62e-ddc84a998ea4","html_url":"https://github.com/gpac/ComplianceWarden","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2FComplianceWarden","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2FComplianceWarden/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2FComplianceWarden/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpac%2FComplianceWarden/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpac","download_url":"https://codeload.github.com/gpac/ComplianceWarden/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250247027,"owners_count":21398993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T22:23:08.203Z","updated_at":"2025-04-22T13:24:33.766Z","avatar_url":"https://github.com/gpac.png","language":"Assembly","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Compliance Warden\n# A pluggable compliance checker (ISOBMFF, HEIF/MIAF/AVIF, AV1 HDR10+, AV1-ISOBMFF)\n\n## Introduction\n\nThe [Compliance Warden](https://github.com/gpac/ComplianceWarden), often abbreviated as \"CW\" or \"cw\" or \"the warden\", is a compliance checker.\nCW has been developed as a reference software for MPEG MIAF, AOM AVIF, and AOM AV1 HDR10+.\nIt is meant to be extended to check MP4, CMAF, and many other file formats.\n\nCW decouples the processing phases. First it parses the input to build an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) stored in very generic structures. Then it processes the AST to validate sets of rules attached to specifications. This approach offers a lot of flexibility and extensibility.\n\nCW is written in modern C++. Binary test vectors are described in assembly (x86 nasm syntax) because [why](#Test-vectors-edition) or [Why Not](https://twitter.com/daemon404/status/1301885488928878593). CW derives from a more generic effort called [Abstract](https://www.motionspell.com/compliance-testing/) started by contributors from the [GPAC](http://gpac.io) open-source project.\n\nThe Compliance Warden is distributed under the [BSD-3 permissive license](https://raw.githubusercontent.com/gpac/ComplianceWarden/master/LICENSE).\n\n## Useful information\n\n### Online version\n\nAn online version is available [here](https://gpac.github.io/ComplianceWarden-wasm/). Note that the software is executed in your browser and doesn't upload any data outside your computer.\n\n### Usage\n\n[New option parser](https://github.com/gpac/ComplianceWarden/issues/48) (introduced in July 2023):\n\n```\n$ bin/cw.exe -h\nCompliance Warden, version v32-master-rev14-g363d8d3\n\nUsage:\n    -s, --spec                              Specification name.\n    -f, --format                            Output format: \"raw\" (default), or \"json\"\n    -l, --list                              List available specifications or available rules.\n    -v, --version                           Print version and exit.\n    -h, --help                              Print usage and exit.\n    -t, --test                              Don't print warnings when switching to legacy mode.\n```\n\n\nThe old usage is deprecated and will be removed in v34:\n\n```\n$ bin/cw.exe\nCompliance Warden, version v32-master-rev14-g363d8d3\n\nUsage:\n- Run conformance:          bin/cw.exe \u003cspec\u003e input.mp4 [json]\n- List specifications:      bin/cw.exe list\n- List specification rules: bin/cw.exe \u003cspec\u003e list\n- Print version:            bin/cw.exe version\n```\n\n### Specifications\n\nCW is mainly [sponsored](#Acknowledgments) by companies and standardization groups to validate specific versions of specifications they develop or use.\n\nThe master branch only references official specifications. Draft versions or updates are meant to be in separate branches. To know more please read the [design principles](https://github.com/gpac/ComplianceWarden/issues/55#issuecomment-1517063130).\n\n\nHowever, once a specification is validated, we accept to add new rules progressively.\n\n```\n$ bin/cw.exe --list\n================================================================================\nSpecification name: av1hdr10plus\n            detail: HDR10+ AV1 Metadata Handling Specification, 7 December 2022\nhttps://github.com/AOMediaCodec/av1-hdr10plus/commit/63bacd21bc5f75ea6094fc11a03f0e743366fbdf\nhttps://aomediacodec.github.io/av1-hdr10plus/\n        depends on: \"av1isobmff\" specifications.\n================================================================================\n\n================================================================================\nSpecification name: av1isobmff\n            detail: AV1 Codec ISO Media File Format Binding v1.2.0, 12 December 2019\nhttps://github.com/AOMediaCodec/av1-isobmff/commit/ee2f1f0d2c342478206767fb4b79a39870c0827e\nhttps://aomediacodec.github.io/av1-isobmff/v1.2.0.html\n        depends on: \"isobmff\" specifications.\n================================================================================\n\n================================================================================\nSpecification name: avif\n            detail: AVIF v1.2.0, 8 January 2025\nhttps://aomediacodec.github.io/av1-avif/v1.2.0.html\n        depends on: \"miaf\" specifications.\n================================================================================\n\n================================================================================\nSpecification name: isobmff\n            detail: ISO Base Media File Format\nMPEG-4 part 12 - ISO/IEC 14496-12 - m17277 (6th+FDAM1+FDAM2+COR1-R4)\n        depends on: none.\n================================================================================\n\n================================================================================\nSpecification name: heif\n            detail: HEIF - ISO/IEC 23008-12 - 2nd Edition N18310\n        depends on: \"isobmff\" specifications.\n================================================================================\n\n================================================================================\nSpecification name: miaf\n            detail: MIAF (Multi-Image Application Format)\nMPEG-A part 22 - ISO/IEC 23000-22 - w18260 FDIS - Jan 2019\n        depends on: \"heif\" specifications.\n================================================================================\n```\n\n## Building\n\n### Prerequisites\n\n[NASM](https://nasm.us/) and a C++14 compiler.\n\n### Native build\n\nLinux, Windows:\n```\n$ make -j\n```\n\nMacOS X and BSD-likes:\n\n```\n$ CXX=scripts/darwin.sh make -j\n```\n\nor\n\n```\n$ export CXX=scripts/darwin.sh\n$ make -j\n```\n\n### Cross-compiling\n\nSimply override CXX to use your target toolchain.\n\nExample for a Windows 64 bit target:\n\n```\n$ CXX=x86_64-w64-mingw32-g++ make\n```\n\nor\n\n```\n$ export CXX=x86_64-w64-mingw32-g++\n$ make\n```\n\nor\n\n```\n$ BIN=bin_32 CXX=i686-linux-gnu-g++-12 ./check\n```\n\n### Emscripten (WASM)\n\n```\nem++ -std=c++14 -DCW_WASM bin/cw_version.cpp `find src -name '*.cpp'` -Isrc -o ComplianceWarden.js -O3 -s WASM=1 -s EXPORTED_FUNCTIONS=_specFindC,_specCheckC,_specListRulesC,_printVersion,_malloc,_free -s FORCE_FILESYSTEM=1 -s EXIT_RUNTIME=0 -s ALLOW_MEMORY_GROWTH=1 -s EXPORTED_RUNTIME_METHODS=stringToUTF8 -sASSERTIONS --pre-js scripts/wasm-fs-pre.js\n```\n\nSee https://gpac.github.io/ComplianceWarden-wasm/ for a demo.\n\nThe HTML integration source code is hosted at https://github.com/gpac/ComplianceWarden-wasm.\n\n## Testing\n\nThe Compliance Warden includes known good tests and known bad tests. This ensures the software is stable to false positives.\n\n```\n./check\n```\n\nNB: don't forget to set ```CXX``` when your toolchain requires so e.g. for Darwin (MacOS) ```CXX=scripts/darwin.sh ./check```.\n\n## Contributing\n\n### Build dependencies\n\n - GNU Bash\n - GNU g++ version 7+\n - GNU make\n - NASM 2.01+\n\n### Code formatter (optional)\n\nInstall ```clang-format```.\n\n### Pre-commit: format, build and run tests before committing\n\n```\n./check\n```\n\n### Ensure good code coverage\n\nYou need ```lcov```.\n\n```\nscripts/cov.sh\n```\n\n\u003e Note: On Darwin (MacOS) systems you may need to install GNU version of ```g++``` and ```gcov``` (e.g. ```brew install gcc```). Then change ```./scripts/darwin.sh``` to alias GNU versions instead of Clang versions.\n\n### Modifying test results\n\nThe tests (launched with ```./check```) will stop running on first error.\n\nTo update the test results, uncomment the ```# cp \"$new\" \"$ref\"``` line in the ```tests/run``` script. This avoids tests to halt when an error occurs. Please review carefully the changes before updating test results.\n\n## Code architecture\n\n### Repository file structure\n\n```\ncheck                      Top-level full-test script. Reformats + builds + tests.\n                           Must pass without error before each commit.\n\nsrc/                       Source files\n\ntests/                     Integration tests (tests calling the entry points)\ntests/run                  Entry point for the test script. Usage: \"tests/run bin\"\nscripts/cov.sh             Coverage script. Generates a coverage report reflecting the\n                           current status of the test suite,\nscripts/sanitize.sh        Runs the test suite under asan+ubsan.\n```\n\n### Principles\n\nThe Compliance Warden is made of three parts:\n - a file parser ```common_boxes.cpp``` that can be extended (or superseded) by each specification,\n - some array of rules stored in ```specs/```,\n - an application stored in ```src/app/cw.cpp``` that probes the files, launches the tests, and produces a human-readable report.\n\nThe parsing is decoupled from the rules. This allows a lot of flexibility such as:\n - the replacement of the parser by an external tool,\n - the implementation of rules in a different language.\n\nThe result of the parsing phase is comparable to an [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree). This AST is then processed by the rules.\n\nThe datastructures are generic. This allows to easily serialize them. This is useful when plugging new languages or building new tests.\n\n### Tests\n\nA test is a pair of a file format description in the [NASM syntax](https://en.wikipedia.org/wiki/Netwide_Assembler) ([example](https://raw.githubusercontent.com/gpac/ComplianceWarden/9ebfd86c392221714f42a625673536e43835938c/tests/isobmff/invalid-track-identifiers.asm)) and a reference result ([example](https://raw.githubusercontent.com/gpac/ComplianceWarden/9ebfd86c392221714f42a625673536e43835938c/tests/isobmff/invalid-track-identifiers.ref)).\n\n### Test vectors edition\n\nTest vectors are represented using some x86 assembly, in a textual human-editable form. In practice only labels and two instructions (```db``` to write 8 bits and ```dd``` to write 32 bits) are used. See this example of a ```mdat``` box containing some AV1 OBU:\n\n```\nmdat_start:\n    dd BE(mdat_end - mdat_start)\n    dd \"mdat\"\n     ; obu(0) \n    db 0x0A ; forbidden(1) obu_type(4) obu_extension_flag(1) obu_has_size_field(1) obu_reserved_1bit(1) \n    db 0x0F ; leb128_byte(8) \nmdat_end:\n```\n\nAt the time of creating the project, we couldn't find any way to create both valid and invalid editable test vectors. Annotated assembly (with symbols' names, num_bits, and values) looked like a sensible choice.\n\nThe easiest way is to create a new test vector is to derive an existing one.\n\nWhen this is not possible, one needs to disassemble an existing binary file. Contact romain.bouqueau@motionspell.com if you need some help. The assembly file needs to be stripped from its data (generally shortining radically the ```mdat``` box) and metadata (removing unused boxes and strings).\n\nThe key point is to understand that these test vectors are not intended to be valid media files. We may want to add valid samples to the tests though (e.g. retrieving files and testing them) ; in this case other tools (e.g. ```MP4Box -diso``` or ```gpac -i FILE inspect:deep:analyze=bs```) already provides some deep view of what's in the file.\n\n### Adding a test\n\nA test is a function:\n- Input is both a box tree (from the parsing phase) and a link to the report.\n- Output is written to the report: warning, errors, and ```covered()``` to assess that the rule was exercised by the input sample.\n\n```\nstruct RuleDesc\n{\n  [...]\n  // human-readable description of the rule\n  const char* caption;\n\n  // optional id from the specification\n  const char* id = nullptr;\n\n  // apply this rule to the file 'root',\n  // will push the results (messages) to the 'out' report.\n  void (* check)(Box const\u0026 root, IReport* out);\n};\n```\n\n### Adding a specification\n\nA specification is just a list of rules with some general information:\n```\nstruct SpecDesc\n{\n  // short name for the spec (used for command-line spec selection).\n  const char* name;\n\n  // human-readable description of the spec (name, version, date, etc.).\n  const char* caption;\n\n  // list of specs which this spec depends on.\n  std::vector\u003cconst char*\u003e dependencies;\n\n  // list of compliance checks for this spec.\n  std::vector\u003cRuleDesc\u003e rules;\n\n  // checks will only be executed if this returns true.\n  bool (* valid)(Box const\u0026 root) = nullptr;\n};\n```\n\n## Limitations\n\nSome aspects are not activated:\n - Brand presence checks are not fully activated. When activated, relaxed brands (e.g. 'MiPr') emit a lot of messages that bring little value to the user. Aggressive shall/should normative statements need to be balanced at standardization level.\n - Codec-level parsing is incomplete. It should be deferred in most case to an external project that can analyze both the metadata and the data (e.g. [GPAC](http://gpac.io)). This hasn't been done due to licensing concerns.\n - Some rules related to pixel formats (color spaces, ...) (computations and consistency) may only be checked by a player. Hence they are considered outside of the scope of this project.\n - Some rules related to pixel formats are only processed for AV1. Because we embed some codec-level parsing for AV1.\n - Some rules are not implemented due to missing content (e.g. Apple Audio Twos).\n\n## Acknowledgments\n\nThis work was initiated as part of the MPEG MIAF conformance software.\n\nThe [Alliance for Open Media (AOM)](https://aomedia.org/) sponsored the work on AVIF, AV1-ISOBMFF, and AV1 HDR10+.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpac%2Fcompliancewarden","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpac%2Fcompliancewarden","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpac%2Fcompliancewarden/lists"}