{"id":13738713,"url":"https://github.com/apple/ml-hierarchical-confusion-matrix","last_synced_at":"2025-05-16T12:12:01.961Z","repository":{"id":37494795,"uuid":"472595300","full_name":"apple/ml-hierarchical-confusion-matrix","owner":"apple","description":"Neo: Hierarchical Confusion Matrix Visualization (CHI 2022)","archived":false,"fork":false,"pushed_at":"2025-05-01T20:07:57.000Z","size":2816,"stargazers_count":310,"open_issues_count":4,"forks_count":32,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-13T02:11:45.938Z","etag":null,"topics":["confusion-matrix","data-visualization","machine-learning","visual-analytics"],"latest_commit_sha":null,"homepage":"https://apple.github.io/ml-hierarchical-confusion-matrix/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-03-22T03:02:11.000Z","updated_at":"2025-05-07T17:56:10.000Z","dependencies_parsed_at":"2023-12-01T21:24:16.742Z","dependency_job_id":"c09bd0df-b05c-421e-bd13-104dd1dd1fd7","html_url":"https://github.com/apple/ml-hierarchical-confusion-matrix","commit_stats":{"total_commits":193,"total_committers":4,"mean_commits":48.25,"dds":"0.14507772020725385","last_synced_commit":"e2bbc801879bd42933ff83b705a18f308c2b137f"},"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hierarchical-confusion-matrix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hierarchical-confusion-matrix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hierarchical-confusion-matrix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hierarchical-confusion-matrix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-hierarchical-confusion-matrix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254527099,"owners_count":22085919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confusion-matrix","data-visualization","machine-learning","visual-analytics"],"created_at":"2024-08-03T03:02:34.384Z","updated_at":"2025-05-16T12:12:01.929Z","avatar_url":"https://github.com/apple.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# Neo: Hierarchical Confusion Matrix\n\n[![npm version](https://img.shields.io/npm/v/@apple/hierarchical-confusion-matrix.svg)](https://www.npmjs.com/package/@apple/hierarchical-confusion-matrix)\n\nThe confusion matrix, a ubiquitous visualization for helping people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. Neo is a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications.\n\nThis code accompanies the research paper:\n\n**[Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels](https://machinelearning.apple.com/research/generalizing-confusion-matrix)**  \nJochen Görtler, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, Donghao Ren, Rahul Nair, Marc Kirchner, Kayur Patel  \n_ACM Conference on Human Factors in Computing Systems (CHI), 2022._  \n[\u003cimg width=\"16\" alt=\"image\" src=\"https://user-images.githubusercontent.com/5741691/166707770-eac596b5-2622-4899-9ff8-8c3037b03b3a.png\"\u003e Paper](https://arxiv.org/abs/2110.12536), [\u003cimg width=\"12\" alt=\"image\" src=\"https://user-images.githubusercontent.com/5741691/166709218-c8fa7af7-a5a4-467b-b77d-679aaf1aabda.png\"\u003e Live demo](https://apple.github.io/ml-hierarchical-confusion-matrix/), [\u003cimg width=\"18\" alt=\"image\" src=\"https://user-images.githubusercontent.com/5741691/166709344-e11b80c9-1b24-4412-bd7d-03c7e3dfaecc.png\"\u003e Video](https://www.youtube.com/watch?v=LmsJJDHfGlI), [\u003cimg width=\"18\" alt=\"image\" src=\"https://user-images.githubusercontent.com/5741691/166709459-5d0e44a5-0dbc-4679-b071-10ed0a3cdd92.png\"\u003e Video Preview](https://www.youtube.com/watch?v=8ZxvsLPIF_Q), [\u003cimg width=\"20\" alt=\"image\" src=\"https://user-images.githubusercontent.com/5741691/166706116-4d2c003f-fd68-4b75-b896-8f9a588318c1.png\"\u003e Code](https://github.com/apple/ml-hierarchical-confusion-matrix/)\n\n## Documentation\n\nYou can embed our confusion matrix visualization into your own project. There are two ways to use it.\n\n### NPM\n\nInstall with `npm install --save @apple/hierarchical-confusion-matrix` or `yarn add @apple/hierarchical-confusion-matrix`.\n\nThen you can import the module in your project\n\n```js\nimport confMat from '@apple/hierarchical-confusion-matrix';\n\nconst spec = {\n    classes: ['root'],\n};\n\nconst confusions = [\n    {\n        actual: ['root:a'],\n        observed: ['root:a'],\n        count: 1,\n    },\n    {\n        actual: ['root:a'],\n        observed: ['root:b'],\n        count: 2,\n    },\n    {\n        actual: ['root:b'],\n        observed: ['root:a'],\n        count: 3,\n    },\n    {\n        actual: ['root:b'],\n        observed: ['root:b'],\n        count: 4,\n    },\n];\n\nconfMat.embed('matContainer', spec, confusions);\n```\n\n### Embed the Compiled File\n\nIf you prefer to load the compiled JavaScript directly, you have to compile it. To do this, run `yarn install` and copy the `public/confMat.js` into your project. Here is a simple example of a small confusion matrix:\n\n```html\n\u003c!DOCTYPE html\u003e\n\u003chtml\u003e\n    \u003chead\u003e\n        \u003cmeta charset=\"utf8\" /\u003e\n        \u003cmeta name=\"viewport\" content=\"width=device-width\" /\u003e\n        \u003ctitle\u003eNeo: Hierarchical Confusion Matrix\u003c/title\u003e\n    \u003c/head\u003e\n\n    \u003cbody\u003e\n        \u003cdiv id=\"matContainer\"\u003e\u003c/div\u003e\n        \u003cscript src=\"confMat.js\"\u003e\u003c/script\u003e\n        \u003cscript\u003e\n            const spec = {\n                classes: ['root'],\n            };\n\n            const confusions = [\n                {\n                    actual: ['root:a'],\n                    observed: ['root:a'],\n                    count: 1,\n                },\n                {\n                    actual: ['root:a'],\n                    observed: ['root:b'],\n                    count: 2,\n                },\n                {\n                    actual: ['root:b'],\n                    observed: ['root:a'],\n                    count: 3,\n                },\n                {\n                    actual: ['root:b'],\n                    observed: ['root:b'],\n                    count: 4,\n                },\n            ];\n\n            confMat.embed('matContainer', spec, confusions);\n        \u003c/script\u003e\n    \u003c/body\u003e\n\u003c/html\u003e\n```\n\n### Specification\n\nYou can find all the options that you can pass via the `spec` argument in [`src/specification.ts`](src/specification.ts).\n\n### Loaders\n\nThe different loaders can be found in [`src/loaders`](src/loaders), which include loading data from `json`, `csv`, `vega`, and a synthetic example `synth` for testing.\n\n### Confusion Data Format Examples\n\n#### Example 1: Conventional Confusions\n\nThe confusions for data with `actual` labels of `fruit:lemon` that are incorrectly predicted as `fruit:apple`, of which there are `count` 1 of them.\n\n```json\n{\n    \"actual\": [\"fruit:lemon\"],\n    \"observed\": [\"fruit:apple\"],\n    \"count\": 1\n}\n```\n\n#### Example 2: Hierarchical Confusions\n\nThe confusions for hierarchical data with `actual` labels of `fruit:citrus:lemon` that are incorrectly predicted as `fruit:pome:apple`, of which there are `count` 2 of them. Note `:` denotes hierarchies.\n\n```json\n{\n    \"actual\": [\"fruit:citrus:lemon\"],\n    \"observed\": [\"fruit:pome:apple\"],\n    \"count\": 2\n}\n```\n\n#### Example 3: Multi-output Confusions\n\nThe confusions for multi-output data with `actual` labels of `fruit:lemon,taste:sweet` that are incorrectly predicted as `fruit:apple,taste:sour`, of which there are `count` 3 of them. Note `,` denotes multi-ouput labels.\n\n```json\n{\n    \"actual\": [\"fruit:lemon\", \"taste:sweet\"],\n    \"observed\": [\"fruit:apple\", \"taste:sour\"],\n    \"count\": 3\n}\n```\n\n#### Example 4: Hierarchical and Multi-output Confusions\n\nThe confusions for hierarchical and multi-output data with `actual` labels of `fruit:citrus:lemon,taste:sweet,ripeness:ripe` that are incorrectly predicted as `fruit:pome:apple,taste:sour,ripeness:not-ripe`, of which there are `count` 4 of them.\n\n```json\n{\n    \"actual\": [\n        \"fruit:citrus:lemon\",\n        \"taste:sweet\",\n        \"ripeness:ripe\"\n    ],\n    \"observed\": [\n        \"fruit:pome:apple\",\n        \"taste:sour\"\n        \"ripeness:not-ripe\"\n    ],\n    \"count\": 4\n}\n```\n\nSee [`fruit.json`](public/data/fruit.json) for a complete example of confusions for a hierarchical fruit, taste, and ripeness classification model.\n\n## Development\n\nBuild:\n\n```sh\nyarn install\nyarn build\n```\n\nTest:\n\n```sh\nyarn test:unit\n```\n\nDev Server:\n\n```sh\nyarn dev\n```\n\nLint \u0026 Fix:\n\n```sh\nyarn lint\n```\n\n## Contributing\n\nWhen making contributions, refer to the [`CONTRIBUTING`](CONTRIBUTING.md) guidelines and read the [`CODE OF CONDUCT`](CODE_OF_CONDUCT.md).\n\n## BibTeX\n\nTo cite our paper, please use:\n\n```bibtex\n@inproceedings{goertler2022neo,\n  title={Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels},\n  author={Görtler, Jochen and Hohman, Fred and Moritz, Dominik and Wongsuphasawat, Kanit and Ren, Donghao and Nair, Rahul and Kirchner, Marc and Patel, Kayur},\n  booktitle={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},\n  year={2022},\n  organization={ACM},\n  doi={10.1145/3491102.3501823}\n}\n```\n\n## License\n\nThis code is released under the [`LICENSE`](LICENSE) terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-hierarchical-confusion-matrix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-hierarchical-confusion-matrix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-hierarchical-confusion-matrix/lists"}