Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/apple/ml-hierarchical-confusion-matrix
Neo: Hierarchical Confusion Matrix Visualization (CHI 2022)
https://github.com/apple/ml-hierarchical-confusion-matrix
confusion-matrix data-visualization machine-learning visual-analytics
Last synced: about 1 month ago
JSON representation
Neo: Hierarchical Confusion Matrix Visualization (CHI 2022)
- Host: GitHub
- URL: https://github.com/apple/ml-hierarchical-confusion-matrix
- Owner: apple
- License: other
- Created: 2022-03-22T03:02:11.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-08-01T20:07:08.000Z (about 2 months ago)
- Last Synced: 2024-08-01T22:29:01.129Z (about 2 months ago)
- Topics: confusion-matrix, data-visualization, machine-learning, visual-analytics
- Language: TypeScript
- Homepage: https://apple.github.io/ml-hierarchical-confusion-matrix/
- Size: 2.03 MB
- Stars: 299
- Watchers: 16
- Forks: 29
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
Awesome Lists containing this project
README
# Neo: Hierarchical Confusion Matrix
[](https://www.npmjs.com/package/@apple/hierarchical-confusion-matrix)
The confusion matrix, a ubiquitous visualization for helping people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. Neo is a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications.
This code accompanies the research paper:
**[Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels](https://machinelearning.apple.com/research/generalizing-confusion-matrix)**
Jochen Görtler, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, Donghao Ren, Rahul Nair, Marc Kirchner, Kayur Patel
_ACM Conference on Human Factors in Computing Systems (CHI), 2022._
[
Paper](https://arxiv.org/abs/2110.12536), [
Live demo](https://apple.github.io/ml-hierarchical-confusion-matrix/), [
Video](https://www.youtube.com/watch?v=LmsJJDHfGlI), [
Video Preview](https://www.youtube.com/watch?v=8ZxvsLPIF_Q), [
Code](https://github.com/apple/ml-hierarchical-confusion-matrix/)
## Documentation
You can embed our confusion matrix visualization into your own project. There are two ways to use it.
### NPM
Install with `npm install --save @apple/hierarchical-confusion-matrix` or `yarn add @apple/hierarchical-confusion-matrix`.
Then you can import the module in your project
```js
import confMat from '@apple/hierarchical-confusion-matrix';
const spec = {
classes: ['root'],
};
const confusions = [
{
actual: ['root:a'],
observed: ['root:a'],
count: 1,
},
{
actual: ['root:a'],
observed: ['root:b'],
count: 2,
},
{
actual: ['root:b'],
observed: ['root:a'],
count: 3,
},
{
actual: ['root:b'],
observed: ['root:b'],
count: 4,
},
];
confMat.embed('matContainer', spec, confusions);
```
### Embed the Compiled File
If you prefer to load the compiled JavaScript directly, you have to compile it. To do this, run `yarn install` and copy the `public/confMat.js` into your project. Here is a simple example of a small confusion matrix:
```html
Neo: Hierarchical Confusion Matrix
const spec = {
classes: ['root'],
};
const confusions = [
{
actual: ['root:a'],
observed: ['root:a'],
count: 1,
},
{
actual: ['root:a'],
observed: ['root:b'],
count: 2,
},
{
actual: ['root:b'],
observed: ['root:a'],
count: 3,
},
{
actual: ['root:b'],
observed: ['root:b'],
count: 4,
},
];
confMat.embed('matContainer', spec, confusions);
```
### Specification
You can find all the options that you can pass via the `spec` argument in [`src/specification.ts`](src/specification.ts).
### Loaders
The different loaders can be found in [`src/loaders`](src/loaders), which include loading data from `json`, `csv`, `vega`, and a synthetic example `synth` for testing.
### Confusion Data Format Examples
#### Example 1: Conventional Confusions
The confusions for data with `actual` labels of `fruit:lemon` that are incorrectly predicted as `fruit:apple`, of which there are `count` 1 of them.
```json
{
"actual": ["fruit:lemon"],
"observed": ["fruit:apple"],
"count": 1
}
```
#### Example 2: Hierarchical Confusions
The confusions for hierarchical data with `actual` labels of `fruit:citrus:lemon` that are incorrectly predicted as `fruit:pome:apple`, of which there are `count` 2 of them. Note `:` denotes hierarchies.
```json
{
"actual": ["fruit:citrus:lemon"],
"observed": ["fruit:pome:apple"],
"count": 2
}
```
#### Example 3: Multi-output Confusions
The confusions for multi-output data with `actual` labels of `fruit:lemon,taste:sweet` that are incorrectly predicted as `fruit:apple,taste:sour`, of which there are `count` 3 of them. Note `,` denotes multi-ouput labels.
```json
{
"actual": ["fruit:lemon", "taste:sweet"],
"observed": ["fruit:apple", "taste:sour"],
"count": 3
}
```
#### Example 4: Hierarchical and Multi-output Confusions
The confusions for hierarchical and multi-output data with `actual` labels of `fruit:citrus:lemon,taste:sweet,ripeness:ripe` that are incorrectly predicted as `fruit:pome:apple,taste:sour,ripeness:not-ripe`, of which there are `count` 4 of them.
```json
{
"actual": [
"fruit:citrus:lemon",
"taste:sweet",
"ripeness:ripe"
],
"observed": [
"fruit:pome:apple",
"taste:sour"
"ripeness:not-ripe"
],
"count": 4
}
```
See [`fruit.json`](public/data/fruit.json) for a complete example of confusions for a hierarchical fruit, taste, and ripeness classification model.
## Development
Build:
```sh
yarn install
yarn build
```
Test:
```sh
yarn test:unit
```
Dev Server:
```sh
yarn dev
```
Lint & Fix:
```sh
yarn lint
```
## Contributing
When making contributions, refer to the [`CONTRIBUTING`](CONTRIBUTING.md) guidelines and read the [`CODE OF CONDUCT`](CODE_OF_CONDUCT.md).
## BibTeX
To cite our paper, please use:
```bibtex
@inproceedings{goertler2022neo,
title={Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels},
author={Görtler, Jochen and Hohman, Fred and Moritz, Dominik and Wongsuphasawat, Kanit and Ren, Donghao and Nair, Rahul and Kirchner, Marc and Patel, Kayur},
booktitle={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},
year={2022},
organization={ACM},
doi={10.1145/3491102.3501823}
}
```
## License
This code is released under the [`LICENSE`](LICENSE) terms.