https://github.com/ml5js/ml5-extra-imagesegmentation
https://github.com/ml5js/ml5-extra-imagesegmentation
Last synced: about 1 month ago
JSON representation
- Host: GitHub
- URL: https://github.com/ml5js/ml5-extra-imagesegmentation
- Owner: ml5js
- Created: 2024-08-20T09:06:25.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-10T13:44:00.000Z (over 1 year ago)
- Last Synced: 2025-09-26T00:46:36.475Z (9 months ago)
- Language: JavaScript
- Size: 393 KB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## ml5-extra-imagesegmentation
This is an experiment in incorporating [transformers.js](https://xenova.github.io/transformers.js/) into [ml5.js](https://github.com/ml5js) for use with [p5.js](https://github.com/processing/p5.js). Done by [@gohai](https://github.com/gohai), [@chanel1130](https://github.com/chanel1130), [@Lisa-HuangZijin](https://github.com/Lisa-HuangZijin) and [@Ricci-Liu](https://github.com/Ricci-Liu) in Summer of 2024.
We're making use of the [detr-resnet-50-panoptic](https://huggingface.co/Xenova/detr-resnet-50-panoptic) image segmentation model here, which we found to provide acceptable performance in the browser (albeit not quite fast enough to run it continuously from a webcam).
### Examples
* [Basic](https://editor.p5js.org/gohai/sketches/O2GfdTOEL)
* [Webcam Collage](https://editor.p5js.org/gohai/sketches/Rj4yMiY59) by Chanel
* [Webcam Drag & Drop Storytelling](https://editor.p5js.org/gohai/sketches/kmWX0BWG7) by Lisa
* [Webcam Grid Storytelling](https://editor.p5js.org/gohai/sketches/DHXNvA4Ix) by Lisa
* [Webcam Parallax](https://editor.p5js.org/gohai/sketches/KoI-eYYkW) by Ruiqi
### Usage
In the head section of your `index.html`, add this line *after* the line that loads ml5.js:
```
````
Load the image segmenter like so:
```
let segmentation;
function preload() {
segmentation = ml5.imageSegmentation({ feature_extractor_size: 256 });
}
```
Omitting the `feature_extractor_size` option will improve the accuracy over speed.
To start a detection, pass an image or video to the `detect()` method, together with the name of a callback function.
```
segmentation.detect(img, gotResults);
```
Once the segmentation has finished, the callback function is called with an array of segments as argument.
```
function gotResults(results) {
for (let i=0; i < results.length; i++) {
result = results[i];
console.log(result);
}
}
```
Each result has the following properties: `label`, `score` as well as a `mask` image. The `mask` is ready to be used with p5's [mask()](https://p5js.org/reference/p5.Image/mask/#:~:text=mask()%20uses%20another%20p5,the%20mask%20will%20be%20scaled.) method.
### Acknowledgements
Many thanks to [@xenova](https://github.com/xenova) - first and foremost for transformers.js, but also for kindly providing feedback during the research for this project!