https://github.com/jacobmarks/concept-space-traversal-plugin
Traverse the space of concepts with a multi-modal similarity index in FiftyOne
https://github.com/jacobmarks/concept-space-traversal-plugin
clip-model computer-vision embeddings fiftyone javascript mui multimodal plugin python react
Last synced: 8 months ago
JSON representation
Traverse the space of concepts with a multi-modal similarity index in FiftyOne
- Host: GitHub
- URL: https://github.com/jacobmarks/concept-space-traversal-plugin
- Owner: jacobmarks
- Created: 2023-09-19T19:01:25.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-04T23:48:27.000Z (over 2 years ago)
- Last Synced: 2024-04-16T07:21:07.668Z (about 2 years ago)
- Topics: clip-model, computer-vision, embeddings, fiftyone, javascript, mui, multimodal, plugin, python, react
- Language: TypeScript
- Homepage:
- Size: 188 KB
- Stars: 3
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Concept Space Traversal Plugin
This plugin allows you to "traverse" the concept space of a similarity index
by adding text prompts (with configurable strength) to a base image.
**Note:** This plugin requires a similarity index that supports prompts (i.e.
embeds text and images) to be present on the dataset. You can create one with:
```py
import fiftyone as fo
import fiftyone.brain as fob
dataset = fo.load_dataset("my_dataset")
fob.compute_similarity(
dataset,
brain_key="my_brain_key",
model="clip-vit-base32-torch",
metric="cosine",
)
```
## Watch On Youtube
[](https://www.youtube.com/watch?v=XwstS_YCxoc&list=PLuREAXoPgT0RZrUaT0UpX_HzwKkoB-S9j&index=10)
## Installation
```shell
fiftyone plugins download https://github.com/jacobmarks/concept-space-traversal-plugin
```
Refer to the [main README](https://github.com/voxel51/fiftyone-plugins) for
more information about managing downloaded plugins and developing plugins
locally.
## Operators
### `open_traversal_panel`
- Opens the concept space traversal panel on click
- Only activated when the dataset has a similarity index
### `traverser`
- Runs the Traverser on the dataset
### `get_sample_url`
- Returns the resource URL of the media file associated with a sample, from the sample's ID.
## 💪 Development
This plugin was a joint creation between myself and [Ibrahim Manjra](https://github.com/imanjra). Couldn't have done it without his JavaScript wizardry!