https://github.com/daspartho/magicmix
Implementation of MagicMix: Semantic Mixing with Diffusion Models paper
https://github.com/daspartho/magicmix
deep-learning diffusers diffusion-models gradio huggingface-spaces machine-learning paper-implementations
Last synced: 7 months ago
JSON representation
Implementation of MagicMix: Semantic Mixing with Diffusion Models paper
- Host: GitHub
- URL: https://github.com/daspartho/magicmix
- Owner: daspartho
- License: mit
- Created: 2022-12-09T06:04:29.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-15T17:22:38.000Z (about 3 years ago)
- Last Synced: 2023-03-11T15:20:30.356Z (about 3 years ago)
- Topics: deep-learning, diffusers, diffusion-models, gradio, huggingface-spaces, machine-learning, paper-implementations
- Language: Jupyter Notebook
- Homepage: https://huggingface.co/spaces/daspartho/MagicMix
- Size: 13.6 MB
- Stars: 9
- Watchers: 3
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# MagicMix
[](https://huggingface.co/spaces/daspartho/MagicMix)
Implementation of [MagicMix: Semantic Mixing with Diffusion Models](https://arxiv.org/pdf/2210.16056.pdf) paper.
The aim of the method is to mix two different concepts in a semantic manner to synthesize a new concept while preserving the spatial layout and geometry.
The method takes an image that provides the layout semantics and a prompt that provides the content semantics for the mixing process.
There are 3 parameters for the method-
- `v`: It is the interpolation constant used in the layout generation phase. The greater the value of v, the greater the influence of the prompt on the layout generation process.
- `kmax` and `kmin`: These determine the range for the layout and content generation process. A higher value of kmax results in loss of more information about the layout of the original image and a higher value of kmin results in more steps for content generation process.
### Usage
```python
from PIL import Image
from magic_mix import magic_mix
img = Image.open('phone.jpg')
out_img = magic_mix(img, 'bed', kmax=0.5)
out_img.save("mix.jpg")
```
```
python3 magic_mix.py \
"phone.jpg" \
"bed" \
"mix.jpg" \
--kmin 0.3 \
--kmax 0.6 \
--v 0.5 \
--steps 50 \
--seed 42 \
--guidance_scale 7.5
```
Also, check out the [demo notebook](https://github.com/daspartho/MagicMix/blob/main/demo.ipynb) for example usage of the implementation to reproduce examples from the paper.
You can also use the community pipeline on the diffusers libary.
```python
from diffusers import DiffusionPipeline, DDIMScheduler
from PIL import Image
pipe = DiffusionPipeline.from_pretrained(
"CompVis/stable-diffusion-v1-4",
custom_pipeline="magic_mix",
scheduler = DDIMScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler"),
).to('cuda')
img = Image.open('phone.jpg')
mix_img = pipe(
img,
prompt = 'bed',
kmin = 0.3,
kmax = 0.5,
mix_factor = 0.5,
)
mix_img.save('mix.jpg')
```
### Some examples reproduced from the paper:
##### Input Image:
##### Prompt: "Bed"
##### Output Image:
##### Input Image:
##### Prompt: "Family"
##### Output Image:
##### Input Image:
##### Prompt: "ice-cream"
##### Output Image:
##### Input Image:
##### Prompt: "Cake"
##### Output Image:
### Note
**I'm not the author of the paper, and this is not an official implementation**