https://github.com/shariqfarooq123/LooseControl

Lifting ControlNet for Generalized Depth Conditioning
https://github.com/shariqfarooq123/LooseControl

Last synced: about 1 month ago
JSON representation

Lifting ControlNet for Generalized Depth Conditioning

• Host: GitHub
• URL: https://github.com/shariqfarooq123/LooseControl
• Owner: shariqfarooq123
• License: mit
• Created: 2023-12-06T21:41:12.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2023-12-07T19:01:33.000Z (over 1 year ago)
• Last Synced: 2024-10-31T00:40:07.856Z (6 months ago)
• Language: Python
• Size: 3.04 MB
• Stars: 449
• Watchers: 16
• Forks: 28
• Open Issues: 16
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-diffusion-categorized - [Code

README

        
# LooseControl: Lifting ControlNet for Generalized Depth Conditioning

[![Open in Spaces](https://huggingface.co/datasets/huggingface/badges/raw/main/open-in-hf-spaces-sm.svg)](https://huggingface.co/spaces/shariqfarooq/LooseControl)

[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT) ![PyTorch](https://img.shields.io/badge/PyTorch_v1.10.1-EE4C2C?&logo=pytorch&logoColor=white) 

This is the official repository for our paper:

>#### [LooseControl: Lifting ControlNet for Generalized Depth Conditioning](#)

> ##### [Shariq Farooq Bhat](https://shariqfarooq123.github.io), [Niloy J. Mitra](http://www0.cs.ucl.ac.uk/staff/n.mitra/), [Peter Wonka](http://peterwonka.net/) 

[[Project Page]](https://shariqfarooq123.github.io/loose-control/) [[Paper]](https://arxiv.org/abs/2312.03079) [[Demo 🤗]](https://huggingface.co/spaces/shariqfarooq/LooseControl) [[Weights (3D Box Control)]](https://huggingface.co/shariqfarooq/loose-control-3dbox)

![teaser](assets/looseControl_teaser.png)

# Usage

```bash

git clone https://github.com/shariqfarooq123/LooseControl && cd LooseControl

```

Start the UI:

```python

gradio app.py

```

or use via python API:

```python

from loosecontrol import LooseControlNet

lcn = LooseControlNet("shariqfarooq/loose-control-3dbox")

boxy_depth = ...

prompt = "A photo of a snowman in a desert"

negative_prompt = "blurry, text, caption, lowquality,lowresolution, low res, grainy, ugly"

gen_image_1 = lcn(prompt, negative_prompt=negative_prompt, control_image=boxy_depth)

```

Style preserving edits:

```python

# Fix the 'style' and edit

# Edit 'boxy_depth' -> 'boxy_depth_edited'

lcn.set_cf_attention()

gen_image_edited = lcn.edit(boxy_depth, boxy_depth_edited, prompt, negative_prompt=negative_prompt)

```

# Credits

The Cross Frame attention is adapted from [Text2Video-Zero](https://github.com/Picsart-AI-Research/Text2Video-Zero)

# Citation

```bibtex

@misc{bhat2023loosecontrol,

      title={LooseControl: Lifting ControlNet for Generalized Depth Conditioning}, 

      author={Shariq Farooq Bhat and Niloy J. Mitra and Peter Wonka},

      year={2023},

      eprint={2312.03079},

      archivePrefix={arXiv},

      primaryClass={cs.CV}

}

```