https://github.com/zyddnys/dit-improved

https://github.com/zyddnys/dit-improved

Last synced: 5 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/zyddnys/dit-improved
• Owner: zyddnys
• Created: 2023-10-28T15:21:56.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-01-21T19:26:58.000Z (10 months ago)
• Last Synced: 2024-01-21T20:58:42.051Z (10 months ago)
• Language: Python
• Size: 762 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Diffusion Transformer (DiT) improved

1. Use Xpos instead of sinusoid positional embeddings

2. Use more efficient time+condition injection in [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha/tree/master)

3. Use "register" tokens https://arxiv.org/abs/2309.16588

4. Use DeepSeek MoE https://arxiv.org/abs/2401.06066

# TODO:

1. Use masked training objective

2. Use smooth latent space https://arxiv.org/abs/2312.04410

# Run

Run `cifar10_dit.py`

# Samples

From bs=256 step=156k

![a](samples/156000.png)

![a](samples/156000-proc.png)