Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dailenson/One-DM
Official Code for ECCV 2024 paper — One-Shot Diffusion Mimicker for Handwritten Text Generation
https://github.com/dailenson/One-DM
computer-vision deep-learning diffusion-models handwriting-imitator handwritten-text-generation image-generation latent-diffusion pytorch-implementation
Last synced: 10 days ago
JSON representation
Official Code for ECCV 2024 paper — One-Shot Diffusion Mimicker for Handwritten Text Generation
- Host: GitHub
- URL: https://github.com/dailenson/One-DM
- Owner: dailenson
- License: mit
- Created: 2024-07-15T02:06:50.000Z (2 months ago)
- Default Branch: main
- Last Pushed: 2024-09-10T07:05:30.000Z (11 days ago)
- Last Synced: 2024-09-10T17:14:30.247Z (11 days ago)
- Topics: computer-vision, deep-learning, diffusion-models, handwriting-imitator, handwritten-text-generation, image-generation, latent-diffusion, pytorch-implementation
- Language: Python
- Homepage:
- Size: 4.56 MB
- Stars: 81
- Watchers: 5
- Forks: 4
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-diffusion-categorized - [Code
README
One-DM:One-Shot Diffusion Mimicker for Handwritten Text Generation
## 🌟 Introduction
- We propose a One-shot Diffusion Mimicker (One-DM) for stylized handwritten text generation, which only requires a single reference sample as style input, and imitates its writing style to generate handwritten text with arbitrary content.
- Previous state-of-the-art methods struggle to accurately extract a user's handwriting style from a single sample due to their limited ability to learn styles. To address this issue, we introduce the high-frequency components of the reference sample to
enhance the extraction of handwriting style. The proposed style-enhanced module can effectively capture the writing style patterns and suppress the interference of background noise.
- Extensive experiments on handwriting datasets in English, Chinese, and Japanese demonstrate that our approach with a single style reference even
outperforms previous methods with 15x-more references.
Overview of the proposed One-DM
## 🌠 Release
- [2024/9/07]🔥🔥🔥 We open-source the first version of One-DM that can generate the handwritten words. (Later versions that can support Chinese and Japanese will be released soon.)
## 🔨 Requirements
```
conda create -n One-DM python=3.8 -y
conda activate One-DM
# install all dependencies
conda env create -f environment.yml
```
## ☀️ Datasets
We provide English datasets in [Google Drive](https://drive.google.com/drive/folders/108TB-z2ytAZSIEzND94dyufybjpqVyn6) | [Baidu Netdisk](https://pan.baidu.com/s/14ESFRk0RaTr98eeLzcr_xw?pwd=4vsv). Please download these datasets, uzip them and move the extracted files to /data.
## 🐳 Model Zoo
| Model|Google Drive|Baidu Netdisk|
|---------------|---------|-----------------------------------------|
|Pretrained One-DM|[Google Drive](https://drive.google.com/drive/folders/10KOQ05HeN2kaR2_OCZNl9D_Kh1p8BDaa)|[Baidu Netdisk](https://pan.baidu.com/s/1VwckEw9TN734CirfWvZgiw?pwd=pfl8)
|Pretrained OCR model|[Google Drive](https://drive.google.com/drive/folders/10KOQ05HeN2kaR2_OCZNl9D_Kh1p8BDaa)|[Baidu Netdisk](https://pan.baidu.com/s/1VwckEw9TN734CirfWvZgiw?pwd=pfl8)
|Pretrained Resnet18|[Google Drive](https://drive.google.com/drive/folders/10KOQ05HeN2kaR2_OCZNl9D_Kh1p8BDaa)|[Baidu Netdisk](https://pan.baidu.com/s/1VwckEw9TN734CirfWvZgiw?pwd=pfl8)
**Note**:
Please download these weights, and move them to /model_zoo.
## 🏋️ Training & Test
- **training on English dataset**
```Shell
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=2 train.py \
--feat_model model_zoo/RN18_class_10400.pth \
--log English
```
- **finetune on English dataset**
```Shell
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 train_finetune.py \
--one_dm ./Saved/IAM64_scratch/English-timestamp/model/epoch-ckpt.pt \
--ocr_model ./model_zoo/vae_HTR138.pth --log English
```
**Note**:
Please modify ``timestamp`` and ``epoch`` according to your own path.
- **test on English dataset**
```Shell
CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 test.py \
--one_dm ./Saved/IAM64_finetune/English-timestamp/model/epoch-ckpt.pt \
--generate_type oov_u --dir ./Generated/English
```
**Note**:
Please modify ``timestamp`` and ``epoch`` according to your own path.
## 📺 Exhibition
- **Comparisons with industrial image generation methods on handwritten text generation**
- **Comparisons with industrial image generation methods on Chinese handwriting generation**
- **English handwritten text generation**
- **Chinese and Japanese handwriting generation**
## ❤️ Citation
If you find our work inspiring or use our codebase in your research, please cite our work:
```
@inproceedings{one-dm2024,
title={One-Shot Diffusion Mimicker for Handwritten Text Generation},
author={Dai, Gang and Zhang, Yifan and Ke, Quhui and Guo, Qiangya and Huang, Shuangping},
booktitle={European Conference on Computer Vision},
year={2024}
}
```
## ⭐ StarGraph
[](https://star-history.com/#dailenson/One-DM&Timeline)