{"id":28491530,"url":"https://github.com/line/layout-corrector","last_synced_at":"2025-07-04T23:30:34.575Z","repository":{"id":271910945,"uuid":"855643249","full_name":"line/Layout-Corrector","owner":"line","description":"Official implementation of \"Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model\" (ECCV2024)","archived":false,"fork":false,"pushed_at":"2024-11-01T09:02:01.000Z","size":1205,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-06-08T08:07:37.154Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/line.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-11T08:07:50.000Z","updated_at":"2025-06-07T10:57:28.000Z","dependencies_parsed_at":"2025-01-10T18:02:34.620Z","dependency_job_id":null,"html_url":"https://github.com/line/Layout-Corrector","commit_stats":null,"previous_names":["line/layout-corrector"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/line/Layout-Corrector","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/line%2FLayout-Corrector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/line%2FLayout-Corrector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/line%2FLayout-Corrector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/line%2FLayout-Corrector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/line","download_url":"https://codeload.github.com/line/Layout-Corrector/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/line%2FLayout-Corrector/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263635328,"owners_count":23492173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-08T08:07:38.576Z","updated_at":"2025-07-04T23:30:34.568Z","avatar_url":"https://github.com/line.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# [ECCV2024] Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model\n\n[[Paper (ECVA)]](https://www.ecva.net/papers/eccv_2024/papers_ECCV/html/4969_ECCV_2024_paper.php)\n[[Project page]](https://iwa-shi.github.io/Layout-Corrector-Project-Page/)\n[[Supplementary material]](https://www.ecva.net/papers/eccv_2024/papers_ECCV/papers/04969-supp.pdf)\n[[Video]](https://youtu.be/rk1G8GrlO3g)\n\n\nThis is the official PyTorch implementation of \n\"Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model (ECCV2024)\".\n\n[Shoma Iwai\u003csup\u003e1\u003c/sup\u003e](https://iwa-shi.github.io/), Atsuki Osanai\u003csup\u003e2\u003c/sup\u003e, [Shunsuke Kitada\u003csup\u003e2\u003c/sup\u003e](https://shunk031.me/), and [Shinichiro Omachi\u003csup\u003e1\u003c/sup\u003e](http://www.iic.ecei.tohoku.ac.jp/~machi/index-e.html)\n\n\u003csup\u003e1\u003c/sup\u003e Tohoku University, \u003csup\u003e2\u003c/sup\u003e LY Corporation\n\n\n\u003cimg src=\"assets/thumbnail.jpg\" width=\"80%\"\u003e\n\n# Setup\n\u003cdetails\u003e\n\u003csummary\u003eSetup details\u003c/summary\u003e\n\n## Making environment with pip (Recommended)\n\nThe environment is constructed under `Python 3.10`.\n\n- Run the container for Development\n\n    ```bash\n    make build\n    make run\n    --- in container ---\n    cd /app\n    pip3 install -e .\n    ```\n\n## Download Starter Kit (Datasets, Pre-trained Models, etc.)\nPlease download the [starter kit (GDrive, 952 MB)](https://drive.google.com/file/d/1og3l0enR67rDwiAN44K4RchcFYAgsbNq/view) and unzip it:\n```bash\nunzip ./layout_corrector_starter_kit.zip\n```\nFor more details about the starter kit, please refer to the README.md file included in the zip.\n\n\n### [Optional] Custom Dataset\nPubLayNet, Rico, and Crello datasets are included in our starter kit.\nFor custom datasets, see [docs/custom_dataset.md](./docs/custom_dataset.md)\n\n\u003c/details\u003e\n\n---\n\n\n# Training\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining details\u003c/summary\u003e\n\n### LayoutDM, VQDiffusion, MaskGIT\n```bash\nbash bin/train.sh \u003cDATASET\u003e \u003cEXPERIMENT_NAME\u003e \u003cADDITIONAL_ARGS\u003e\n```\n\nFor example, \n```bash\nbash bin/train.sh rico25 layoutdm seed=0,1,2\n```\n\n### Layout-Corrector\n```bash\nbash bin/corrector_train.sh \u003cDATASET\u003e \u003cEXPERIMENT_NAME\u003e \u003cDIFFUSION_JOB_DIR\u003e \u003cADDITIONAL_ARGS\u003e\n```\n- `\u003cEXPERIMENT_NAME\u003e`: Filename of .yaml file in `src/trainer/trainer/config/experiment`.\n- `\u003cDIFFUSION_JOB_DIR\u003e`: Job directory of pre-trained diffusion model (e.g., LayoutDM)\n- `\u003cADDITIONAL_ARGS\u003e`: Optional (e.g., `seed=0,1,2`)\n\nFor example, \n```bash\nbash bin/corrector_train.sh rico25 layout_corrector download/pretrained_weights/rico25/layoutdm seed=0,1,2 training.epochs=20\n```\n\n\u003c/details\u003e\n\n\n---\n\n\n# Testing\n\n\u003cdetails\u003e\n\u003csummary\u003eTesting details\u003c/summary\u003e\n\nYou can try a quick demo using `notebooks/demo.ipynb`.\n\nTo generate layouts and calculate metrics, run the following commands:\n\n### LayoutDM, VQDiffusion, MaskGIT\n```bash\npython bin/test_eval.py \u003cDIFFUSION_JOB_DIR\u003e \u003cDATASET\u003e -t \u003cTIMESTEPS\u003e [-d \u003cDEVICE_ID\u003e]\n```\n\nFor example, \n```bash\npython bin/test_eval.py tmp/jobs/rico25/layoutdm_jobdir rico25 -t 100\n```\n\n### DDM + Layout-Corrector (Ours)\n```bash\npython ./bin/corrector_test_eval.py \u003cJOB_DIR\u003e \u003cDATASET\u003e -t \u003cNUM_TIMESTEPS\u003e [-d \u003cDEVICE_ID\u003e]\n```\n- `\u003cJOB_DIR\u003e`: Path to the pre-trained corrector job_dir (not that of a generator).\n\nFor example, \n```bash\npython ./bin/corrector_test_eval.py ./download/pretrained_weights/layout_corrector rico25 -t 100\n```\n\n### Masking Methods in Layout-Corrector\n\n#### Threshold-masking (Default)\n- To specify threshold-masking, add `--corrector_mask_mode thresh --corrector_mask_threshold \u003cTHRESHOLD\u003e`:\n```bash\npython ./bin/corrector_test_eval.py ./download/pretrained_weights/rico25/layout_corrector rico25 -t 100 --corrector_mask_mode thresh --corrector_mask_threshold 0.7 \n```\n\n#### Top-K-masking\n- To specify Top-K-masking, add `--corrector_mask_mode topk`:\n```bash\npython ./bin/corrector_test_eval.py ./download/pretrained_weights/rico25/layout_corrector rico25 -t 100 --corrector_mask_mode topk\n```\n\n\u003c/details\u003e\n\n---\n\n\n# Preliminary Experiments and Analysis\n\n\u003cdetails\u003e\n\u003csummary\u003ePreliminary Experiment details\u003c/summary\u003e\n\n### Token-Correction Test\nTest Corrupted Token-Correction Capability of LayoutDM and its conjunction with Layout-Corrector.\n\n```bash\npython bin/test_token_correction.py \u003cLAYOUTDM_JOB_DIR\u003e [--start_timesteps \u003cTIMESTEP1\u003e \u003cTIMESTEP2\u003e ...] [--mask] [--num_replace \u003cNUM_REPLACE_TOKENS\u003e] [--save_dir \u003cSAVE_DIR\u003e]\n```\n\n- `--start_timesteps`: The start timesteps when the generation runs from (default: [10]).\n- `--mask`: If given, the randomly selected tokens are replaced with MASK.\n- `--num_replace`: The number of tokens to be replaced (default: 1).\n- `--save_dir`: A directory where the result is saved (default: `token_correction_results`).\n\nThe result `json` includes two metrics.\n- `token_wise`: The token-wise accuracy of restoring the corrupted tokens to the ground truth.\n- `full`: Requiring to correctly restore all corrupted tokens.\n\nTo compare the token correction capability for different schedules, as in Fig.2 (b) of our paper, please see [layoutdm_token_correction.ipynb](./notebooks/layoutdm_token_correction.ipynb).\n\n### Corrupted Token Detection Test for Corrector\nTest Corrupted Token Detection Accuracy of Layout-Corrector.\n\n```bash\npython bin/test_error_token_detection.py \u003cLAYOUTDM_JOB_DIR\u003e --corr_job_dir \u003cCORRECTOR_JOB_DIR\u003e --corr_timesteps \u003cCORR_TIMESTEP1\u003e \u003cCORR_TIMESTEP2\u003e ... [--num_replace \u003cNUM_REPLACE_TOKENS\u003e] [--save_dir \u003cSAVE_DIR\u003e]\n```\n\n- `--corr_job_dir`: Corrector job directory where ckpt is included. If not given, the evaluation runs only for LayoutDM. \n- `--corr_timesteps`: The timesteps at which the corrector is applied (default: [10]).\n- `--num_replace`: The number of tokens to be replaced (default: 1).\n- `--save_dir`: A directory where the result is saved (default: `error_token_detection_results`).\n\nThe result `json` includes two metrics at each timestep.\n- `token_wise_acc`: The token-wise accuracy of detecting the corrupted tokens.\n- `full_acc`: Requiring to correctly detect all corrupted tokens.\n\nTo plot the accuracy of corrupted token detection, as in Fig.4 of our paper, please see [layout_corrector_error_token_detection.ipynb](./notebooks/layout_corrector_error_token_detection.ipynb).\n\n### Visualize Generation Process\nSave Layout Generation Process at all timesteps as a pickle file and images.\n\n```bash\npython tools/visualize_generation_process.py \u003cLAYOUTDM_JOB_DIR\u003e [--corr_job_dir \u003cCORRECTOR_JOB_DIR\u003e] [--num_samples \u003cNUM_SAMPLES\u003e]  [--corr_timesteps \u003cCORR_TIMESTEP1\u003e \u003cCORR_TIMESTEP2\u003e ...] [--save_dir \u003cSAVE_DIR\u003e] [--save_images]\n```\n\n- `--corr_job_dir`: Corrector job directory where ckpt is included. If not given, the results are by just LayoutDM. \n- `--num_samples`: The total number of generated samples (default: 100).\n- `--corr_timesteps`: The timesteps at which the corrector is applied (default: [10, 20, 30]).\n- `--corrector_mask_mode`: The masking strategy for the corrector. `topk` or `thresh` are allowed (default: `thresh`).\n- `--corrector_threshold`: The threshold value to determine tokens to reset to MASK when `corrector_mask_mode == \"thresh\"` (default: 0.7).\n- `--save_dir`: A directory where the result is saved (default: `generation_process_results`).\n- `--save_images`: Whether saving the results as images or not.\n\n\n### Analyze Sticking Rate\nPlot the token and element sticking rate of LayoutDM.\nNote that you need to run `tools/visualize_generation_process.py` before using this tool.\nThe output is saved in the directory where the pickle file is located.\n\n```bash\npython tools/analyze_token_sticking.py \u003cPICKLE_PATH\u003e\n```\n\nTo compare the sticking rate for different schedules, as in Fig.2 (a) of our paper, please see [layoutdm_token_sticking.ipynb](./notebooks/layoutdm_token_sticking.ipynb).\n\n\n\u003c/details\u003e\n\n---\n\n## Acknowledgement\nThis codebase is largely based on [LayoutDM (Inoue+, CVPR2023)](https://github.com/CyberAgentAILab/layout-dm). We sincerely appreciate their effort and contribution to the research community.\n\n\n## Citation\n\nIf you find this code useful for your research, please consider citing our paper:\n\n```\n@inproceedings{iwai2024layout,\n  title={Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model},\n  author={Shoma Iwai and Atsuki Osanai and Shunsuke Kitada and Shinichiro Omachi},\n  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},\n  year={2024},\n}\n```\n\n## Issues\n- If you have any questions or encounterd any issues, please feel free to open an issue!\n- Pull requests are welcome! We hope to open an issue first to discuss what you would like to change.\n\n## License\n[MIT](./LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fline%2Flayout-corrector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fline%2Flayout-corrector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fline%2Flayout-corrector/lists"}