{"id":31211507,"url":"https://github.com/cyberagentailab/text2palette","last_synced_at":"2025-09-21T05:30:14.413Z","repository":{"id":203448159,"uuid":"699788915","full_name":"CyberAgentAILab/text2palette","owner":"CyberAgentAILab","description":"Implementation of \"Multimodal Color Recommendation for Vector Graphic Documents\" ACM MM'23","archived":false,"fork":false,"pushed_at":"2024-09-09T01:03:55.000Z","size":72338,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-10T07:42:52.599Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyberAgentAILab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-03T10:47:48.000Z","updated_at":"2025-06-04T07:59:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"b909e0e1-e84b-4ee7-9e7e-727a3281b965","html_url":"https://github.com/CyberAgentAILab/text2palette","commit_stats":null,"previous_names":["cyberagentailab/text2palette"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CyberAgentAILab/text2palette","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ftext2palette","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ftext2palette/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ftext2palette/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ftext2palette/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyberAgentAILab","download_url":"https://codeload.github.com/CyberAgentAILab/text2palette/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2Ftext2palette/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":276195627,"owners_count":25601152,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-21T02:00:07.055Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-21T05:30:12.133Z","updated_at":"2025-09-21T05:30:14.377Z","avatar_url":"https://github.com/CyberAgentAILab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multimodal Color recommendation in vector graphic documents \n\nOfficial implementation of [Multimodal Color Recommendation in Vector Graphic Documents, ACM MM 2023](https://arxiv.org/abs/2308.04118).\n\n![Overview_image](docs/overview.png)\n\n## Introduction\n\nIn this study, we propose a multimodal masked color model that integrates both color and textual contexts to provide text-aware color recommendation for graphic documents. Our proposed model comprises self-attention networks to capture the relationships between colors in multiple palettes, and cross-attention networks that incorporate both color and [CLIP-based](https://github.com/openai/CLIP) text representations. \n\nThis proposal is applicable for two color recommendation tasks, color palette completion, which recommends colors based on the given colors and text, and full palette generation, which generates a complete color palette corresponding to the given text. The code for these two tasks is organized in two separate folders.\n\n## Setup\n\nThis project has been developed and tested in a Google Cloud Platform (GCP) notebook instance.\nPS: We use google-cloud-vision to detect image labels. An updated version with Open-source-software(OSS) will be released soon.\n- Machine type: n1-standard-4\n- Environment: TensorFlow Enterprise 2.3\n- GPU: NVIDIA T4 * 1.\n- Python:3.7.12\n- CUDA: 11.6\n\nInstall [pytorch](https://pytorch.org/get-started/locally/) and requirements\n```\npip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu116\npip3 install requirement.txt\n```\n\n## Color palette completion\n\nTask: Recommends colors based on the given colors and text\n\nTarget: multiple palettes in a graphic document\n\n\u003cimg src=\"docs/palette_completion.png\" width=\"650px\"\u003e\n\n### Quick demo\n\n[color_palette_completion/](color_palette_completion/)\n\n[notebooks/palette_compl.ipynb](color_palette_completion/notebooks/palette_compl.ipynb): recommend colors for multiple palettes in a design\n- Trained model of color prediction are in `data/trained_models/`.\n- Json files for test are pre-created in `data/samples/crello_samples/`.\n- New input image files for test are in `data/samples/image_samples/`.\n\nYou can also create a json file for test from crello dataset on a notebook [notebooks/create_json_file.ipynb](color_palette_completion/notebooks/create_json_file.ipynb).\n\n### Train models\n\nStep1: Create color and text embedding files of train, validation, and test on a notebook [notebooks/preprocess.ipynb](color_palette_completion/notebooks/preprocess.ipynb)\n- `data/data_t2p/color`: color corpus of train, validation, and test dataset, and color vocabulary from train dataset\n- `data/data_t2p/text`: text contents and image labels of train, validation, and test dataset\n- `data/data_t2p/text/emb_clip_imagemust_seq`: pre-created text embedings of text contents and image labels for train, validation, and test\n\nStep2: Train a color model on a notebook [notebooks/train_model.ipynb](color_palette_completion/notebooks/train_model.ipynb).\n\n### Data\n\n`data/data_colors/data_colors_labels/.`: extracted color palettes for Image-SVG-Text elements, text contents and image lables from [Crello-dataset-v2](https://storage.googleapis.com/ailab-public/canvas-vae/crello-dataset-v2.zip) ([the lastest Crello-dataset](https://github.com/CyberAgentAILab/canvas-vae/blob/main/docs/crello-dataset.md)).\n- Rawdata in `data/crello-dataset-v2`: Download and unzip the crello dataset v2 and get .tfrecord files for train/val/test.\n- Data filter: high frequent image labels, English contents\n\n`data/trained_model`: trained model for text-aware color completion\n\n`data/samples`: json sample files for testing the results of color completion\n\n## Full palette completion\n\nTask: Generates a complete color palette corresponding to the given text\n\nTarget: A single palette for an image\n\n\u003cimg src=\"docs/full_palette_generation.png\" width=\"300px\"\u003e\n\n### Quick demo\n\n[full_palette_generation/](full_palette_generation/)\n\nStep1: Create color and text embedding files of train, validation, and test on a notebook [notebooks/preprocess.ipynb](full_palette_generation/notebooks/preprocess.ipynb)\n- `data/data_t2p/color`: color corpus of train, validation, and test dataset, and color vocabulary from train dataset\n- `data/data_t2p/text`: text input of train, validation, and test dataset\n- `data/data_t2p/text/emb_clip`: pre-created text embedings of text contents and image labels for train, validation, and test\n\nStep2: Generate colors based on given text on a notebook [notebooks/palette_gen.ipynb](full_palette_generation/notebooks/palette_gen.ipynb)\n- Trained model of color generation are in `data/trained_models/`.\n\n### Train models\n\nTrain a color model on a notebook [notebooks/train_model.ipynb](full_palette_generation/notebooks/train_model.ipynb).\n\n### Data\n\n`data/data_colors/.`: palette-text pairs for train, validation, and test from [Palette-And-Text dataset](https://github.com/awesome-davian/Text2Colors)\n\n`data/trained_model`: trained model for text-aware palette generation\n\n## Citation\n\n```bibtex\n@inproceedings{qiu2023multimodal,\n  title = {Multimodal Color Recommendation in Vector Graphic Documents},\n  author = {Qiu, Qianru and Wang, Xueting and Otani, Mayu},\n  booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},\n  pages = {4003--4011},\n  year = {2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Ftext2palette","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyberagentailab%2Ftext2palette","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Ftext2palette/lists"}