{"id":31034639,"url":"https://github.com/freedomintelligence/sharegpt-4o-image","last_synced_at":"2025-09-14T02:46:42.910Z","repository":{"id":300794953,"uuid":"1006430770","full_name":"FreedomIntelligence/ShareGPT-4o-Image","owner":"FreedomIntelligence","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-22T06:08:45.000Z","size":4431,"stargazers_count":238,"open_issues_count":4,"forks_count":10,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-22T08:38:49.401Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FreedomIntelligence.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-22T08:52:50.000Z","updated_at":"2025-07-22T06:08:49.000Z","dependencies_parsed_at":"2025-06-23T17:37:25.165Z","dependency_job_id":null,"html_url":"https://github.com/FreedomIntelligence/ShareGPT-4o-Image","commit_stats":null,"previous_names":["freedomintelligence/sharegpt-4o-image"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/FreedomIntelligence/ShareGPT-4o-Image","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FShareGPT-4o-Image","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FShareGPT-4o-Image/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FShareGPT-4o-Image/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FShareGPT-4o-Image/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FreedomIntelligence","download_url":"https://codeload.github.com/FreedomIntelligence/ShareGPT-4o-Image/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FreedomIntelligence%2FShareGPT-4o-Image/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275054971,"owners_count":25397576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-14T02:00:10.474Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-14T02:46:37.294Z","updated_at":"2025-09-14T02:46:42.902Z","avatar_url":"https://github.com/FreedomIntelligence.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation\n\n\u003c!--\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./assets/logo.jpg\" alt=\"logo\" width=\"170\" class=\"center\"/\u003e\u003cbr\u003e\n\u003c/p\u003e\n--\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ch4\u003e\n 📃 \u003ca href=\"https://arxiv.org/abs/2506.18095\" target=\"_blank\"\u003ePaper\u003c/a\u003e\n\u003c/h4\u003e\n\u003ch4\u003e\n  📚 \u003ca href=\"https://huggingface.co/datasets/FreedomIntelligence/ShareGPT-4o-Image\" target=\"_blank\"\u003eShareGPT-4o-Image\u003c/a\u003e\n\u003c/h4\u003e\n  \u003ch4\u003e\n  🤗 \u003ca href=\"https://huggingface.co/FreedomIntelligence/Janus-4o-7B\" target=\"_blank\"\u003eJanus-4o-7B\u003c/a\u003e\n\u003c/h4\u003e\n\u003c/div\u003e\n\n## ⚡ Introduction\n**ShareGPT-4o-Image** is a large-scale, high-quality dataset of 92K samples generated by **GPT-4o’s image generation capabilities**, including **45K text-to-image** and  **46K text-and-image-to-image** examples. It aims to support the development of open multimodal models aligned with GPT-4o’s strengths in image generation. Besides, we introduce **Janus-4o**, a new multimodal large language model (MLLM) capable of both text-to-image and text-and-image-to-image generation. \n\n\u003e ⚠️ **Statement**: **ShareGPT-4o-Image** is a distilled dataset from GPT-4o-Image, offering 4o-level data quality (_referring to data, not model capability_). **Janus-4o** is a fine-tuned version of Janus-Pro on this dataset, with added image editing support. Fine-tuning brings noticeable gains in image generation, but **Janus-4o still lags behind GPT-4o-Image in overall performance**.\n\n\u003cdiv align=center\u003e\n\u003cimg src=\"./assets/fig_0.png\"  width = \"70%\" alt=\"mainpic\" align=center/\u003e\n\u003c/div\u003e\n\n## 🖼️ ShareGPT-4o-Image\n\n**ShareGPT-4o-Image** contains a total of **91K image generation samples** from GPT-4o, categorized as follows:\n\n| Data Type               | Number of Samples |\n| ----------------------- | ----------------- |\n| Text-to-Image           | 45,717            |\n| Text-and-Image-to-Image | 46,539            |\n| **Total**               | **92,256**        |\n\n\n\u003e 🔗 Dataset on Hugging Face: [FreedomIntelligence/ShareGPT-4o-Image](https://huggingface.co/datasets/FreedomIntelligence/ShareGPT-4o-Image)\n\n## 🎨 Janus-4o Model\n**Janus-4o** is a multimodal LLM capable of both text-to-image and text-and-image-to-image generation. It is fine-tuned from Janus-Pro using the [ShareGPT-4o-Image](https://huggingface.co/datasets/FreedomIntelligence/ShareGPT-4o-Image) dataset. Compared to Janus-Pro, Janus-4o **newly supports text-and-image-to-image generation capabilities**, along with notable improvements in text-to-image generation.\n\n\n#### Model Access\nJanus-4o is available on Huggingface:\n\n|                 | Backbone           | Checkpoint                                                                            |\n|----------------------|--------------------|---------------------------------------------------------------------------------------|\n| **Janus-4o**  | Janus-Pro-7B         | [FreedomIntelligence/Janus-4o-7B](https://huggingface.co/FreedomIntelligence/Janus-4o-7B)             |\n\n#### Model Inference\n\n**📦 Step 1: Install Janus**\n\nClone the [Janus](https://github.com/deepseek-ai/Janus) repository and install the package:\n```shell\ngit clone https://github.com/deepseek-ai/Janus.git\ncd Janus\npip install -e .\n```\n\n**🧠 Step 2: Run Inference**\n\u003cdetails open\u003e\n\u003csummary\u003e\u003ch4\u003eA. Launch with Gradio Demo\u003c/h4\u003e\u003c/summary\u003e\n\n```shell\npip install -e .[gradio]\npython demo/app_janus4o.py\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003ch4\u003eB. Text-to-Image Generation\u003c/h4\u003e\u003c/summary\u003e\n\n```Python\nimport os\nimport PIL.Image\nimport torch\nimport numpy as np\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\n\n# Load model and processor\nmodel_path = \"FreedomIntelligence/Janus-4o-7B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True,torch_dtype=torch.bfloat16\n)\nvl_gpt = vl_gpt.cuda().eval()\n\n# Define text-to-image generation function\ndef text_to_image_generate(input_prompt, output_path, vl_chat_processor, vl_gpt, temperature = 1.0, parallel_size = 2, cfg_weight = 5):\n\n    torch.cuda.empty_cache()\n\n    conversation = [\n            {\n             \"role\": \"\u003c|User|\u003e\",\n            \"content\": input_prompt,\n        },\n        {\"role\": \"\u003c|Assistant|\u003e\", \"content\": \"\"},\n    ]\n\n    sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(\n        conversations=conversation,\n        sft_format=vl_chat_processor.sft_format,\n        system_prompt=\"\",\n    )\n\n    prompt = sft_format + vl_chat_processor.image_start_tag\n\n    mmgpt = vl_gpt\n\n    image_token_num_per_image = 576\n    img_size = 384\n    patch_size = 16\n\n    with torch.inference_mode():\n        input_ids = vl_chat_processor.tokenizer.encode(prompt)\n        input_ids = torch.LongTensor(input_ids)\n\n        tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda()\n        for i in range(parallel_size*2):\n            tokens[i, :] = input_ids\n            if i % 2 != 0:\n                tokens[i, 1:-1] = vl_chat_processor.pad_id\n\n        inputs_embeds = mmgpt.language_model.get_input_embeddings()(tokens)\n\n        generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()\n\n        for i in range(image_token_num_per_image):\n            outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)\n            hidden_states = outputs.last_hidden_state\n\n            logits = mmgpt.gen_head(hidden_states[:, -1, :])\n            logit_cond = logits[0::2, :]\n            logit_uncond = logits[1::2, :]\n            \n            logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)\n            probs = torch.softmax(logits / temperature, dim=-1)\n\n            next_token = torch.multinomial(probs, num_samples=1)\n            generated_tokens[:, i] = next_token.squeeze(dim=-1)\n\n            next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)\n            img_embeds = mmgpt.prepare_gen_img_embeds(next_token)\n            inputs_embeds = img_embeds.unsqueeze(dim=1)\n\n        dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])\n        dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)\n\n        dec = np.clip((dec + 1) / 2 * 255, 0, 255)\n\n        visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)\n        visual_img[:, :, :] = dec\n\n        os.makedirs(output_path, exist_ok=True)\n        output_images = []\n        for i in range(parallel_size):\n            save_path = output_path.replace('.png','') + f'_{i}.png'\n            PIL.Image.fromarray(visual_img[i]).save(save_path)\n            output_images.append(save_path)\n        return output_images\n\n# Run\nprompt = \"A stunning princess from kabul in red, white traditional clothing, blue eyes, brown hair\"\nimage_output_path = \"./test.png\"\ntext_to_image_generate(prompt, image_output_path, vl_chat_processor, vl_gpt, parallel_size = 2)\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \n\u003csummary\u003e\u003ch4\u003eC. Text-and-Image-to-Image Generation\u003c/h4\u003e\u003c/summary\u003e\n  \n```Python\nimport os\nimport PIL.Image\nimport torch\nimport numpy as np\nfrom transformers import AutoModelForCausalLM\nfrom janus.models import MultiModalityCausalLM, VLChatProcessor\nfrom dataclasses import dataclass\n@dataclass\nclass VLChatProcessorOutput():\n    sft_format: str\n    input_ids: torch.Tensor\n    pixel_values: torch.Tensor\n    num_image_tokens: torch.IntTensor\n\n    def __len__(self):\n        return len(self.input_ids)\n\ndef process_image(image_paths,vl_chat_processor):\n    images = [PIL.Image.open(image_path).convert(\"RGB\") for image_path in image_paths]\n    images_outputs = vl_chat_processor.image_processor(images, return_tensors=\"pt\")\n    return images_outputs['pixel_values']\n\n# Load model and processor\nmodel_path = \"FreedomIntelligence/Janus-4o-7B\"\nvl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)\ntokenizer = vl_chat_processor.tokenizer\nvl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(\n    model_path, trust_remote_code=True,torch_dtype=torch.bfloat16\n)\nvl_gpt = vl_gpt.cuda().eval()\n\n# Define text+image-to-image generation function\ndef text_and_image_to_image_generate(input_prompt, input_image_path, output_path, vl_chat_processor, vl_gpt, temperature = 1.0, parallel_size = 2, cfg_weight = 5, cfg_weight2 = 5):\n    torch.cuda.empty_cache()\n\n    input_img_tokens = vl_chat_processor.image_start_tag + vl_chat_processor.image_tag*vl_chat_processor.num_image_tokens +vl_chat_processor.image_end_tag + vl_chat_processor.image_start_tag + vl_chat_processor.pad_tag*vl_chat_processor.num_image_tokens +vl_chat_processor.image_end_tag\n    output_img_tokens = vl_chat_processor.image_start_tag \n\n    pre_data = []\n    input_images = [input_image_path]\n    img_len = len(input_images)\n    prompts = input_img_tokens * img_len + input_prompt\n    conversation = [\n                    {\"role\": \"\u003c|User|\u003e\",\"content\": prompts},\n                    {\"role\": \"\u003c|Assistant|\u003e\", \"content\": \"\"}\n                ]\n    sft_format = vl_chat_processor.apply_sft_template_for_multi_turn_prompts(\n        conversations=conversation,\n        sft_format=vl_chat_processor.sft_format,\n        system_prompt=\"\",\n    )\n\n    sft_format = sft_format + output_img_tokens\n\n    mmgpt = vl_gpt\n\n    image_token_num_per_image = 576\n    img_size = 384\n    patch_size = 16\n\n    with torch.inference_mode():\n        input_image_pixel_values = process_image(input_images,vl_chat_processor).to(torch.bfloat16).cuda()\n        quant_input, emb_loss_input, info_input = mmgpt.gen_vision_model.encode(input_image_pixel_values)\n        image_tokens_input = info_input[2].detach().reshape(input_image_pixel_values.shape[0], -1)\n        image_embeds_input = mmgpt.prepare_gen_img_embeds(image_tokens_input)\n\n        input_ids =  torch.LongTensor(vl_chat_processor.tokenizer.encode(sft_format))\n        \n        encoder_pixel_values = process_image(input_images,vl_chat_processor).cuda()\n        tokens = torch.zeros((parallel_size*3, len(input_ids)), dtype=torch.long)\n        for i in range(parallel_size*3):\n            tokens[i, :] = input_ids\n            if i % 3 == 2:\n                tokens[i, 1:-1] = vl_chat_processor.pad_id\n                pre_data.append(VLChatProcessorOutput(sft_format=sft_format, pixel_values=encoder_pixel_values, input_ids=tokens[i-2], num_image_tokens=[vl_chat_processor.num_image_tokens] * img_len))\n                pre_data.append(VLChatProcessorOutput(sft_format=sft_format, pixel_values=encoder_pixel_values, input_ids=tokens[i-1], num_image_tokens=[vl_chat_processor.num_image_tokens] * img_len))\n                pre_data.append(VLChatProcessorOutput(sft_format=sft_format, pixel_values=None, input_ids=tokens[i], num_image_tokens=[]))\n\n        prepare_inputs = vl_chat_processor.batchify(pre_data)\n\n        inputs_embeds = mmgpt.prepare_inputs_embeds(\n                    input_ids=tokens.cuda(),\n                    pixel_values=prepare_inputs['pixel_values'].to(torch.bfloat16).cuda(),\n                    images_emb_mask=prepare_inputs['images_emb_mask'].cuda(),\n                    images_seq_mask=prepare_inputs['images_seq_mask'].cuda()\n                )\n\n        image_gen_indices = (tokens == vl_chat_processor.image_end_id).nonzero()\n\n        for ii, ind in enumerate(image_gen_indices):\n            if ii % 4 == 0:\n                offset = ind[1] + 2\n                inputs_embeds[ind[0],offset: offset+image_embeds_input.shape[1],:] = image_embeds_input[(ii // 2) % img_len]\n\n        generated_tokens = torch.zeros((parallel_size, image_token_num_per_image), dtype=torch.int).cuda()\n\n        for i in range(image_token_num_per_image):\n            outputs = mmgpt.language_model.model(inputs_embeds=inputs_embeds, use_cache=True, past_key_values=outputs.past_key_values if i != 0 else None)\n            hidden_states = outputs.last_hidden_state\n\n            logits = mmgpt.gen_head(hidden_states[:, -1, :])\n            logit_cond_full = logits[0::3, :]\n            logit_cond_part = logits[1::3, :]\n            logit_uncond = logits[2::3, :]\n\n            logit_cond = (logit_cond_full + cfg_weight2 * (logit_cond_part)) / (1 + cfg_weight2)\n            logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond)\n            probs = torch.softmax(logits / temperature, dim=-1)\n\n            next_token = torch.multinomial(probs, num_samples=1)\n            generated_tokens[:, i] = next_token.squeeze(dim=-1)\n\n            next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1)\n            img_embeds = mmgpt.prepare_gen_img_embeds(next_token)\n            inputs_embeds = img_embeds.unsqueeze(dim=1)\n\n        dec = mmgpt.gen_vision_model.decode_code(generated_tokens.to(dtype=torch.int), shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size])\n        dec = dec.to(torch.float32).cpu().numpy().transpose(0, 2, 3, 1)\n\n        dec = np.clip((dec + 1) / 2 * 255, 0, 255)\n\n        visual_img = np.zeros((parallel_size, img_size, img_size, 3), dtype=np.uint8)\n        visual_img[:, :, :] = dec\n\n        output_images = []\n        for i in range(parallel_size):\n            save_path = output_path.replace('.png','') + f'_{i}.png'\n            PIL.Image.fromarray(visual_img[i]).save(save_path)\n            output_images.append(save_path)\n        return output_images\n\n# Run\nprompt = \"Turn the image into a nighttime scene.\"\ninput_image_path = \"./test_input.png\"\nimage_output_path = \"./test_output.png\"\ntext_and_image_to_image_generate(prompt, input_image_path, image_output_path, vl_chat_processor, vl_gpt, parallel_size = 2)\n```\n\n\u003c/details\u003e\n\n\n## 🏋️‍♂️ Model Training\n\nWe provide the code so you can train **Janus-Pro on ShareGPT-4o-Image** to reproduce **Janus-4o**. The training supports both **text-to-image** and **image-text-to-image** generation. Please run the following command:\n\n```shell\naccelerate launch --config_file configs/sft.yaml \\\n    --num_processes 8  \\\n    --num_machines 1 \\\n    --machine_rank 0 \\\n    --deepspeed_multinode_launcher standard train_janus.py \\\n    --model_path deepseek-ai/Janus-Pro-7B \\\n    --data_path [FreedomIntelligence/ShareGPT-4o-Image] \\\n    --n_epochs 3 \\\n    --train_bsz_per_gpu 1 \\\n    --learning_rate 5e-6 \\\n    --gradient_accumulation_steps 8\n```\n\n\n## Citation\n```\n@misc{chen2025sharegpt4oimg,\n      title={ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation}, \n      author={Junying Chen and Zhenyang Cai and Pengcheng Chen and Shunian Chen and Ke Ji and Xidong Wang and Yunjin Yang and Benyou Wang},\n      year={2025},\n      eprint={2506.18095},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2506.18095}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fsharegpt-4o-image","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffreedomintelligence%2Fsharegpt-4o-image","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffreedomintelligence%2Fsharegpt-4o-image/lists"}