{"id":13660594,"url":"https://github.com/researchmm/img2poem","last_synced_at":"2025-04-24T19:31:33.988Z","repository":{"id":108692250,"uuid":"141544334","full_name":"researchmm/img2poem","owner":"researchmm","description":"[MM'18] Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training","archived":false,"fork":false,"pushed_at":"2021-08-23T03:00:09.000Z","size":11390,"stargazers_count":281,"open_issues_count":3,"forks_count":60,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-11-10T14:42:47.850Z","etag":null,"topics":["code","dataset","poem-generator"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/researchmm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-07-19T07:55:21.000Z","updated_at":"2024-10-14T04:52:31.000Z","dependencies_parsed_at":"2023-04-05T06:01:48.681Z","dependency_job_id":null,"html_url":"https://github.com/researchmm/img2poem","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchmm%2Fimg2poem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchmm%2Fimg2poem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchmm%2Fimg2poem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchmm%2Fimg2poem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/researchmm","download_url":"https://codeload.github.com/researchmm/img2poem/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250693599,"owners_count":21472282,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["code","dataset","poem-generator"],"created_at":"2024-08-02T05:01:23.420Z","updated_at":"2025-04-24T19:31:28.978Z","avatar_url":"https://github.com/researchmm.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training\nAn implementation of the image-to-poem model described in the paper:\n\"Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training.\"\nAccepted as the best paper of ACM MM2018.\n\nBei Liu, Jianlong Fu, Makoto P. Kato, Masatoshi Yoshikawa\n\nFull text available at: https://arxiv.org/abs/1804.08473\n\n## Contents\n* [Model Overview](#model-overview)\n    * [Introduction](#introduction)\n    * [Architecture](#architecture)\n* [Getting Started](#getting-started)\n    * [Install Required Packages](#install-required-packages)\n    * [Prepare the Training Data](#prepare-the-training-data)\n    * [Download Trained Model](#download-trained-model)\n* [Generating Poems](#generating-poems)\n\n## Model Overview\n\n### Introduction\n\nThe Img2poem model is a deep neural network that learns how to generate poems from images. For example:\n![](img/example.png)\n\n### Architecture\n![](img/framework.png)\n\n## Getting Started\n\n### Install Required Packages\n(It is recommended to install the dependencies under Conda environment.)  \n* python2.7  \n* tensorflow1.6  \n* mxnet  \n* opencv  \n* tqdm  \n* colorama  \n* flask\n\n### Prepare the Training Data\n\n| Name | #Poem | #Line/poem | #Word/line |\n| :------:| :------: | :------: | :-----: |\n| MultiM-Poem | 8,292 | 7.2 | 5.7 |\n| UniM-Poem | 93,265 | 5.7 | 6.2 |\n| MultiM-Poem(Ex) | 26,161 | 5.4 | 5.9 |\n\nBoth datasets are formatted in JSON files.\n\nMultiM-Poem.json: image and poem pairs\n\n```json\n[\n    {\n        \"poem\": str,\n        \"image_url\": str,\n        \"id\": int\n    },\n    ...\n]\n```\n\nUniM-Poem.json: poem corpus\n\n```json\n[\n    {\n        \"poem\": str,\n        \"id\": int\n    },\n    ...\n]\n```\n\n![](img/dataset.png)\n\n### Download Trained Model\nPlease download models from https://1drv.ms/u/s!AkLgJBAHL_VFgSyyfpeGyGFZux56 and put it under \"code/\".\n\n## Generating Poems\nThe following command line will generate poem for an image.\n```bash\npython test.py\n```\nType in the relative path to the test image in the console and the poem will be generated.\n```bash\n../images/test.jpg\n```\n\nExample output:\n```txt\nthe sun is singing in the forest wind\nand let us go to the wind of the sun\nlet the sun be free\nlet us be the storm of heaven\nand let us be the slow sun\nwe keep our own strength together\nwe live in love and hate\n```\n\n## Results\n![](img/results.png)\n\nHere are some examples of poems generated by eight methods for an image.\n![](img/example2.png)\n\n\n## Citation\n\nIf you find this repo useful in your research, please consider citing the following papers:\n\n```latex\n@inproceedings{liu2018beyond,\n  title={Beyond narrative description: Generating poetry from images by multi-adversarial training},\n  author={Liu, Bei and Fu, Jianlong and Kato, Makoto P and Yoshikawa, Masatoshi},\n  booktitle={Proceedings of the 26th ACM international conference on Multimedia},\n  pages={783--791},\n  year={2018}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchmm%2Fimg2poem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresearchmm%2Fimg2poem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchmm%2Fimg2poem/lists"}