Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/researchmm/img2poem
[MM'18] Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
https://github.com/researchmm/img2poem
code dataset poem-generator
Last synced: 3 months ago
JSON representation
[MM'18] Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
- Host: GitHub
- URL: https://github.com/researchmm/img2poem
- Owner: researchmm
- Created: 2018-07-19T07:55:21.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2021-08-23T03:00:09.000Z (about 3 years ago)
- Last Synced: 2024-06-20T19:40:57.369Z (5 months ago)
- Topics: code, dataset, poem-generator
- Language: Python
- Homepage:
- Size: 10.9 MB
- Stars: 282
- Watchers: 14
- Forks: 62
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
An implementation of the image-to-poem model described in the paper:
"Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training."
Accepted as the best paper of ACM MM2018.
Bei Liu, Jianlong Fu, Makoto P. Kato, Masatoshi Yoshikawa
Full text available at: https://arxiv.org/abs/1804.08473
## Contents
* [Model Overview](#model-overview)
* [Introduction](#introduction)
* [Architecture](#architecture)
* [Getting Started](#getting-started)
* [Install Required Packages](#install-required-packages)
* [Prepare the Training Data](#prepare-the-training-data)
* [Download Trained Model](#download-trained-model)
* [Generating Poems](#generating-poems)
## Model Overview
### Introduction
The Img2poem model is a deep neural network that learns how to generate poems from images. For example:
### Architecture
## Getting Started
### Install Required Packages
(It is recommended to install the dependencies under Conda environment.)
* python2.7
* tensorflow1.6
* mxnet
* opencv
* tqdm
* colorama
* flask
### Prepare the Training Data
| Name | #Poem | #Line/poem | #Word/line |
| :------:| :------: | :------: | :-----: |
| MultiM-Poem | 8,292 | 7.2 | 5.7 |
| UniM-Poem | 93,265 | 5.7 | 6.2 |
| MultiM-Poem(Ex) | 26,161 | 5.4 | 5.9 |
Both datasets are formatted in JSON files.
MultiM-Poem.json: image and poem pairs
```json
[
{
"poem": str,
"image_url": str,
"id": int
},
...
]
```
UniM-Poem.json: poem corpus
```json
[
{
"poem": str,
"id": int
},
...
]
```
### Download Trained Model
Please download models from https://1drv.ms/u/s!AkLgJBAHL_VFgSyyfpeGyGFZux56 and put it under "code/".
## Generating Poems
The following command line will generate poem for an image.
```bash
python test.py
```
Type in the relative path to the test image in the console and the poem will be generated.
```bash
../images/test.jpg
```
Example output:
```txt
the sun is singing in the forest wind
and let us go to the wind of the sun
let the sun be free
let us be the storm of heaven
and let us be the slow sun
we keep our own strength together
we live in love and hate
```
## Results
Here are some examples of poems generated by eight methods for an image.
## Citation
If you find this repo useful in your research, please consider citing the following papers:
```latex
@inproceedings{liu2018beyond,
title={Beyond narrative description: Generating poetry from images by multi-adversarial training},
author={Liu, Bei and Fu, Jianlong and Kato, Makoto P and Yoshikawa, Masatoshi},
booktitle={Proceedings of the 26th ACM international conference on Multimedia},
pages={783--791},
year={2018}
}
```