{"id":20543143,"url":"https://github.com/opennmt/im2text","last_synced_at":"2025-10-08T19:07:42.207Z","repository":{"id":85315644,"uuid":"76819060","full_name":"OpenNMT/Im2Text","owner":"OpenNMT","description":"Im2Text extension to OpenNMT","archived":false,"fork":false,"pushed_at":"2017-08-09T04:45:00.000Z","size":7541,"stargazers_count":138,"open_issues_count":5,"forks_count":22,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-14T09:44:44.050Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenNMT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-12-19T01:53:09.000Z","updated_at":"2025-01-26T04:00:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"729f7f83-6eb8-4dd5-800e-c8f6ae13fbcf","html_url":"https://github.com/OpenNMT/Im2Text","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenNMT/Im2Text","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FIm2Text","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FIm2Text/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FIm2Text/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FIm2Text/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenNMT","download_url":"https://codeload.github.com/OpenNMT/Im2Text/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FIm2Text/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000701,"owners_count":26082805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T01:36:00.239Z","updated_at":"2025-10-08T19:07:42.202Z","avatar_url":"https://github.com/OpenNMT.png","language":"Lua","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Im2Text\n\nA deep learning-based approach to learning the image-to-text conversion, built on top of the \u003ca href=\"https://opennmt.github.io/\"\u003eOpenNMT\u003c/a\u003e system. It is completely data-driven, hence can be used for a variety of image-to-text problems, such as image captioning, optical character recognition and LaTeX decompilation. \n\nTake LaTeX decompilation as an example, given a formula image:\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"http://lstm.seas.harvard.edu/latex/results/website/images/119b93a445-orig.png\"\u003e\u003c/p\u003e\n\nThe goal is to infer the LaTeX source that can be compiled to such an image:\n\n```\n d s _ { 1 1 } ^ { 2 } = d x ^ { + } d x ^ { - } + l _ { p } ^ { 9 } \\frac { p _ { - } } { r ^ { 7 } } \\delta ( x ^ { - } ) d x ^ { - } d x ^ { - } + d x _ { 1 } ^ { 2 } + \\; \\cdots \\; + d x _ { 9 } ^ { 2 } \n```\n\nThe paper (http://arxiv.org/pdf/1609.04938v1.pdf) provides more technical details of this model.\n\n## Installation\n\nIm2Text is built on top of \u003ca href=\"https://opennmt.github.io/\"\u003eOpenNMT\u003c/a\u003e, which is packed in this project. It also depends on `tds`, `class`, `cudnn`, `cutorch` and `paths`. Currently we only support **GPU**.\n\n\n## Quick Start\n\nTo get started, we provide a toy Math-to-LaTex example. We assume that the working directory is `Im2Text` throughout this document.\n\nIm2Text consists of two commands:\n\n1) Train the model.\n\n```\nth src/train.lua -phase train -gpu_id 1 -input_feed -model_dir model \\\n-image_dir data/images -data_path data/train.txt -val_data_path data/validate.txt -label_path data/labels.txt -vocab_file data/vocab.txt \\\n-batch_size 20 -beam_size 1 \\\n-max_num_tokens 150 -max_image_width 500 -max_image_height 160 \\\n-max_grad_norm 20.0 -learning_rate 0.1 -decay perplexity_only\n```\n\n2) Translate the images.\n\n```\nth src/train.lua -phase test -gpu_id 1 -load_model -model_dir model \\\n-image_dir data/images -data_path data/test.txt \\\n-output_dir results \\\n-batch_size 2 -beam_size 5 \\\n-max_num_tokens 500 -max_image_width 800 -max_image_height 800\n```\n\nThe above dataset is sampled from the [processed-im2latex-100k-dataset](http://lstm.seas.harvard.edu/latex/processed-im2latex-100k-dataset.tgz). We provide a trained model [[link]](http://lstm.seas.harvard.edu/latex/model_latest) on this dataset. In order to use it, download and put it under `model_dir` before translating the images.\n\n## Data Format\n\n* `-image_dir`: The directory containing the images. Since images of the same size can be batched together, we suggest padding images of similar sizes to the same size in order to facilitate training.\n\n* `-label_path`: The file storing the tokenized labels, one label per line. It shall look like:\n```\n\u003clabel0_token0\u003e \u003clabel0_token1\u003e ... \u003clabel0_tokenN0\u003e\n\u003clabel1_token0\u003e \u003clabel1_token1\u003e ... \u003clabel1_tokenN1\u003e\n\u003clabel2_token0\u003e \u003clabel2_token1\u003e ... \u003clabel2_tokenN2\u003e\n...\n```\n\n* `-data_path`: The file storing the image-label pairs. Each line starts with the path of the image (relative to `image_dir`), followed by the index of the label in `label_path` (index counts from 0). At test time, the label indexes can be omitted.\n```\n\u003cimage0_path\u003e \u003clabel_index0\u003e\n\u003cimage1_path\u003e \u003clabel_index1\u003e\n\u003cimage2_path\u003e \u003clabel_index2\u003e\n...\n```\n\n* `-vocab_file`: The vocabulary file. Each line corresponds to a token. The tokens not in `vocab_file` will be considered unknown (UNK).\n\n\n## Options\n\nFor a complete set of options, run `th src/train.lua -h`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopennmt%2Fim2text","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopennmt%2Fim2text","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopennmt%2Fim2text/lists"}