{"id":13609921,"url":"https://github.com/abelriboulot/onnxt5","last_synced_at":"2025-04-12T22:32:09.675Z","repository":{"id":54806196,"uuid":"284230650","full_name":"abelriboulot/onnxt5","owner":"abelriboulot","description":"Summarization, translation, sentiment-analysis, text-generation and more at blazing speed using a T5 version implemented in ONNX.","archived":false,"fork":false,"pushed_at":"2022-11-02T18:43:57.000Z","size":1037,"stargazers_count":252,"open_issues_count":10,"forks_count":30,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-22T23:17:39.648Z","etag":null,"topics":["inference","nlp","nlp-machine-learning","onnx","onnxruntime","sentiment-analysis","summarization","text-classification","text-generation","transformer","transformers","translation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/abelriboulot.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-08-01T09:38:35.000Z","updated_at":"2025-02-10T05:42:43.000Z","dependencies_parsed_at":"2022-08-14T03:20:59.960Z","dependency_job_id":null,"html_url":"https://github.com/abelriboulot/onnxt5","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abelriboulot%2Fonnxt5","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abelriboulot%2Fonnxt5/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abelriboulot%2Fonnxt5/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/abelriboulot%2Fonnxt5/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/abelriboulot","download_url":"https://codeload.github.com/abelriboulot/onnxt5/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248640887,"owners_count":21138108,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["inference","nlp","nlp-machine-learning","onnx","onnxruntime","sentiment-analysis","summarization","text-classification","text-generation","transformer","transformers","translation"],"created_at":"2024-08-01T19:01:39.298Z","updated_at":"2025-04-12T22:32:09.352Z","avatar_url":"https://github.com/abelriboulot.png","language":"Python","funding_links":[],"categories":["文本数据和NLP"],"sub_categories":[],"readme":"![ONNX T5](https://github.com/abelriboulot/onnxt5/blob/master/data/social_preview.png?raw=true)\n[![Actions Status](https://github.com/abelriboulot/onnxt5/workflows/Tests/badge.svg)](https://github.com/abelriboulot/onnxt5/actions)\n![Actions Status](https://img.shields.io/github/license/abelriboulot/onnxt5)\n![Version](https://img.shields.io/github/v/release/abelriboulot/onnxt5.svg)\n[![Downloads](https://pepy.tech/badge/onnxt5/week)](https://pepy.tech/project/onnxt5/week)\n[![Slack](https://img.shields.io/badge/slack-@onnxt5-yellow.svg?logo=slack)](https://join.slack.com/t/onnxt5/shared_invite/zt-gdjudd03-xutGvyQuYLMjBGnKH8fbLw)\n\nSummarization, translation, Q\u0026A, text generation and more at blazing speed using a T5 version implemented in ONNX. \n\nThis package is still in alpha stage, therefore some functionalities such as beam searches are still in development.\n\n## Installation\nONNX-T5 is available on PyPi.\n```bash\npip install onnxt5\n```\n\nFor the dev version you can run the following.\n```bash\ngit clone https://github.com/abelriboulot/onnxt5\ncd onnxt5\npip install -e .\n```\n\n## Usage\n\nThe simplest way to get started for generation is to use the default pre-trained\nversion of T5 on ONNX included in the package.\n\n\u003e **_NOTE:_**  Please note that the first time you call get_encoder_decoder_tokenizer, the models are\nbeing downloaded which might take a minute or two.\n\n```python\nfrom onnxt5 import GenerativeT5\nfrom onnxt5.api import get_encoder_decoder_tokenizer\ndecoder_sess, encoder_sess, tokenizer = get_encoder_decoder_tokenizer()\ngenerative_t5 = GenerativeT5(encoder_sess, decoder_sess, tokenizer, onnx=True)\nprompt = 'translate English to French: I was a victim of a series of accidents.'\n\noutput_text, output_logits = generative_t5(prompt, max_length=100, temperature=0.)\n# output_text: \"J'ai été victime d'une série d'accidents.\"\n```\n\nOther tasks just require to change the prefix in your prompt, for instance for summarization:\n```python\nprompt = 'summarize: \u003cPARAGRAPH\u003e'\noutput_text, output_logits = generative_t5(prompt, max_length=100, temperature=0.)\n```\n\nIf you want to get the embeddings of text, you can run the following\n```python\nfrom onnxt5.api import get_encoder_decoder_tokenizer, run_embeddings_text\n\ndecoder_sess, encoder_sess, tokenizer = get_encoder_decoder_tokenizer()\nprompt = 'Listen, Billy Pilgrim has come unstuck in time.'\nencoder_embeddings, decoder_embeddings = run_embeddings_text(encoder_sess, decoder_sess, tokenizer, prompt)\n```\n\nONNXT5 also lets you export and use your own models. See the `examples\\` folder for more detailed examples.\n\nT5 works with tokens such as `summarize:`, `translate English to German:`, or `question: ... context:`. You can see a \nlist of the pretrained tasks and token in the appendix D of the [original paper](https://arxiv.org/pdf/1910.10683.pdf).\n\n## Functionalities\n* Run any of the T5 trained tasks in a line (translation, summarization, sentiment analysis, completion, generation)\n* Export your own T5 models to ONNX easily\n* Utility functions to generate what you need quickly\n* Up to 4X speedup compared to PyTorch execution for smaller contexts\n\n## Benchmarks\nThe outperformance varies heavily based on the length of the context. For contexts less than ~500 words,\nONNX outperforms greatly, going up to a 4X speedup compared to PyTorch. However, the longer the context, the smaller the \nspeedup of ONNX, with Pytorch being faster above 500 words.\n\n#### GPU Benchmark, Embedding Task\n\n![Benchmark Embedding](https://github.com/abelriboulot/onnxt5/blob/master/data/Embedding_benchmark.png?raw=true)\n#### GPU Benchmark, Generation Task\n\n![Benchmark Generation](https://github.com/abelriboulot/onnxt5/blob/master/data/Generation_benchmark.png?raw=true)\n\n## Contributing\nThe project is still in its infancy, so I would love your feedback, to know what problems you are trying to solve, hear issues you're encountering, \nand discuss features that would help you. Therefore feel free to shoot me an e-mail (see [my profile](https://github.com/abelriboulot) for the address!) or \njoin our [slack community](https://join.slack.com/t/onnxt5/shared_invite/zt-gdjudd03-xutGvyQuYLMjBGnKH8fbLw).\n\n## Acknowledgements\nThis repo is based on the work of Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and \nMichael Matena and Yanqi Zhou and Wei Li and Peter J. Liu from Google, as well as the implementation of T5 from the \nhuggingface team, the work of the Microsoft ONNX and onnxruntime teams, in particular Tianlei Wu, and the work of Thomas Wolf on generation of text.\n\n[Original T5 Paper](https://arxiv.org/pdf/1910.10683.pdf)\n```\n@article{2019t5,\n  author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},\n  title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},\n  journal = {arXiv e-prints},\n  year = {2019},\n  archivePrefix = {arXiv},\n  eprint = {1910.10683},\n}\n```\n[Microsoft onnxruntime repo](https://github.com/microsoft/onnxruntime)\n\n[HuggingFace implementation of T5](https://huggingface.co/transformers/model_doc/t5.html)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabelriboulot%2Fonnxt5","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fabelriboulot%2Fonnxt5","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fabelriboulot%2Fonnxt5/lists"}