{"id":15036057,"url":"https://github.com/opennmt/ctranslate2","last_synced_at":"2025-10-08T19:07:44.997Z","repository":{"id":37248311,"uuid":"210299376","full_name":"OpenNMT/CTranslate2","owner":"OpenNMT","description":"Fast inference engine for Transformer models","archived":false,"fork":false,"pushed_at":"2025-04-08T19:43:21.000Z","size":15196,"stargazers_count":3810,"open_issues_count":227,"forks_count":358,"subscribers_count":58,"default_branch":"master","last_synced_at":"2025-05-21T12:01:49.532Z","etag":null,"topics":["avx","avx2","cpp","cuda","deep-learning","deep-neural-networks","gemm","inference","intrinsics","machine-translation","mkl","neon","neural-machine-translation","onednn","openmp","opennmt","parallel-computing","quantization","thrust","transformer-models"],"latest_commit_sha":null,"homepage":"https://opennmt.net/CTranslate2","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenNMT.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-09-23T08:10:42.000Z","updated_at":"2025-05-21T10:23:46.000Z","dependencies_parsed_at":"2022-08-08T20:00:32.214Z","dependency_job_id":"e4c5ddf4-0ea8-4bf8-b817-c0fd39731c73","html_url":"https://github.com/OpenNMT/CTranslate2","commit_stats":{"total_commits":2113,"total_committers":46,"mean_commits":45.93478260869565,"dds":0.5177472787505916,"last_synced_commit":"c95fd4e4f8c775e6a8248fea9f7ab313c170e8dd"},"previous_names":[],"tags_count":167,"template":false,"template_full_name":null,"purl":"pkg:github/OpenNMT/CTranslate2","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FCTranslate2","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FCTranslate2/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FCTranslate2/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FCTranslate2/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenNMT","download_url":"https://codeload.github.com/OpenNMT/CTranslate2/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenNMT%2FCTranslate2/sbom","scorecard":{"id":105445,"data":{"date":"2025-08-11","repo":{"name":"github.com/OpenNMT/CTranslate2","commit":"617405f4b050e994e829d527da6caa0e0030afe7"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.6,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":4,"reason":"Found 14/29 approved changesets -- score normalized to 4","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/ci.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/ci.yml:257"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":0,"reason":"24 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2023-299 / GHSA-282v-666c-3fvg","Warn: Project is vulnerable to: GHSA-37mw-44qp-f5jm","Warn: Project is vulnerable to: GHSA-37q5-v5qm-c9v8","Warn: Project is vulnerable to: PYSEC-2023-300 / GHSA-3863-2447-669p","Warn: Project is vulnerable to: GHSA-6rvg-6v2m-4j46","Warn: Project is vulnerable to: GHSA-9356-575x-2w9m","Warn: Project is vulnerable to: GHSA-fpwr-67px-3qhx","Warn: Project is vulnerable to: PYSEC-2024-229 / GHSA-hxxf-235m-72v3","Warn: Project is vulnerable to: GHSA-jjph-296x-mrcr","Warn: Project is vulnerable to: GHSA-phhr-52qp-3mj4","Warn: Project is vulnerable to: GHSA-q2wp-rjmx-x6x9","Warn: Project is vulnerable to: PYSEC-2025-40 / GHSA-qq3j-4f4f-9583","Warn: Project is vulnerable to: PYSEC-2024-227 / GHSA-qxrp-vhvm-j765","Warn: Project is vulnerable to: PYSEC-2023-301 / GHSA-v68g-wm8c-6x7j","Warn: Project is vulnerable to: PYSEC-2024-228 / GHSA-wrfc-pvp9-mr9g","Warn: Project is vulnerable to: PYSEC-2013-22 / GHSA-27x4-j476-jp5f","Warn: Project is vulnerable to: PYSEC-2025-49 / GHSA-5rjg-fvgr-3xxf","Warn: Project is vulnerable to: GHSA-cx63-2mw6-8hw5","Warn: Project is vulnerable to: PYSEC-2022-43012 / GHSA-r9hx-vwmv-q579","Warn: Project is vulnerable to: PYSEC-2022-43017 / GHSA-qwmp-2cf2-g9g6","Warn: Project is vulnerable to: GHSA-3749-ghw9-m3mg","Warn: Project is vulnerable to: PYSEC-2025-41 / GHSA-53q9-r3pm-6pq6","Warn: Project is vulnerable to: GHSA-887c-mr87-cxwp","Warn: Project is vulnerable to: PYSEC-2024-259"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:190: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:194: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:202: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:229: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:232: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:264: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:271: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:281: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:291: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:308: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:311: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:316: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:339: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:85: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:146: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:150: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ci.yml:155: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ci.yml:173: update your workflow using https://app.stepsecurity.io/secureworkflow/OpenNMT/CTranslate2/ci.yml/master?enable=pin","Warn: containerImage not pinned by hash: docker/Dockerfile:1","Warn: containerImage not pinned by hash: docker/Dockerfile:80: pin your Docker image by updating nvidia/cuda:12.2.2-base-ubuntu22.04 to nvidia/cuda:12.2.2-base-ubuntu22.04@sha256:1a8a738e81d4adbef0c709241f5238cec5bb77186dcb5b2103db293315ed42d1","Warn: containerImage not pinned by hash: examples/wngt2020/Dockerfile.cpu:1","Warn: containerImage not pinned by hash: examples/wngt2020/Dockerfile.cpu:74: pin your Docker image by updating ubuntu:20.04 to ubuntu:20.04@sha256:8feb4d8ca5354def3d8fce243717141ce31e2c428701f6682bd2fafe15388214","Warn: containerImage not pinned by hash: examples/wngt2020/Dockerfile.gpu:1","Warn: containerImage not pinned by hash: examples/wngt2020/Dockerfile.gpu:61","Warn: containerImage not pinned by hash: tools/benchmark/opennmt_ende_wmt14/ctranslate2/Dockerfile:1","Warn: containerImage not pinned by hash: tools/benchmark/opennmt_ende_wmt14/ctranslate2/Dockerfile:20","Warn: containerImage not pinned by hash: tools/benchmark/opennmt_ende_wmt14/fastertransformer/Dockerfile:1: pin your Docker image by updating nvcr.io/nvidia/pytorch:22.09-py3 to nvcr.io/nvidia/pytorch:22.09-py3@sha256:ad07f7144606cb749dceb1ce7ed2286eeb69a63327ea7eccc69f0ac8ac1e0c68","Warn: containerImage not pinned by hash: tools/benchmark/opennmt_ende_wmt14/opennmt_py/Dockerfile:1: pin your Docker image by updating pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime to pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime@sha256:1e26efd426b0fecbfe7cf3d3ae5003fada6ac5a76eddc1e042857f5d049605ee","Warn: containerImage not pinned by hash: tools/benchmark/opennmt_ende_wmt14/opennmt_tf/Dockerfile:1: pin your Docker image by updating tensorflow/tensorflow:2.11.0-gpu to tensorflow/tensorflow:2.11.0-gpu@sha256:67f1a7b35fd52bdda071c0cd311655be7477f2bc1b6f27e014b9a57231bd55b3","Warn: containerImage not pinned by hash: tools/benchmark/opus_mt_ende/ctranslate2/Dockerfile:1","Warn: containerImage not pinned by hash: tools/benchmark/opus_mt_ende/marian/Dockerfile:1","Warn: containerImage not pinned by hash: tools/benchmark/opus_mt_ende/transformers/Dockerfile:1: pin your Docker image by updating pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime to pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime@sha256:1e26efd426b0fecbfe7cf3d3ae5003fada6ac5a76eddc1e042857f5d049605ee","Warn: pipCommand not pinned by hash: tools/benchmark/opennmt_ende_wmt14/ctranslate2/Dockerfile:14","Warn: pipCommand not pinned by hash: tools/benchmark/opennmt_ende_wmt14/ctranslate2/Dockerfile:24","Warn: pipCommand not pinned by hash: tools/benchmark/opennmt_ende_wmt14/fastertransformer/Dockerfile:14","Warn: pipCommand not pinned by hash: tools/benchmark/opennmt_ende_wmt14/opennmt_py/Dockerfile:11","Warn: pipCommand not pinned by hash: tools/benchmark/opennmt_ende_wmt14/opennmt_tf/Dockerfile:9","Warn: pipCommand not pinned by hash: tools/benchmark/opus_mt_ende/transformers/Dockerfile:5","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:238","Warn: pipCommand not pinned by hash: .github/workflows/ci.yml:329","Info:   0 out of  14 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   5 third-party GitHubAction dependencies pinned","Info:   0 out of  14 containerImage dependencies pinned","Info:   3 out of  11 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 30 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-15T11:00:18.653Z","repository_id":37248311,"created_at":"2025-08-15T11:00:18.653Z","updated_at":"2025-08-15T11:00:18.653Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000701,"owners_count":26082805,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avx","avx2","cpp","cuda","deep-learning","deep-neural-networks","gemm","inference","intrinsics","machine-translation","mkl","neon","neural-machine-translation","onednn","openmp","opennmt","parallel-computing","quantization","thrust","transformer-models"],"created_at":"2024-09-24T20:30:00.617Z","updated_at":"2025-10-08T19:07:44.979Z","avatar_url":"https://github.com/OpenNMT.png","language":"C++","readme":"[![CI](https://github.com/OpenNMT/CTranslate2/workflows/CI/badge.svg)](https://github.com/OpenNMT/CTranslate2/actions?query=workflow%3ACI) [![PyPI version](https://badge.fury.io/py/ctranslate2.svg)](https://badge.fury.io/py/ctranslate2) [![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://opennmt.net/CTranslate2/) [![Gitter](https://badges.gitter.im/OpenNMT/CTranslate2.svg)](https://gitter.im/OpenNMT/CTranslate2?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge) [![Forum](https://img.shields.io/discourse/status?server=https%3A%2F%2Fforum.opennmt.net%2F)](https://forum.opennmt.net/)\n\n# CTranslate2\n\nCTranslate2 is a C++ and Python library for efficient inference with Transformer models.\n\nThe project implements a custom runtime that applies many performance optimization techniques such as weights quantization, layers fusion, batch reordering, etc., to [accelerate and reduce the memory usage](#benchmarks) of Transformer models on CPU and GPU.\n\nThe following model types are currently supported:\n\n* Encoder-decoder models: Transformer base/big, M2M-100, NLLB, BART, mBART, Pegasus, T5, Whisper\n* Decoder-only models: GPT-2, GPT-J, GPT-NeoX, OPT, BLOOM, MPT, Llama, Mistral, Gemma, CodeGen, GPTBigCode, Falcon, Qwen2\n* Encoder-only models: BERT, DistilBERT, XLM-RoBERTa\n\nCompatible models should be first converted into an optimized model format. The library includes converters for multiple frameworks:\n\n* [OpenNMT-py](https://opennmt.net/CTranslate2/guides/opennmt_py.html)\n* [OpenNMT-tf](https://opennmt.net/CTranslate2/guides/opennmt_tf.html)\n* [Fairseq](https://opennmt.net/CTranslate2/guides/fairseq.html)\n* [Marian](https://opennmt.net/CTranslate2/guides/marian.html)\n* [OPUS-MT](https://opennmt.net/CTranslate2/guides/opus_mt.html)\n* [Transformers](https://opennmt.net/CTranslate2/guides/transformers.html)\n\nThe project is production-oriented and comes with [backward compatibility guarantees](https://opennmt.net/CTranslate2/versioning.html), but it also includes experimental features related to model compression and inference acceleration.\n\n## Key features\n\n* **Fast and efficient execution on CPU and GPU**\u003cbr/\u003eThe execution [is significantly faster and requires less resources](#benchmarks) than general-purpose deep learning frameworks on supported models and tasks thanks to many advanced optimizations: layer fusion, padding removal, batch reordering, in-place operations, caching mechanism, etc.\n* **Quantization and reduced precision**\u003cbr/\u003eThe model serialization and computation support weights with [reduced precision](https://opennmt.net/CTranslate2/quantization.html): 16-bit floating points (FP16), 16-bit brain floating points (BF16), 16-bit integers (INT16), 8-bit integers (INT8) and AWQ quantization (INT4).\n* **Multiple CPU architectures support**\u003cbr/\u003eThe project supports x86-64 and AArch64/ARM64 processors and integrates multiple backends that are optimized for these platforms: [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html), [oneDNN](https://github.com/oneapi-src/oneDNN), [OpenBLAS](https://www.openblas.net/), [Ruy](https://github.com/google/ruy), and [Apple Accelerate](https://developer.apple.com/documentation/accelerate).\n* **Automatic CPU detection and code dispatch**\u003cbr/\u003eOne binary can include multiple backends (e.g. Intel MKL and oneDNN) and instruction set architectures (e.g. AVX, AVX2) that are automatically selected at runtime based on the CPU information.\n* **Parallel and asynchronous execution**\u003cbr/\u003eMultiple batches can be processed in parallel and asynchronously using multiple GPUs or CPU cores.\n* **Dynamic memory usage**\u003cbr/\u003eThe memory usage changes dynamically depending on the request size while still meeting performance requirements thanks to caching allocators on both CPU and GPU.\n* **Lightweight on disk**\u003cbr/\u003eQuantization can make the models 4 times smaller on disk with minimal accuracy loss.\n* **Simple integration**\u003cbr/\u003eThe project has few dependencies and exposes simple APIs in [Python](https://opennmt.net/CTranslate2/python/overview.html) and C++ to cover most integration needs.\n* **Configurable and interactive decoding**\u003cbr/\u003e[Advanced decoding features](https://opennmt.net/CTranslate2/decoding.html) allow autocompleting a partial sequence and returning alternatives at a specific location in the sequence.\n* **Support tensor parallelism for distributed inference**\u003cbr/\u003eVery large model can be split into multiple GPUs. Following this [documentation](docs/parallel.md#model-and-tensor-parallelism) to set up the required environment.\n\nSome of these features are difficult to achieve with standard deep learning frameworks and are the motivation for this project.\n\n## Installation and usage\n\nCTranslate2 can be installed with pip:\n\n```bash\npip install ctranslate2\n```\n\nThe Python module is used to convert models and can translate or generate text with few lines of code:\n\n```python\ntranslator = ctranslate2.Translator(translation_model_path)\ntranslator.translate_batch(tokens)\n\ngenerator = ctranslate2.Generator(generation_model_path)\ngenerator.generate_batch(start_tokens)\n```\n\nSee the [documentation](https://opennmt.net/CTranslate2) for more information and examples.\n\n## Benchmarks\n\nWe translate the En-\u003eDe test set *newstest2014* with multiple models:\n\n* [OpenNMT-tf WMT14](https://opennmt.net/Models-tf/#translation): a base Transformer trained with OpenNMT-tf on the WMT14 dataset (4.5M lines)\n* [OpenNMT-py WMT14](https://opennmt.net/Models-py/#translation): a base Transformer trained with OpenNMT-py on the WMT14 dataset (4.5M lines)\n* [OPUS-MT](https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models/en-de#opus-2020-02-26zip): a base Transformer trained with Marian on all OPUS data available on 2020-02-26 (81.9M lines)\n\nThe benchmark reports the number of target tokens generated per second (higher is better). The results are aggregated over multiple runs. See the [benchmark scripts](tools/benchmark) for more details and reproduce these numbers.\n\n**Please note that the results presented below are only valid for the configuration used during this benchmark: absolute and relative performance may change with different settings.**\n\n#### CPU\n\n| | Tokens per second | Max. memory | BLEU |\n| --- | --- | --- | --- |\n| **OpenNMT-tf WMT14 model** | | | |\n| OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0) | 209.2 | 2653MB | 26.93 |\n| **OpenNMT-py WMT14 model** | | | |\n| OpenNMT-py 3.0.4 (with PyTorch 1.13.1) | 275.8 | 2012MB | 26.77 |\n| - int8 | 323.3 | 1359MB | 26.72 |\n| CTranslate2 3.6.0 | 658.8 | 849MB | 26.77 |\n| - int16 | 733.0 | 672MB | 26.82 |\n| - int8 | 860.2 | 529MB | 26.78 |\n| - int8 + vmap | 1126.2 | 598MB | 26.64 |\n| **OPUS-MT model** | | | |\n| Transformers 4.26.1 (with PyTorch 1.13.1) | 147.3 | 2332MB | 27.90 |\n| Marian 1.11.0 | 344.5 | 7605MB | 27.93 |\n| - int16 | 330.2 | 5901MB | 27.65 |\n| - int8 | 355.8 | 4763MB | 27.27 |\n| CTranslate2 3.6.0 | 525.0 | 721MB | 27.92 |\n| - int16 | 596.1 | 660MB | 27.53 |\n| - int8 | 696.1 | 516MB | 27.65 |\n\nExecuted with 4 threads on a [*c5.2xlarge*](https://aws.amazon.com/ec2/instance-types/c5/) Amazon EC2 instance equipped with an Intel(R) Xeon(R) Platinum 8275CL CPU.\n\n#### GPU\n\n| | Tokens per second | Max. GPU memory | Max. CPU memory | BLEU |\n| --- | --- | --- | --- | --- |\n| **OpenNMT-tf WMT14 model** | | | | |\n| OpenNMT-tf 2.31.0 (with TensorFlow 2.11.0) | 1483.5 | 3031MB | 3122MB | 26.94 |\n| **OpenNMT-py WMT14 model** | | | | |\n| OpenNMT-py 3.0.4 (with PyTorch 1.13.1) | 1795.2 | 2973MB | 3099MB | 26.77 |\n| FasterTransformer 5.3 | 6979.0 | 2402MB | 1131MB | 26.77 |\n| - float16 | 8592.5 | 1360MB | 1135MB | 26.80 |\n| CTranslate2 3.6.0 | 6634.7 | 1261MB | 953MB | 26.77 |\n| - int8 | 8567.2 | 1005MB | 807MB | 26.85 |\n| - float16 | 10990.7 | 941MB | 807MB | 26.77 |\n| - int8 + float16 | 8725.4 | 813MB | 800MB | 26.83 |\n| **OPUS-MT model** | | | | |\n| Transformers 4.26.1 (with PyTorch 1.13.1) | 1022.9 | 4097MB | 2109MB | 27.90 |\n| Marian 1.11.0 | 3241.0 | 3381MB | 2156MB | 27.92 |\n| - float16 | 3962.4 | 3239MB | 1976MB | 27.94 |\n| CTranslate2 3.6.0 | 5876.4 | 1197MB | 754MB | 27.92 |\n| - int8 | 7521.9 | 1005MB | 792MB | 27.79 |\n| - float16 | 9296.7 | 909MB | 814MB | 27.90 |\n| - int8 + float16 | 8362.7 | 813MB | 766MB | 27.90 |\n\nExecuted with CUDA 11 on a [*g5.xlarge*](https://aws.amazon.com/ec2/instance-types/g5/) Amazon EC2 instance equipped with a NVIDIA A10G GPU (driver version: 510.47.03).\n\n## Additional resources\n\n* [Documentation](https://opennmt.net/CTranslate2)\n* [Forum](https://forum.opennmt.net)\n* [Gitter](https://gitter.im/OpenNMT/CTranslate2)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopennmt%2Fctranslate2","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopennmt%2Fctranslate2","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopennmt%2Fctranslate2/lists"}