{"id":17030202,"url":"https://github.com/shenxiangzhuang/bleuscore","last_synced_at":"2025-07-01T03:42:07.514Z","repository":{"id":233451543,"uuid":"787210285","full_name":"shenxiangzhuang/bleuscore","owner":"shenxiangzhuang","description":"BLEU Score in Rust","archived":false,"fork":false,"pushed_at":"2024-05-14T10:24:59.000Z","size":9905,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-15T12:14:43.706Z","etag":null,"topics":["bleu","bleu-score","deep-learning","maturin","ngrams","nlp","pyo3","python","rust","tokenizer"],"latest_commit_sha":null,"homepage":"https://docs.rs/bleuscore","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shenxiangzhuang.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-16T05:14:23.000Z","updated_at":"2024-05-15T12:14:43.707Z","dependencies_parsed_at":"2024-10-14T08:05:03.131Z","dependency_job_id":null,"html_url":"https://github.com/shenxiangzhuang/bleuscore","commit_stats":null,"previous_names":["shenxiangzhuang/bleuscore"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenxiangzhuang%2Fbleuscore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenxiangzhuang%2Fbleuscore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenxiangzhuang%2Fbleuscore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shenxiangzhuang%2Fbleuscore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shenxiangzhuang","download_url":"https://codeload.github.com/shenxiangzhuang/bleuscore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241945475,"owners_count":20046865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bleu","bleu-score","deep-learning","maturin","ngrams","nlp","pyo3","python","rust","tokenizer"],"created_at":"2024-10-14T08:05:01.515Z","updated_at":"2025-07-01T03:42:07.469Z","avatar_url":"https://github.com/shenxiangzhuang.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bleuscore\n\n[![codecov](https://codecov.io/gh/shenxiangzhuang/bleuscore/graph/badge.svg?token=ckgU5oGbxf)](https://codecov.io/gh/shenxiangzhuang/bleuscore)\n[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)\n[![Crates.io](https://img.shields.io/crates/v/bleuscore)](https://crates.io/crates/bleuscore)\n[![PyPI - Version](https://img.shields.io/pypi/v/bleuscore)](https://pypi.org/project/bleuscore/)\n[![npm version](https://img.shields.io/npm/v/bleuscore-js)](https://www.npmjs.com/package/bleuscore-js)\n[![docs.rs](https://img.shields.io/docsrs/bleuscore)](https://docs.rs/bleuscore/0.1.3/bleuscore/)\n\n\n[`bleuscore`](https://github.com/shenxiangzhuang/bleuscore)\nis a fast BLEU score calculator written in rust.\n\n## Installation\nThe python package has been published to [pypi](https://pypi.org/project/bleuscore/),\nso we can install it directly with many ways: \n\n- `pip`\n    ```bash\n    pip install bleuscore\n    ```\n\n- `poetry`\n    ```bash\n    poetry add bleuscore\n    ```\n\n- `uv`\n    ```bash\n    uv pip install bleuscore\n    ```\n\n## Quick Start\nThe usage is exactly same with [huggingface evaluate](https://huggingface.co/spaces/evaluate-metric/bleu):\n\n```diff\n- import evaluate\n+ import bleuscore\n\npredictions = [\"hello there general kenobi\", \"foo bar foobar\"]\nreferences = [\n    [\"hello there general kenobi\", \"hello there !\"],\n    [\"foo bar foobar\"]\n]\n\n- bleu = evaluate.load(\"bleu\")\n- results = bleu.compute(predictions=predictions, references=references)\n+ results = bleuscore.compute(predictions=predictions, references=references)\n\nprint(results)\n# {'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, \n# 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}\n\n```\n\n## Benchmark\n\n**TLDR: We got more than 10x speedup when the corpus size beyond 100K**\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./benchmark/bench.png\" alt=\"Benchmark\" width=\"400\" height=\"300\"\u003e\n\u003c/p\u003e\n\nWe use the demo data shown in quick start to do this simple benchmark.\nYou can check the [benchmark/simple](./benchmark/simple) for the benchmark source code.\n\n- `rs_bleuscore`: bleuscore python library\n- `local_hf_bleu`: huggingface evaluate bleu algorithm in **local**\n- `sacre_bleu`: sacrebleu\n  - Note that we got different result with sacrebleu in the simple demo data and all the rests have same result\n- `hf_evaluate`: huggingface evaluate bleu algorithm with **evaluate** package\n\n\nThe `N` is used to enlarge the predictions/references size by simply duplication the demo data as shown before.\nWe can see that as `N` increase, the bleuscore gets better performance.\nYou can navigate [benchmark](./benchmark/README.md) for more benchmark details.\n\n\n\n### N=100\n\n```bash\nhyperfine --warmup 5 --runs 10   \\\n  \"python simple/rs_bleuscore.py 100\" \\\n  \"python simple/local_hf_bleu.py 100\" \\\n  \"python simple/sacre_bleu.py 100\"   \\\n  \"python simple/hf_evaluate.py 100\"\n\nBenchmark 1: python simple/rs_bleuscore.py 100\n  Time (mean ± σ):      19.0 ms ±   2.6 ms    [User: 17.8 ms, System: 5.3 ms]\n  Range (min … max):    14.8 ms …  23.2 ms    10 runs\n\nBenchmark 2: python simple/local_hf_bleu.py 100\n  Time (mean ± σ):      21.5 ms ±   2.2 ms    [User: 19.0 ms, System: 2.5 ms]\n  Range (min … max):    16.8 ms …  24.1 ms    10 runs\n\nBenchmark 3: python simple/sacre_bleu.py 100\n  Time (mean ± σ):      45.9 ms ±   2.2 ms    [User: 38.7 ms, System: 7.1 ms]\n  Range (min … max):    43.5 ms …  50.9 ms    10 runs\n\nBenchmark 4: python simple/hf_evaluate.py 100\n  Time (mean ± σ):      4.504 s ±  0.429 s    [User: 0.762 s, System: 0.823 s]\n  Range (min … max):    4.163 s …  5.446 s    10 runs\n\nSummary\n  python simple/rs_bleuscore.py 100 ran\n    1.13 ± 0.20 times faster than python simple/local_hf_bleu.py 100\n    2.42 ± 0.35 times faster than python simple/sacre_bleu.py 100\n  237.68 ± 39.88 times faster than python simple/hf_evaluate.py 100\n```\n\n### N = 1K ~ 1M\n\n| Command                                  |       Mean [ms] | Min [ms] | Max [ms] |        Relative |\n|:-----------------------------------------|----------------:|---------:|---------:|----------------:|\n| `python simple/rs_bleuscore.py 1000`     |      20.3 ± 1.3 |     18.2 |     21.4 |            1.00 |\n| `python simple/local_hf_bleu.py 1000`    |      45.8 ± 1.2 |     44.2 |     47.5 |     2.26 ± 0.16 |\n| `python simple/rs_bleuscore.py 10000`    |      37.8 ± 1.5 |     35.9 |     39.5 |     1.87 ± 0.14 |\n| `python simple/local_hf_bleu.py 10000`   |     295.0 ± 5.9 |    288.6 |    304.2 |    14.55 ± 0.98 |\n| `python simple/rs_bleuscore.py 100000`   |     219.6 ± 3.3 |    215.3 |    224.0 |    10.83 ± 0.72 |\n| `python simple/local_hf_bleu.py 100000`  |   2781.4 ± 42.2 |   2723.1 |   2833.0 |   137.13 ± 9.10 |\n| `python simple/rs_bleuscore.py 1000000`  |   2048.8 ± 31.4 |   2013.2 |   2090.3 |   101.01 ± 6.71 |\n| `python simple/local_hf_bleu.py 1000000` | 28285.3 ± 100.9 |  28182.1 |  28396.1 | 1394.51 ± 90.21 |\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshenxiangzhuang%2Fbleuscore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshenxiangzhuang%2Fbleuscore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshenxiangzhuang%2Fbleuscore/lists"}