{"id":27155249,"url":"https://github.com/xhluca/bm25s","last_synced_at":"2026-04-29T04:08:40.355Z","repository":{"id":244873116,"uuid":"784528890","full_name":"xhluca/bm25s","owner":"xhluca","description":"Fast BM25 search in Python, powered by Numpy and Numba","archived":false,"fork":false,"pushed_at":"2026-04-24T04:02:47.000Z","size":2219,"stargazers_count":1637,"open_issues_count":1,"forks_count":99,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-04-24T06:02:58.794Z","etag":null,"topics":["bm25","bm25-l","bm25-plus","information-retrieval","lexical-search","okapi-bm25","rag","retrieval","robertson","search"],"latest_commit_sha":null,"homepage":"https://bm25s.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xhluca.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-04-10T02:55:05.000Z","updated_at":"2026-04-24T04:02:15.000Z","dependencies_parsed_at":"2024-06-18T01:34:35.617Z","dependency_job_id":"0ffbf428-8beb-4445-8d74-79dd7bb3c37a","html_url":"https://github.com/xhluca/bm25s","commit_stats":null,"previous_names":["xhluca/bm25s"],"tags_count":57,"template":false,"template_full_name":null,"purl":"pkg:github/xhluca/bm25s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xhluca%2Fbm25s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xhluca%2Fbm25s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xhluca%2Fbm25s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xhluca%2Fbm25s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xhluca","download_url":"https://codeload.github.com/xhluca/bm25s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xhluca%2Fbm25s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32410019,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T03:46:11.172Z","status":"ssl_error","status_checked_at":"2026-04-29T03:37:55.317Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bm25","bm25-l","bm25-plus","information-retrieval","lexical-search","okapi-bm25","rag","retrieval","robertson","search"],"created_at":"2025-04-08T19:00:48.134Z","updated_at":"2026-04-29T04:08:40.347Z","avatar_url":"https://github.com/xhluca.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003ch1\u003eBM25S⚡\u003c/h1\u003e\n\n\u003ci\u003eBM25S (or BM25-Sparse) is an ultrafast implementation of BM25 in pure Python, powered by Numpy\u003c/i\u003e\n\n\u003ctable\u003e\n      \u003ctr\u003e\n            \u003ctd\u003e\n                  \u003ca href=\"https://github.com/xhluca/bm25s\"\u003e💻 GitHub\u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd\u003e\n                  \u003ca href=\"https://bm25s.github.io\"\u003e🏠 Homepage\u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd\u003e\n                  \u003ca href=\"https://arxiv.org/abs/2407.03618\"\u003e📝 Technical Report\u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd\u003e\n                  \u003ca href=\"https://huggingface.co/blog/xhluca/bm25s\"\u003e🤗 Blog Post\u003c/a\u003e\n            \u003c/td\u003e\n            \u003ctd\u003e\n                  \u003ca href=\"#installation\"\u003e🛠️ Installation\u003c/a\u003e\n            \u003c/td\u003e\n      \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003ca href=\"https://pepy.tech/projects/bm25s\"\u003e\u003cimg src=\"https://static.pepy.tech/badge/bm25s\" alt=\"PyPI Downloads\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/bm25s/\"\u003e\u003cimg alt=\"PyPI - Version\" src=\"https://img.shields.io/pypi/v/bm25s\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/xhluca/bm25s/blob/main/LICENSE\"\u003e\u003cimg alt=\"GitHub License\" src=\"https://img.shields.io/github/license/xhluca/bm25s?link=https%3A%2F%2Fgithub.com%2Fxhluca%2Fbm25s%2Fblob%2Fmain%2FLICENSE\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/xhluca/bm25s/pulls?q=is%3Apr+is%3Aclosed\"\u003e\u003cimg alt=\"GitHub Issues or Pull Requests\" src=\"https://img.shields.io/github/issues-pr-closed/xhluca/bm25s\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/xhluca/bm25s/discussions\"\u003e\u003cimg alt=\"GitHub Discussions\" src=\"https://img.shields.io/github/discussions/xhluca/bm25s?link=https%3A%2F%2Fgithub.com%2Fxhluca%2Fbm25s%2Fdiscussions\"\u003e\u003c/a\u003e\n\n\n\u003c/div\u003e\n\nWelcome to `bm25s`, a library that implements BM25 in Python, allowing you to rank documents based on a query. BM25 is a widely used ranking function used for text retrieval tasks, and is a core component of search services like Elasticsearch.\n\nIt is designed to be:\n* **Fast**: `bm25s` is implemented in pure Python and leverage sparse matrices to store eagerly computed scores for all document tokens. This allows extremely fast scoring at query time, improving performance over popular libraries by orders of magnitude (see benchmarks below).\n* **Simple**: `bm25s` is designed to be easy to use and understand. You can install it with pip and start using it in minutes. There is no dependencies on Java or Pytorch - all you need is Numpy, and optional lightweight dependencies for stemming and speedup via `numba` compilation.\n\nBelow, we compare `bm25s` with Elasticsearch in terms of speedup over `rank-bm25`, the most popular Python implementation of BM25. We measure the throughput in queries per second (QPS) on a few popular datasets from [BEIR](https://github.com/beir-cellar/beir) in a single-threaded setting.\n\n![comparison](assets/comparison.png)\n\n\u003e [!IMPORTANT]\n\u003e New in version 0.2.0: We are rolling out support for a numba backend, which gives around [2x speedup for larger datasets](https://github.com/xhluca/bm25-benchmarks?tab=readme-ov-file#queries-per-second)! Learn more about it and share your thoughts in [the version 0.2.0 release thread](https://github.com/xhluca/bm25s/discussions/58).\n\n\n\u003cdetails open\u003e\n\u003csummary\u003eShow/Hide citation\u003c/summary\u003e\u003cbr\u003e\n\n```\n@misc{bm25s,\n      title={BM25S: Orders of magnitude faster lexical search via eager sparse scoring}, \n      author={Xing Han Lù},\n      year={2024},\n      eprint={2407.03618},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https://arxiv.org/abs/2407.03618},\n}\n```\n\n\u003c/details\u003e\n\n## Installation\n\nYou can install `bm25s` with pip:\n\n```bash\npip install bm25s\n```\n\nYou can install the recommended (but optional) dependencies:\n\n```bash\n# HIGHLY RECOMMENDED: To install all core dependencies (json loading, progress bar, stemming, JIT compilation)\npip install \"bm25s[core]\"\n\n# If you just want to use stemming for better results, you can install a stemmer\npip install PyStemmer\n\n# Install all extra dependencies\npip install \"bm25s[full]\"\n\n```\n\n## Quickstart\n\nHere is a simple example of how to use `bm25s`:\n\n```python\nimport bm25s\nimport Stemmer  # optional: for stemming\n\n# Create your corpus here\ncorpus = [\n    \"a cat is a feline and likes to purr\",\n    \"a dog is the human's best friend and loves to play\",\n    \"a bird is a beautiful animal that can fly\",\n    \"a fish is a creature that lives in water and swims\",\n]\n\n# optional: create a stemmer\nstemmer = Stemmer.Stemmer(\"english\")\n\n# Tokenize the corpus and only keep the ids (faster and saves memory)\ncorpus_tokens = bm25s.tokenize(corpus, stopwords=\"en\", stemmer=stemmer)\n\n# Create the BM25 model and index the corpus\nretriever = bm25s.BM25()\nretriever.index(corpus_tokens)\n\n# Query the corpus\nquery = \"does the fish purr like a cat?\"\nquery_tokens = bm25s.tokenize(query, stemmer=stemmer)\n\n# Get top-k results as a tuple of (doc ids, scores). Both are arrays of shape (n_queries, k).\n# To return docs instead of IDs, set the `corpus=corpus` parameter.\nresults, scores = retriever.retrieve(query_tokens, k=2)\n\nfor i in range(results.shape[1]):\n    doc, score = results[0, i], scores[0, i]\n    print(f\"Rank {i+1} (score: {score:.2f}): {doc}\")\n\n# You can save the arrays to a directory...\nretriever.save(\"animal_index_bm25\")\n\n# You can save the corpus along with the model\nretriever.save(\"animal_index_bm25\", corpus=corpus)\n\n# ...and load them when you need them\nimport bm25s\nreloaded_retriever = bm25s.BM25.load(\"animal_index_bm25\", load_corpus=True)\n# set load_corpus=False if you don't need the corpus\n```\n\n### Corpus Format\n\n`bm25s` separates the text you index from the values you get back. Pass\nstrings to `bm25s.tokenize()`, then pass the resulting tokenized corpus to\n`retriever.index()`. The optional `corpus` argument on `BM25(...)`,\n`retrieve(...)`, and `save(...)` is the list of values returned for matching\ndocument IDs.\n\nBoth plain strings and dictionaries are valid corpus entries:\n\n```python\ntext_corpus = [\n    \"a cat is a feline and likes to purr\",\n    \"a dog is the human's best friend and loves to play\",\n]\n\nmetadata_corpus = [\n    {\"id\": \"cat-doc\", \"title\": \"About Cat\", \"text\": text_corpus[0]},\n    {\"id\": \"dog-doc\", \"title\": \"About Dog\", \"text\": text_corpus[1]},\n]\n\n# Pick the text field(s) you want to index.\ncorpus_tokens = bm25s.tokenize([doc[\"text\"] for doc in metadata_corpus])\n\n# Retrieved documents will be the original dictionary entries.\nretriever = bm25s.BM25(corpus=metadata_corpus)\nretriever.index(corpus_tokens)\n```\n\nDictionaries do not have required keys; use the shape that fits your\napplication, such as `id`, `title`, `text`, or nested metadata. Retrieval is\nposition-based, so `corpus[i]` is returned for document ID `i`. Keep the corpus\nyou pass to `BM25(...)`, `retrieve(...)`, or `save(...)` in the same order and\nlength as the indexed documents. When saving, entries must be strings,\ndictionaries, lists, or tuples that can be serialized to JSON. String entries\nare written to `corpus.jsonl` as `{\"id\": i, \"text\": doc}`; dictionaries, lists,\nand tuples are written as provided.\n\nFor an example that shows how to quickly index a 2M-documents corpus (Natural Questions), check out [`examples/index_nq.py`](examples/index_nq.py).\n\n## High Level API\n\n\u003e [!TIP]\n\u003e **New:** We now recommend using the [**`BM25`** package on PyPI](https://pypi.org/project/BM25/) for a simpler, beginner-friendly experience. It includes all necessary dependencies (stemming, CLI, etc.) and provides the same high-level API shown below.\n\nIf you want to quickly search on a local file, you can use the `bm25s.high_level` module:\n\n```python\nimport bm25s.high_level as bm25\n\n# Load a file (csv, json, jsonl, txt)\n# For csv/jsonl, you can specify the column/key to use as document text\ncorpus = bm25.load(\"tests/data/dummy.csv\", document_column=\"text\")\n# Index the corpus\nretriever = bm25.index(corpus)\n\n# Search\nresults = retriever.search([\"your query here\"], k=5)\nfor result in results[0]:\n    print(result)\n```\n\nThe `load` function handles file reading, while `index` handles tokenization, indexing, and provides a simple search interface.\n\n## Command-Line Interface\n\n`bm25s` provides a terminal-based CLI for quick indexing and searching without writing Python code.\n\n### Indexing Documents\n\nCreate an index from a CSV, TXT, JSON, or JSONL file:\n\n```bash\n# Index a CSV file (uses first column by default)\nbm25 index documents.csv -o my_index\n\n# Index with a specific column\nbm25 index documents.csv -o my_index -c text\n\n# Index a text file (one document per line)\nbm25 index documents.txt -o my_index\n\n# Index a JSONL file\nbm25 index documents.jsonl -o my_index -c content\n```\n\nIf you don't specify an output directory with `-o`, the index will be saved to `\u003cfilename\u003e_index`.\n\n### User Directory\n\nYou can save indices to a central user directory (`~/.bm25s/indices/`) using the `-u` flag:\n\n```bash\n# Save index to ~/.bm25s/indices/my_docs\nbm25 index documents.csv -u -o my_docs\n\n# Search using the user directory\nbm25 search -u -i my_docs \"your query\"\n```\n\n### Searching\n\nSearch an existing index with a query using `-i` (or `--index`):\n\n```bash\n# Basic search (returns top 10 results)\nbm25 search -i my_index \"what is machine learning?\"\n\n# Search with full path\nbm25 search -i ./path/to/my_index \"your query here\"\n\n# Return more results\nbm25 search -i my_index \"your query here\" -k 20\n\n# Save results to a JSON file\nbm25 search -i my_index \"your query here\" -s results.json\n```\n\n### Interactive Index Picker\n\nWhen using `-u` without specifying an index name, an interactive picker is displayed (requires `bm25s[cli]`):\n\n```bash\n# Install CLI extras for colored interactive picker\npip install \"bm25s[cli]\"\n\n# Interactive picker will show available indices\nbm25 search -u \"your query\"\n```\n\n### Example Workflow\n\n**Basic usage** (index saved to current directory):\n\n```bash\n# 1. Create a simple text file with documents\necho -e \"Machine learning is a subset of AI\\nDeep learning uses neural networks\\nNatural language processing handles text\" \u003e docs.txt\n\n# 2. Index the documents\nbm25 index docs.txt -o my_index\n\n# 3. Search the index\nbm25 search -i my_index \"what is AI?\"\n```\n\n**With user directory** (indices saved to `~/.bm25s/indices/`):\n\n```bash\n# Index to user directory\nbm25 index docs.txt -u -o ml_docs\n\n# Search from user directory\nbm25 search -u -i ml_docs \"what is AI?\"\n\n# Or use the interactive picker\nbm25 search -u \"what is AI?\"\n```\n\n### CLI Reference\n\n| Command | Flag | Description |\n|---------|------|-------------|\n| `index` | `-o, --output` | Output directory for the index |\n| `index` | `-c, --column` | Column name for document text |\n| `index` | `-u, --user` | Save to user directory (`~/.bm25s/indices/`) |\n| `search` | `-i, --index` | Path to index (or name if using `-u`) |\n| `search` | `-k, --top-k` | Number of results (default: 10) |\n| `search` | `-s, --save` | Save results to JSON file |\n| `search` | `-u, --user` | Use user directory; shows picker if `-i` omitted |\n\n## Flexibility\n\n`bm25s` provides a flexible API that allows you to customize the BM25 model and the tokenization process. Here are some of the options you can use:\n\n```python\n# You can provide a list of queries instead of a single query\nqueries = [\"What is a cat?\", \"is the bird a dog?\"]\n\n# Provide your own stopwords list if you don't like the default one\nstopwords = [\"a\", \"the\"]\n\n# For stemming, use any function that is callable on each word list\nstemmer_fn = lambda lst: [word for word in lst]\n\n# Tokenize the queries\nquery_token_ids = bm25s.tokenize(queries, stopwords=stopwords, stemmer=stemmer_fn)\n\n# If you want the tokenizer to return strings instead of token ids, you can do this\nquery_token_strs = bm25s.tokenize(queries, return_ids=False)\n\n# You can use a different corpus for retrieval, e.g., titles instead of full docs\ntitles = [\"About Cat\", \"About Dog\", \"About Bird\", \"About Fish\"]\n\n# You can also choose to only return the documents and omit the scores\n# note: if you pass a new corpus here, it must have the same length as your indexed corpus\nresults = retriever.retrieve(query_token_ids, corpus=titles, k=2, return_as=\"documents\")\n\n# The documents are returned as a numpy array of shape (n_queries, k)\nfor i in range(results.shape[1]):\n    print(f\"Rank {i+1}: {results[0, i]}\")\n```\n\n### Memory Efficient Retrieval\n\n`bm25s` is designed to be memory efficient. You can use the `mmap` option to load the BM25 index as a memory-mapped file, which allows you to load the index without loading the full index into memory. This is useful when you have a large index and want to save memory:\n\n```python\n# Create a BM25 index\n# ...\n\n# let's say you have a large corpus\ncorpus = [\n    \"a very long document that is very long and has many words\",\n    \"another long document that is long and has many words\",\n    # ...\n]\n# Save the BM25 index to a file\nretriever.save(\"bm25s_very_big_index\", corpus=corpus)\n\n# Load the BM25 index as a memory-mapped file, which is memory efficient\n# and reduce overhead of loading the full index into memory\nretriever = bm25s.BM25.load(\"bm25s_very_big_index\", mmap=True)\n```\n\nFor an example of how to use retrieve using the `mmap=True` mode, check out [`examples/retrieve_nq.py`](examples/retrieve_nq.py).\n\n\n## Tokenization\n\nIn addition to using the simple function `bm25s.tokenize`, you can also use the `Tokenizer` class to customize the tokenization process. This is useful when you want to use a different tokenizer, or when you want to use a different tokenization process for queries and documents:\n\n```python\nfrom bm25s.tokenization import Tokenizer\n\ncorpus = [\n      \"a cat is a feline and likes to purr\",\n      \"a dog is the human's best friend and loves to play\",\n      \"a bird is a beautiful animal that can fly\",\n      \"a fish is a creature that lives in water and swims\",\n]\n\n# Pick your favorite stemmer, and pass \nstemmer = None\nstopwords = [\"is\"]\nsplitter = lambda x: x.split() # function or regex pattern\n# Create a tokenizer\ntokenizer = Tokenizer(\n      stemmer=stemmer, stopwords=stopwords, splitter=splitter\n)\n\ncorpus_tokens = tokenizer.tokenize(corpus)\n\n# let's see what the tokens look like\nprint(\"tokens:\", corpus_tokens)\nprint(\"vocab:\", tokenizer.get_vocab_dict())\n\n# note: the vocab dict will either be a dict of `word -\u003e id` if you don't have a stemmer, and a dict of `stemmed word -\u003e stem id` if you do.\n# You can save the vocab. it's fine to use the same dir as your index if filename doesn't conflict\ntokenizer.save_vocab(save_dir=\"bm25s_very_big_index\")\n\n# loading:\nnew_tokenizer = Tokenizer(stemmer=stemmer, stopwords=[], splitter=splitter)\nnew_tokenizer.load_vocab(\"bm25s_very_big_index\")\nprint(\"vocab reloaded:\", new_tokenizer.get_vocab_dict())\n\n# the same can be done for stopwords\nprint(\"stopwords before reload:\", new_tokenizer.stopwords)\ntokenizer.save_stopwords(save_dir=\"bm25s_very_big_index\")\nnew_tokenizer.load_stopwords(\"bm25s_very_big_index\")\nprint(\"stopwords reloaded:\", new_tokenizer.stopwords)\n```\n\nYou can find advanced examples in [examples/tokenizer_class.py](examples/tokenizer_class.py), including how to:\n* Pass a stemmer, stopwords, and splitter function/regex pattern\n* Control whether vocabulary is updated by `tokenizer.tokenize` calls or not (by default, it will only be updated during the first call)\n* Reset the tokenizer to its initial state with `tokenizer.reset_vocab()`\n* Use the tokenizer in generator mode to save memory by `yield`ing one document at a time.\n* Pass different outputs of the tokenizer to the `BM25.retrieve` function.\n\n\n\n## Variants\n\nYou can use the following variants of BM25 in `bm25s` (see [Kamphuis et al. 2020](https://link.springer.com/chapter/10.1007/978-3-030-45442-5_4) for more details):\n\n* Original implementation (`method=\"robertson\"`) - we set `idf\u003e=0` to avoid negatives\n* ATIRE (`method=\"atire\"`)\n* BM25L (`method=\"bm25l\"`)\n* BM25+ (`method=\"bm25+\"`)\n* Lucene (`method=\"lucene\"`)\n\nBy default, `bm25s` uses `method=\"lucene\"`, which is Lucene's BM25 implementation (exact version). You can change the method by passing the `method` argument to the `BM25` constructor:\n\n```python\n# The IR book recommends default values of k1 between 1.2 and 2.0, and b=0.75\nretriever = bm25s.BM25(method=\"robertson\", k1=1.5, b=0.75)\n\n# For BM25+, BM25L, you need a delta parameter (default is 0.5)\nretriever = bm25s.BM25(method=\"bm25+\", delta=1.5)\n\n# You can also choose a different \"method\" for idf, while keeping the default for the rest\n# for example, this is equivalent to rank-bm25 when `epsilon=0`\nretriever = bm25s.BM25(method=\"atire\", idf_method=\"robertson\")\n# and this is equivalent to bm25-pt\nretriever = bm25s.BM25(method=\"atire\", idf_method=\"lucene\")\n```\n\n## Hugging Face Integration\n\n`bm25` can naturally work with Hugging Face's `huggingface_hub`, allowing you to load and save to the model hub. This is useful for sharing BM25 indices and using community models. \n\nFirst, make sure you have a valid [access token for the Hugging Face model hub](https://huggingface.co/settings/tokens). This is needed to save models to the hub, or to load private models. Once you created it, you can add it to your environment variables (e.g. in your `.bashrc` or `.zshrc`):\n\n```bash\nexport HUGGING_FACE_HUB_TOKEN=\"hf_...\"\n```\n\nNow, let's install the `huggingface_hub` library:\n\n```bash\npip install huggingface_hub\n```\n\nLet's see how to use `BM25SHF.save_to_hub` to save a BM25 index to the Hugging Face model hub:\n\n```python\nimport os\nimport bm25s\nfrom bm25s.hf import BM25HF\n\n# Create a BM25 index\nretriever = BM25HF()\n# Create your corpus here\ncorpus = [\n    \"a cat is a feline and likes to purr\",\n    \"a dog is the human's best friend and loves to play\",\n    \"a bird is a beautiful animal that can fly\",\n    \"a fish is a creature that lives in water and swims\",\n]\ncorpus_tokens = bm25s.tokenize(corpus)\nretriever.index(corpus_tokens)\n\n# Set your username and token\nuser = \"your-username\"\ntoken = os.environ[\"HF_TOKEN\"]\nretriever.save_to_hub(f\"{user}/bm25s-animals\", token=token, corpus=corpus)\n# You can also save it publicly with private=False\n```\n\nThen, you can use the following code to load a BM25 index from the Hugging Face model hub:\n\n```python\nimport bm25s\nfrom bm25s.hf import BM25HF\n\n# Load a BM25 index from the Hugging Face model hub\nuser = \"your-username\"\nretriever = BM25HF.load_from_hub(f\"{user}/bm25s-animals\")\n\n# you can specify revision and load_corpus=True if needed\nretriever = BM25HF.load_from_hub(\n    f\"{user}/bm25s-animals\", revision=\"main\", load_corpus=True\n)\n\n# if you want a low-memory usage, you can load as memory map with `mmap=True`\nretriever = BM25HF.load_from_hub(\n    f\"{user}/bm25s-animals\", load_corpus=True, mmap=True\n)\n\n# Query the corpus\nquery = \"does the fish purr like a cat?\"\n\n# Tokenize the query\nquery_tokens = bm25s.tokenize(query)\n\n# Get top-k results as a tuple of (doc ids, scores). Both are arrays of shape (n_queries, k)\nresults, scores = retriever.retrieve(query_tokens, k=2)\n```\n\nFor a complete example, check out:\n* [`examples/index_to_hf.py`](examples/index_to_hf.py) for indexing a corpus and upload to Huggingface Hub\n* [`examples/retrieve_from_hf.py`](examples/retrieve_from_hf.py) for loading an index alongside corpus from Huggingface Hub and querying it.\n* [`examples/mcp/create_index.py`](examples/mcp/create_index.py) for creating a test index for the MCP server.\n* [`examples/mcp/verify_server.py`](examples/mcp/verify_server.py) for programmatically verifying the MCP server.\n\n## MCP Server\n\n`bm25s` comes with a built-in Model Context Protocol (MCP) server, allowing you to expose your BM25 index as a tool for LLMs and other agents.\n\n### Installation\n\nTo use the MCP server, you need to install `bm25s` with the `mcp` extra. We recommend using `uv` for managing your environment:\n\n```bash\n# Create a virtual environment\nuv venv\n\n# Activate the virtual environment\nsource .venv/bin/activate\n\n# Install bm25s with the mcp extra\nuv pip install \"bm25s[mcp]\"\n\n# or locally:\nuv pip install -e \".[mcp]\"\n```\n\n### Launching the Server\n\nYou can launch the MCP server using the `bm25` CLI:\n\n```bash\nsource .venv/bin/activate\nbm25 mcp launch --port 8000 --index-dir /path/to/your/index\n```\n\n### Example\n\nFor example, you can create a test index and launch the server with:\n\n```bash\nsource .venv/bin/activate\npython examples/index_nq.py # creates bm25s_indices/nq\nbm25 mcp launch --port 8000 --index-dir ./bm25s_indices/nq\n```\n\nThen, you can verify the server with:\n\n```bash\npython examples/mcp/verify_server.py\n```\n\n### Tools\n\nThe server exposes the following tools:\n\n*   `retrieve(query: str, k: int = 10)`: Retrieves the top-k documents for a given query.\n*   `get_info()`: Returns information about the loaded index (vocabulary size, number of documents, backend).\n\n### Example Usage with Claude Desktop\n\nTo use `bm25s` with Claude Desktop, add the following to your `claude_desktop_config.json` (located at `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):\n\n```json\n{\n  \"mcpServers\": {\n    \"bm25s\": {\n      \"command\": \"/absolute/path/to/uv\",\n      \"args\": [\n        \"--directory\",\n        \"/ABSOLUTE/PATH/TO/PARENT/FOLDER/bm25s\",\n        \"run\",\n        \"bm25\",\n        \"mcp\",\n        \"launch\",\n        \"--index-dir\",\n        \"/absolute/path/to/your/index\"\n      ]\n    }\n  }\n}\n```\n\nTo find the absolute path to `uv`, you can run `which uv` in the terminal.\n\n## Comparison\n\nHere are some benchmarks comparing `bm25s` to other popular BM25 implementations. We compare the following implementations:\n* `bm25s`: Our implementation of BM25 in pure Python, powered by Numpy and sparse matrices.\n* `rank-bm25` (`Rank`): A popular Python implementation of BM25.\n* `bm25_pt` (`PT`): A Pytorch implementation of BM25.\n* `elasticsearch` (`ES`): Elasticsearch with BM25 configurations.\n\n`OOM` means the implementation ran out of memory during the benchmark.\n\n### Throughput (Queries per second)\n\nWe compare the throughput of the BM25 implementations on various datasets. The throughput is measured in queries per second (QPS), on a single-threaded Intel Xeon CPU @ 2.70GHz (found on Kaggle). For BM25S, we take the average of 10 runs. Instances exceeding 60 queries/s are in **bold**.\n\n| Dataset          |       BM25S | Elastic |    BM25-PT |  Rank-BM25 |\n| :--------------- | ----------: | ------: | ---------: | ---------: |\n| arguana          |  **573.91** |   13.67 | **110.51** |          2 |\n| climate-fever    |       13.09 |    4.02 |        OOM |       0.03 |\n| cqadupstack      |  **170.91** |   13.38 |        OOM |       0.77 |\n| dbpedia-entity   |       13.44 |   10.68 |        OOM |       0.11 |\n| fever            |       20.19 |    7.45 |        OOM |       0.06 |\n| fiqa             |  **507.03** |   16.96 |      20.52 |       4.46 |\n| hotpotqa         |       20.88 |    7.11 |        OOM |       0.04 |\n| msmarco          |        12.2 |   11.88 |        OOM |       0.07 |\n| nfcorpus         | **1196.16** |   45.84 |     256.67 | **224.66** |\n| nq               |       41.85 |   12.16 |        OOM |        0.1 |\n| quora            |  **183.53** |    21.8 |       6.49 |       1.18 |\n| scidocs          |  **767.05** |   17.93 |      41.34 |       9.01 |\n| scifact          |  **952.92** |   20.81 |  **184.3** |       47.6 |\n| trec-covid       |   **85.64** |    7.34 |       3.73 |       1.48 |\n| webis-touche2020 |   **60.59** |   13.53 |        OOM |        1.1 |\n\nMore detailed benchmarks can be found in the [bm25-benchmarks repo](https://github.com/xhluca/bm25-benchmarks).\n\n### Disk usage\n\n`bm25s` is designed to be lightweight. This means the total disk usage of the package is minimal, as it only requires wheels for `numpy` (18MB), and the package itself is less than 100KB. After installation, the full virtual environment takes more space than `rank-bm25` but less than `pyserini` and `bm25_pt`:\n\n| Package           | Disk Usage |\n| ----------------- | ---------- |\n| venv (no package) | 21MB       |\n| `bm25s` (ours)    | 51MB       |\n| `rank-bm25`       | 51MB       |\n| `bm25s` (w/ core) | 188MB      |\n| `elastic`         | 1183MB     |\n| `bm25_pt`         | 5346MB     |\n| `pyserini`        | 6976MB     |\n\n\u003cdetails\u003e\n\u003csummary\u003eShow Details\u003c/summary\u003e\n\nThe disk usage of the virtual environments is calculated using the following command:\n\n```\n$ du -sh venv* --block-size=1MB\n6976    conda-env-pyserini\n5346    venv-bm25-pt\n51     venv-bm25s\n188     venv-bm25s-core\n21      venv-empty\n51      venv-rank-bm25\n```\n\nFor `pyserini`, we use the [recommended installation](https://github.com/castorini/pyserini/blob/master/docs/installation.md) with conda environment to account for Java dependencies.\n\n\u003c/details\u003e\n\n### Optimized RAM usage\n\n`bm25s` allows considerable memory saving through the use of *memory-mapping*, which allows the index to be stored on disk and loaded on demand. \n\nUsing the `index_nq.py` to create an index, we can retrieve with:\n* `examples/retrieve_nq.py`: setting `mmap=False` in the `main` function to load the index in memory, and `mmap=True` to load the index as a memory-mapped file. \n* `examples/retrieve_nq_with_batching.py`: This takes it a step further by batching the retrieval process, which allows for reloading the index after each batch (see *Mmap+Reload* below). This is useful when you have a large index and want to save memory.\n\nWe show the following results on the NQ dataset (2M+ documents, 100M+ tokens):\n\n| Method        | Load Index (s) | Retrieval (s) | RAM post-index (GB) | RAM post-retrieve (GB) |\n| ------------- | -------------- | ------------- | ------------------- | ---------------------- |\n| In-memory     | 8.61           | 21.09         | 4.36                | 4.45                   |\n| Memory-mapped | 0.53           | 20.22         | 0.49                | 2.16                   |\n| Mmap+Reload   | 0.48           | 20.96         | 0.49                | 0.70                   |\n\nWe can see that memory-mapping the index allows for a significant reduction in memory usage, with comparable retrieval times. \n\nSimilarly, for MSMARCO (8M+ documents, 300M+ tokens), we show the following results (running on the validation set), although the retrieval did not complete for the in-memory case:\n\n| Method        | Load Index (s) | Retrieval (s) | RAM post-index (GB) | RAM post-retrieve (GB) |\n| ------------- | -------------- | ------------- | ------------------- | ---------------------- |\n| In-memory     | 25.71          | 93.66         | 10.21               | 10.34                  |\n| Memory-mapped | 1.24           | 90.41         | 1.14                | 4.88                   |\n| Mmap+Reload   | 1.17           | 97.89         | 1.14                | 1.38                   |\n\n## Acknowledgement\n\n* The central idea behind the scoring mechanism in this library is originally from [bm25_pt](https://github.com/jxmorris12/bm25_pt), which was a major inspiration to this project.\n* The API of the [`BM25` class](https://github.com/xhluca/bm25s/blob/main/bm25s/__init__.py) is also heavily inspired by the design of BM25-pt, as well as that of rank-bm25.\n* The multilingual stopwords are sourced from the [NLTK stopwords lists](https://github.com/nltk/nltk/blob/96ee715997e1c8d9148b6d8e1b32f412f31c7ff7/nltk/corpus/__init__.py#L315).\n* The numba implementation are inspired by numba implementations originally proposed by [baguetter](https://github.com/mixedbread-ai/baguetter) and [retriv](https://github.com/AmenRa/retriv).\n* The function `bm25s.utils.beir.evaluate` is taken from the [BEIR library](https://github.com/beir-cellar/beir). It follows the same license as the BEIR library, which is Apache 2.0.\n\n## Citation\n\nIf you use `bm25s` in your work, please use the following bibtex:\n\n```\n@misc{bm25s,\n      title={BM25S: Orders of magnitude faster lexical search via eager sparse scoring}, \n      author={Xing Han Lù},\n      year={2024},\n      eprint={2407.03618},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https://arxiv.org/abs/2407.03618}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxhluca%2Fbm25s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxhluca%2Fbm25s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxhluca%2Fbm25s/lists"}