{"id":18400799,"url":"https://github.com/jankovicsandras/bm25opt","last_synced_at":"2025-04-14T11:32:16.406Z","repository":{"id":259410486,"uuid":"877783186","full_name":"jankovicsandras/bm25opt","owner":"jankovicsandras","description":"faster BM25 search algorithms in Python","archived":false,"fork":false,"pushed_at":"2024-11-14T13:26:45.000Z","size":71,"stargazers_count":19,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T00:44:14.644Z","etag":null,"topics":["bag-of-words","bm25","bm25l","bm25okapi","bm25plus","document-search","python3","text-search"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jankovicsandras.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-10-24T08:30:32.000Z","updated_at":"2025-03-16T00:15:03.000Z","dependencies_parsed_at":"2024-10-25T06:26:12.127Z","dependency_job_id":"92dc6744-7e49-40c6-99ba-0092ff7a89fd","html_url":"https://github.com/jankovicsandras/bm25opt","commit_stats":null,"previous_names":["jankovicsandras/bm25opt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jankovicsandras%2Fbm25opt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jankovicsandras%2Fbm25opt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jankovicsandras%2Fbm25opt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jankovicsandras%2Fbm25opt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jankovicsandras","download_url":"https://codeload.github.com/jankovicsandras/bm25opt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248871922,"owners_count":21175319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bag-of-words","bm25","bm25l","bm25okapi","bm25plus","document-search","python3","text-search"],"created_at":"2024-11-06T02:36:51.831Z","updated_at":"2025-04-14T11:32:16.381Z","avatar_url":"https://github.com/jankovicsandras.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# BM25opt\n## faster BM25 search algorithms in Python\n####  based on https://github.com/dorianbrown/rank_bm25 by Dorian Brown\n####  Apache License, Version 2.0 http://www.apache.org/licenses/LICENSE-2.0\n----\n## News:\n - 1.1.0 supports updating the index with new ```add_documents()```, ```delete_documents()``` and ```update_documents()``` functions, see Example 4\n\n----\n## Usage:\n#### Input:\n - ```corpus``` is a list of strings, e.g. ```[ 'bla bla bla', 'this is document two', ... ]```\n - ```question``` is a string, e.g. ```'which text contains the word two?'```\n - optional arguments:\n   - ```algo``` : BM25 algorithm, the default is ```'okapi'```; ```'l'``` and ```'plus'``` available\n   - ```tokenizer_function``` : the default is ```tokenizer_default``` which is split-on-whitespace, lowercase, remove common punctiations\n   - ```idf_algo``` : default uses the same IDF as ```rank_bm25```; values ```'okapi'```, ```'l'``` and ```'plus'``` can override to fix https://github.com/dorianbrown/rank_bm25/issues/35\n   - ```k1```, ```b```, ```epsilon```, ```delta``` : constants with standard default values, see https://en.wikipedia.org/wiki/Okapi_BM25\n#### Example 1:\nThis example uses the default tokenizer and the default BM25Okapi algorithm and returns the top 5 highest scoring document ids and scores.\n```python\n# creating the index\nbm25opt_index = BM25opt( corpus )\n\n# search\nresults = bm25opt_index.topk( question, 5 )\nprint( 'results[0] id', results[0][0], 'results[0] score', results[0][1], 'results[0] document', corpus[ results[0][0] ] )\n```\n#### Example 2:\nThis example returns the list of document scores (order is the same as the document order in corpus), shows algoritm selection and custom tokenizer function.\n```python\nbm25opt_index = BM25opt( corpus, algo='plus', tokenizer_function=some_tokenizer_function )\ndoc_scores = bm25opt_index.get_scores( question )\n```\n#### Example 3: comparison with rank_bm25\nThis example shows the score list and the similarity with [```rank_bm25```](https://github.com/dorianbrown/rank_bm25), but NOTE: BM25opt input is not tokenized beforehand.\n```python\ncorpus = [ ... ]\nquestion = '...'\ntokenized_corpus = [ tokenizer_default(document) for document in corpus ]\ntokenized_question = tokenizer_default( question )\n\nrank_bm25_index = BM25Okapi( tokenized_corpus )\nbm25opt_index = BM25opt( corpus, algo='okapi' )\n\nrank_bm25_scores = rank_bm25_index.get_scores( tokenizedquestion )\nbm25opt_scores = bm25opt_index.get_scores( question )\n```\n\n#### Example 4: updating the index\n```python\n# creating the index\nbm25opt_index = BM25opt( corpus )\n\n# add new documents\nbm25opt_index.add_documents( corpus2 ) \n\n# delete from the index\ndelete_ids = [ 1, 3, 5 ] # list of document ids (indices in corpus) to delete from the index\nbm25opt_index.delete_documents( delete_ids )\n\n# in-place update changed documents in the index\nupdate_ids = [ 1, 3, 5 ] # list of document ids (indices in corpus) to change\nupdated_documents = [ 'first changed document', 'second changed document', ... ]\nbm25opt_index.update_documents( update_ids, updated_documents )\n\n```\n----\n### Notes:\nThis is an optimized variant of rank_bm25 where the key insight is that we can calculate almost everything at index creation time in ```__init__()``` , resulting a words * documents-score dict, e.g.\n```python\nwsmap = {\n  'word1': [ word1_doc1_score, word1_doc2_score, ... ],\n  'word2': [ word2_doc1_score, word2_doc2_score, ... ],\n  ...\n}\n```\nthen the query function is just adding the score lists for each word in the question, e.g. \n```python\nquestion = 'word1 word2'\ndoc_scores = [ wsmap['word1'][0] + wsmap['word2'][0], wsmap['word1'][1] + wsmap['word2'][1], ... ]\n```\nAnother important change is the un-tokenized inputs and registration of the tokenizer function, which is important to avoid situations where the corpus would be tokenized with a different function than the queries later. A simple ```tokenizer_default()``` function is provided as a default.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjankovicsandras%2Fbm25opt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjankovicsandras%2Fbm25opt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjankovicsandras%2Fbm25opt/lists"}