{"id":25180471,"url":"https://github.com/avnlp/rankers","last_synced_at":"2025-06-22T07:05:50.097Z","repository":{"id":275772497,"uuid":"926686919","full_name":"avnlp/rankers","owner":"avnlp","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-19T16:28:37.000Z","size":725,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-04-04T04:46:51.342Z","etag":null,"topics":["listwise","llm-rankers","pairwise","pointwise","ranker","setwise"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/avnlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-03T17:33:48.000Z","updated_at":"2025-02-19T16:28:41.000Z","dependencies_parsed_at":"2025-02-04T14:38:08.457Z","dependency_job_id":"191ee212-70f5-4032-a954-3ca54be28c14","html_url":"https://github.com/avnlp/rankers","commit_stats":null,"previous_names":["avnlp/rankers"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/avnlp/rankers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avnlp%2Frankers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avnlp%2Frankers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avnlp%2Frankers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avnlp%2Frankers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/avnlp","download_url":"https://codeload.github.com/avnlp/rankers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/avnlp%2Frankers/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261250276,"owners_count":23130541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["listwise","llm-rankers","pairwise","pointwise","ranker","setwise"],"created_at":"2025-02-09T16:18:24.855Z","updated_at":"2025-06-22T07:05:45.075Z","avatar_url":"https://github.com/avnlp.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Rankers\n\n**Paper:** [LLM Rankers](paper/rankers.pdf)\n\n- Haystack components were created for the Listwise, Setwise, Pairwise techniques.\n- Also, Haystack components were created for the RankZephyr and RankVicuna (RankLLM).\n- The pipelines for setwise, pairwise, listwise and pointwise rankers were created on the FIQA, Sci-Fact, NF-Corpus, TREC-19, and TREC-20 datasets.\n- The Dense retrieval pipelines were created using the Mistral, Llama-3, and Phi-3 models.\n- The Dense retrieval pipeline using the LLama-3 model and setwise ranking gave the best performance.\n- Each dense retrieval pipeline was created for the Mistral, Phi-3 and LLama-3 models and evaluated on the NDCG metric.\n\n## Pointwise Ranking\n- In the pointwise ranking method, the reranker takes both the query and a candidate document to directly generate a relevance score. These independent scores assigned to each document are then used to reorder the from the set of documents.\n- LLMs are asked to generate whether the candidate document provided is relevant to the query, with the process repeated for each candidate document.\n## Pairwise Ranking\n- The Pairwise Ranking Prompting, a pair of candidate items along with the user query serve as prompts to guide the LLMs to determine which document is the most relevant to the given query.\n- Pairs are then independently fed into the LLM, and the preferred document is determined for each pair. Subsequently, an aggregation function is employed to assign a score to each document based on the inferred pairwise preferences, and the final ranking is established based on the total score assigned to each document.\n## Listwise Ranking\n- The Listwise Reranker with a LLM takes the query and a list of documents as input and returns a reordered list of the input document identifiers.\n- Current listwise approaches use a sliding window method. This involves re-ranking a window of candidate documents, starting from the bottom of the original ranking list and progressing upwards.\n## Setwise Ranking\n- The Setwise prompting approach instructs LLMs to select the most relevant document to the query from a set of candidate documents.\n- The Setwise prompting technique improves the efficiency of Pairwise prompting (PRP) by comparing multiple documents at each step, as opposed to just a pair. The Setwise prompting approach instructs LLMs to select the most relevant document to the query from a set of candidate documents.\n\n## Results\n- The RankLlama and RankZephyr rankers performed best on the FIQA, TREC-19, SciFact and NFCorpus datasets.\n- The setwise ranker with the heapsort method using the LLama-3 model gave the best performance for the FIQA, TREC-19, SciFact, and NFcorpus datasets.\n- The RankLlama and RankZephyr models with the pointwise and listwise ranking gave the best output.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favnlp%2Frankers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Favnlp%2Frankers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Favnlp%2Frankers/lists"}