{"id":46549445,"url":"https://github.com/datascienceuibk/rankarena","last_synced_at":"2026-03-07T03:03:29.934Z","repository":{"id":318986269,"uuid":"1076883776","full_name":"DataScienceUIBK/RankArena","owner":"DataScienceUIBK","description":"RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback — CIKM ’25, Seoul, Nov 10–14, 2025.","archived":false,"fork":false,"pushed_at":"2025-11-26T12:06:19.000Z","size":74,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-11-29T10:13:56.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataScienceUIBK.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-15T13:29:17.000Z","updated_at":"2025-11-26T12:06:23.000Z","dependencies_parsed_at":"2025-10-17T15:51:27.237Z","dependency_job_id":"ad597893-413d-4a1e-8911-696cde54b686","html_url":"https://github.com/DataScienceUIBK/RankArena","commit_stats":null,"previous_names":["datascienceuibk/rankarena"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataScienceUIBK/RankArena","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataScienceUIBK%2FRankArena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataScienceUIBK%2FRankArena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataScienceUIBK%2FRankArena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataScienceUIBK%2FRankArena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataScienceUIBK","download_url":"https://codeload.github.com/DataScienceUIBK/RankArena/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataScienceUIBK%2FRankArena/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30206339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"online","status_checked_at":"2026-03-07T02:00:06.765Z","response_time":53,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-03-07T03:03:15.590Z","updated_at":"2026-03-07T03:03:29.925Z","avatar_url":"https://github.com/DataScienceUIBK.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🔍 RankArena\n\nA unified platform for evaluating **retrieval**, **reranking**, and **RAG** with **human** and **LLM-as-a-judge** feedback.\n\n\u003e **Paper:** *RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback* — CIKM ’25, Seoul, Nov 10–14, 2025.\n\n- **Live demo:** `https://rankarena.ngrok.io/`\n- **Demo video:** `https://youtu.be/jIYAP4PaSSI`\n- **Contact:** abdelrahman.abdallah@uibk.ac.at\n\n\n\n## 🎉News\n- **[2025-08-01]** Our paper accepted in 🎉 CIKM 2025 🎉.\n\n\n\n## ✨ Features\n\n- **Retriever + Reranker Arena** (offline BM25/DPR/BGE/ColBERT/Contriever on Wiki/MSMARCO; optional online search)\n- **RAG Arena**: dual reranking → dual LLM answers (Together AI) → side-by-side comparison\n- **LLM Judge**: automatic reranking comparisons saved alongside user votes\n- **Leaderboard \u0026 Analytics**: ELO-style ranking + plots (Plotly)\n- **Annotation Tab**: manual relevance ranking, movement metrics \u0026 persisted logs\n\n---\n\n## 🔐 Secrets \u0026 Environment\n\nWe no longer hard-code API keys. Configure them via environment variables (optionally through a `.env` file).\n\nCreate a `.env` file in the repo root (do **not** commit it):\n\n```\nSERPER_API_KEY=your-serper-key-here\nTOGETHER_API_KEY=your-together-key-here\n```\n\n\u003e **What uses these keys?**\n\u003e\n\u003e - `SERPER_API_KEY` — required **only** for the **Online Retriever** (web search via Serper). Offline retrievers do not need it.\n\u003e - `TOGETHER_API_KEY` — required for **RAG generation** (answers from Together’s Llama 3.3 70B Instruct). The UI can still retrieve/rerank without it; only answer generation needs it.\n\nAlso add to `.gitignore`:\n```\n.env\n```\n\nBoth keys are loaded at runtime using `os.getenv(...)` (and `python-dotenv` if installed). The app shows a friendly error if a required key is missing when you choose a feature that needs it.\n\n---\n\n## 🧰 Requirements\n\n- Python 3.9+\n- Recommended: create and activate a virtualenv\n- Install deps:\n  ```bash\n  pip install -r requirements.txt\n  ```\n  If you don’t have a `requirements.txt`, ensure at least:\n  ```bash\n  pip install gradio python-dotenv together plotly pandas numpy scipy scikit-learn\n  ```\n  Plus the project’s own packages/modules (e.g., `rankify`).\n\n---\n\n## 🚀 Quick Start\n\n1) **Set keys** (optional for fully offline usage):\n   - Put them in `.env` (see above) or export as env vars:\n     ```bash\n     export SERPER_API_KEY=...\n     export TOGETHER_API_KEY=...\n     ```\n\n2) **Run the app**:\n   ```bash\n   python app.py\n   ```\n   The app serves on **http://0.0.0.0:7860** by default.\n\n3) **Use it**:\n   - **Offline Retriever**: choose method + corpus, no keys needed.\n   - **Online Retriever**: requires `SERPER_API_KEY`.\n   - **RAG Arena**: requires `TOGETHER_API_KEY` for answer generation.\n\n---\n\n## 🧭 Tabs Overview\n\n- **💬 RAG Arena**: Retrieve → Dual Rerank → Generate answers with Together AI → Compare \u0026 Vote  \n- **🛠️ Retriever + Reranker**: Retrieve, then run two rerankers; compare rankings \u0026 vote  \n- **⚡ Direct Reranker / Anonymous Reranker**: Quick reranking flows (no user metadata)  \n- **📚 BEIR**: Benchmark-oriented workflows  \n- **✏️ Annotation**: Manually rank retrieved contexts, save movement metrics  \n- **📊 Leaderboard**: Combined ELO + BEIR/DL stats + rich analytics  \n- **ℹ️ Information**: Help texts / about\n\n---\n\n## ⚙️ Configuration Notes\n\n- **Online Retriever**: guarded by a key check. If `SERPER_API_KEY` is missing, you’ll get a clear UI error instead of a crash.\n- **Together AI**: the RAG step calls:\n  ```\n  meta-llama/Llama-3.3-70B-Instruct-Turbo-Free\n  ```\n  via the `together` SDK. If `TOGETHER_API_KEY` is unset, RAG answer generation will show an informative error.\n\n- **Optional per-session key input**: You can (optionally) re-enable the Together-key textbox in the RAG tab and pass it through; env var fallback still works.\n\n\n---\n\n## 🧩 Troubleshooting\n\n- **“Missing SERPER_API_KEY”** when using Online Retriever  \n  Set `SERPER_API_KEY` in your environment or `.env`. Use Offline Retriever if you don’t need web search.\n\n- **“Please provide a valid Together AI API key” / “Together AI library not installed”** when running RAG  \n  Set `TOGETHER_API_KEY` and install the SDK:\n  ```bash\n  pip install together\n  ```\n\n- **Plotly/Sklearn not found**  \n  Some analytics are optional; install:\n  ```bash\n  pip install plotly scikit-learn\n  ```\n\n- **Port or host change**  \n  Update `demo.launch(server_name=\"0.0.0.0\", server_port=7860, share=False)` in `app.py`.\n\n---\n\n## 🗂️ Data \u0026 Logs\n\n- User interactions and evaluations are saved under `user_data/\u003csession_id\u003e/...`:\n  - `votes/`, `votellm/`, `annotate/`, etc.\n- Leaderboard reads from `leaderboard/result.csv` (if present) and augments with arena stats.\n\n---\n\n\n## :star2: Citation\n\nPlease kindly cite our paper if helps your research:\n\n```BibTex\n@article{abdallah2025rankarena,\n  title={Rankarena: A unified platform for evaluating retrieval, reranking and rag with human and llm feedback},\n  author={Abdallah, Abdelrahman and Abdalla, Mahmoud and Piryani, Bhawna and Mozafari, Jamshid and Ali, Mohammed and Jatowt, Adam},\n  journal={arXiv preprint arXiv:2508.05512},\n  year={2025}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatascienceuibk%2Frankarena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatascienceuibk%2Frankarena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatascienceuibk%2Frankarena/lists"}