{"id":26011272,"url":"https://github.com/moodlehq/wiki-rag","last_synced_at":"2025-08-03T06:38:29.816Z","repository":{"id":280667487,"uuid":"942761779","full_name":"moodlehq/wiki-rag","owner":"moodlehq","description":"An experimental Retrieval-Augmented Generation (RAG) system specialised in ingesting MediaWiki sites via their API and providing an OpenAI API interface to interact with them.","archived":false,"fork":false,"pushed_at":"2025-07-22T08:18:04.000Z","size":212,"stargazers_count":13,"open_issues_count":3,"forks_count":3,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-07-22T10:32:21.981Z","etag":null,"topics":["experiment","mediawiki","milvus","moodle","rag","research"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/moodlehq.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-04T16:24:42.000Z","updated_at":"2025-07-22T08:18:39.000Z","dependencies_parsed_at":"2025-03-04T17:37:44.605Z","dependency_job_id":"2985d3f3-5e35-406a-bd4a-a172170d88db","html_url":"https://github.com/moodlehq/wiki-rag","commit_stats":null,"previous_names":["moodlehq/wiki-rag"],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/moodlehq/wiki-rag","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moodlehq%2Fwiki-rag","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moodlehq%2Fwiki-rag/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moodlehq%2Fwiki-rag/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moodlehq%2Fwiki-rag/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/moodlehq","download_url":"https://codeload.github.com/moodlehq/wiki-rag/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/moodlehq%2Fwiki-rag/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268505752,"owners_count":24260962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-03T02:00:12.545Z","response_time":2577,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["experiment","mediawiki","milvus","moodle","rag","research"],"created_at":"2025-03-05T23:17:50.882Z","updated_at":"2025-08-03T06:38:29.807Z","avatar_url":"https://github.com/moodlehq.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Wiki-RAG\n\n**Wiki-RAG (Mediawiki Retrieval-Augmented Generation)** is a project that leverages Mediawiki as a source for augmenting text generation tasks, enhancing the quality and relevance of generated content.\n\n## Summary\n\nWiki-RAG integrates Mediawiki's structured knowledge (ingested via [API](https://www.mediawiki.org/wiki/API:Main_page)) with language models to improve text generation. By using retrieval-augmented generation and some interesting techniques, it provides more contextually relevant and accurate responses using any Mediawiki site as KB.\n\nLabeled as an experimental project, Wiki-RAG is part of the [Moodle Research](https://moodle.org/course/view.php?id=17254) initiative, aiming to explore and develop innovative solutions for the educational technology sector.\n\n## Requirements\n\nTo get started with Wiki-RAG, ensure you have the following:\n\n- Git\n- Python 3.12 or later with pip (Python package installer)\n- [Docker](https://www.docker.com/get-started) (if you intend to run the project using Docker)\n- [Milvus 2.5.5](https://milvus.io/docs/release_notes.md#v255) or later (for vector similarity search). Standalone or Distributed deployments are supported. Lite deployments are not supported. It's highly recommended to use the [Docker Compose deployment](https://milvus.io/docs/install_standalone-docker-compose.md) specially for testing and development purposes.\n\n## Configuration\n\n1. **Set Environment Variables (to be replaced soon by `config.yml`file)**\n   - Using the [env_template](env_template) file as source, create a new `.env` file:\n     ```bash\n     cp env_template .env\n     ```\n   - Edit it to suit your needs (mediawiki site and namespaces or exclusions, models to use for both embeddings and generation, etc.)\n\n## Installation\n\n### Running Locally\n\n1. **Clone the Repository**:\n   ```bash\n   git clone https://github.com/moodlehq/wiki-rag.git\n   cd wiki-rag\n   ```\n\n2. **Set Up Virtual Environment** (optional but recommended):\n   ```bash\n   python -m venv .venv\n   source .venv/bin/activate  # On Windows use `venv\\Scripts\\activate`\n   ```\n\n3. **Install Dependencies**:\n   ```bash\n   pip install -e .\n   ```\n   If interested into [contributing](CONTRIBUTING.md) to the project, you can install the development dependencies and enable all the checks by running:\n   ```bash\n   pip install -e .[dev]  # To install all the development dependencies.\n   pre-commit install     # To enable all the check (style, lint, commits, etc.)\n   ```\n\n\n4. **Run the Application**:\n   The application comes with four different executables:\n   - `wr-load`: Will parse all the configured pages in the source Mediawiki site, extracting contents and other important metadata. All the generated information will be stored into a JSON file in the `data` directory.\n   - `wr-index`: In charge of creating the collection in the vector index (Milvus) with all the information extracted in the previous step.\n   - `wr-search`: A tiny CLI utility to perform searches against the RAG system from the command line.\n   - `wr-server`: A comprehensive and secure web service (documented with OpenAPI) that allows users to interact with the RAG system using the OpenAI API (`v1/models` and `v1/chat/completions` endpoints) as if it were a large language model (LLM).\n   - `wr-mcp`: A complete and **UNPROTECTED** built-in MCP server that allows you to access to various parts of Wiki-RAG like prompts (system and use prompt with placeholders), resources (access to the source parsed documents) and tools (retrieve, optimise and generate) using the [MCP Protocol](https://modelcontextprotocol.io/).\n\n### Running with Docker (Milvus elsewhere)\n\n1. Pull the image from GitHub [Container Registry](https://github.com/moodlehq/wiki-rag/pkgs/container/wiki-rag):\n   ```bash\n   docker pull ghcr.io/moodlehq/wiki-rag:latest  # or specify a tag\n   ```\n\n2. Run the container:\n   - **Note 1:** Don't forget to have the `.env` file in the same directory as the command below.\n   - **Note 2:** The `data` directory will be created in the current directory, and it will store all the data generated by the `wr-load` command. If the `LOADER_DUMP_PATH` is set, you will have to change the volume mapping accordingly.\n   ```bash\n   docker run --rm --detach \\\n       --volume $(pwd)/data:/app/data \\\n       --volume $(pwd)/.env:/app/.env \\\n       --env MILVUS_URL=http://milvus-standalone:19530 \\\n       --network milvus \\\n       --publish 8080:8080 \\\n       --env OPENAI_API_KEY=YOUR_OPENAI_API_KEY \\\n       --env LOG_LEVEL=info \\\n       --name wiki-rag \\\n       wiki-rag:latest\n   ```\n   - **Note 3:** The command above will start the `wr-server` automatically, listening on the configured port (8080) and the `OPENAI_API_KEY` is required to interact with the embedding and LLM models. If, instead, you want to execute any of the other commands (`wr-load`, `wr-index`, `wr-search`), you can specify it as the last argument.\n   - **Note 4:** The 2 lines related to Milvus are required to connect to the Milvus server **if also running in Docker**. If it's running elsewhere, you can replace the `MILVUS_URL` with the appropriate URL or configure it in the `.env` file instead\n   - **Note 5:** You can use `docker logs wiki-rag` to check the logs of the running container (`wr-server` logs).\n   - **Note 6:** When running the `wr-server`, you still can execute any of the commands (`wr-load`, `wr-index`, `wr-search`) using `docker exec -it wiki-rag \u003ccommand\u003e`.\n   - **Note 7:** To stop and remove the container, you can use `docker stop wiki-rag`.\n\n### Running with Docker Compose (all-in-one)\n\n1. Download the Milvus Docker Compose file:\n   ```bash\n   wget https://github.com/milvus-io/milvus/releases/download/v2.5.5/milvus-standalone-docker-compose.yml -O milvus-standalone.yml\n   ```\n   OR\n   ```bash\n   curl https://github.com/milvus-io/milvus/releases/download/v2.5.5/milvus-standalone-docker-compose.yml -o milvus-standalone.yml\n   ```\n\n2. Run Wiki-RAG own Docker Compose file:\n   - **Note 1:** Don't forget to have the `.env` file in the same directory as the command below.\n   - **Note 2:** The `data` directory will be created in the current directory, and it will store all the data generated by the `wr-load` command. If the `LOADER_DUMP_PATH` is set, you will have to change the volume mapping accordingly.\n   - **Note 3:** The `volumes` directory will be created in the current directory, ant it will store all the data required by the Milvus containers.\n   ```bash\n   docker compose up -d\n   ```\n   - **Note 4:** Some useful commands include:\n     - To stop all the containers, you can use `docker compose stop`.\n     - To start again all the containers, you can use `docker compose start`.\n     - To stop and remove all the containers, you can use `docker compose down`.\n   - **Note 5:** You can use `docker logs wiki-rag` to check the logs of the running container (`wr-server` logs).\n    - **Note 6:** When running, you still can execute any of the commands (`wr-load`, `wr-index`, `wr-search`) using `docker exec -it wiki-rag \u003ccommand\u003e`.\n\n## Features\n\n   * Accepts virtually any Mediawiki site as a knowledge base (KB).\n   * Uses Mediawiki API to extract content and metadata (not web crawling).\n   * Supports multiple namespaces and page exclusions.\n   * The whole KB is stored in a (dated) JSON file for later use, access and analysis.\n   * Metadata-rich information containing parent-ship relations, sources and other useful information.\n   * Loads the KB into a vector database (Milvus) for fast retrieval.\n   * Hybrid retrieval using both vector and keyword search with fusion reranking.\n   * Contextual-aware query rewrite for a better chat experience.\n   * Support for chit-chatting cases, avoiding out-of-scope questions (limited but effective).\n   * Performs post-retrieval optimizations to improve the quality of the retrieved documents:\n       * POP (popularity): In charge of boosting the most popular chunks.\n       * POC (parent, own and child): Add those related chunks to the retrieved ones.\n   * Many configuration options:\n       * Models to use for embeddings, query rewrite and answers generation (any OpenAI compatible provider).\n       * Size (in messages count and tokens length) to use as history.\n       * Other settings (embed dimensions, chunk size, etc.)\n   * Supports (opt-in) observability (tracing) with both LangSmith and Langfuse.\n   * Prompts management (also opt-in) with LangSmith and Langfuse. Defaults apply otherwise.\n   * Uses LangGraph for orchestration of the whole LLM pipeline.\n   * Exposed as a model using standard OpenAI API endpoints (`v1/models` and `v1/chat/completions`). Optionally protected with local or remote bearer tokens. Supports streaming responses.\n   * Exposed as a (unprotected!) MCP server (Model Context Protocol) with all the required endpoints (prompts, resources and tools).\n   * Plenty of ideas for future work and improvements (see next section).\n\n## Future Work\n\n   support partial/incremental loading /\n   support partial/incremental indexing /\n   process multiple mediawiki sites /\n   process other sites (docusaurus, ...) /\n   more granular splits /\n   loaders to become plugins /\n   [use markdown as \"backend format\"](https://github.com/moodlehq/wiki-rag/issues/5) /\n   ~~[include/exclude lists](https://github.com/moodlehq/wiki-rag/commit/454d2bc244dfd933589239413589c73437f0f4d9)~~ /\n   compare results with different embeddings /\n   fine tune the embedding /\n   support adding a watermark text to the responses /\n   consider adding summaries, multiple levels and more metadata /\n   [able to use other vector databases](https://github.com/moodlehq/wiki-rag/issues/3) /\n   add other algos, apart from current POP and POC /\n   support multi-level searches /\n   ~~[context-aware rag](https://github.com/moodlehq/wiki-rag/commit/77cac73469da7c07710851be51321b13e1c923fc)~~ /\n   extra querying techniques /\n   prepare an evaluation collection /\n   apply thresholds to results /\n   better Open AI implementation, supporting more parameters /\n   ~~[better prompt management](https://github.com/moodlehq/wiki-rag/commit/2d6127f308fa7da21908f94818108cde59e39076)~~ /\n   ~~[observability alternatives (langfuse, ...)](https://github.com/moodlehq/wiki-rag/commit/2f1be65900255d837bb8d4bf913a4a557cd2f404)~~ /\n   context-size control /\n   add guardrails at different levels /\n   [move configuration away from env file](https://github.com/moodlehq/wiki-rag/issues/2) /\n   better fit in multi-agentic environments /\n   detect out of scope questions better /\n   semantic routing /\n   ~~[support MCP (model context protocol)](https://github.com/moodlehq/wiki-rag/issues/4)~~ /\n   make everything pluggable /\n   add unit and integration tests /\n   automate checks and releases\n\n## Documentation\n\nFor more detailed information, please refer to the following files:\n\n- [License](LICENSE)\n- [Contributing Guidelines](CONTRIBUTING.md)\n- [Code of Conduct](CODE_OF_CONDUCT.md)\n\n## License\n\nThis project is licensed under the BSD 3-Clause License. See the [LICENSE](LICENSE) file for more information.\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for more details.\n\n## Code of Conduct\n\nPlease note that this project adheres to a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you agree to uphold this code.\n\n----\n© 2025 Moodle Research Team\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoodlehq%2Fwiki-rag","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmoodlehq%2Fwiki-rag","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmoodlehq%2Fwiki-rag/lists"}