{"id":14064824,"url":"https://github.com/bitswired/website-to-knowledge-base","last_synced_at":"2026-03-17T23:50:04.483Z","repository":{"id":160946690,"uuid":"630150304","full_name":"bitswired/website-to-knowledge-base","owner":"bitswired","description":null,"archived":false,"fork":false,"pushed_at":"2023-05-03T10:43:13.000Z","size":115,"stargazers_count":66,"open_issues_count":1,"forks_count":27,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-08-13T07:07:36.208Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bitswired.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-19T19:18:39.000Z","updated_at":"2024-08-13T07:07:37.692Z","dependencies_parsed_at":null,"dependency_job_id":"94c6b7b2-4902-487a-aeb2-381c3b82da1b","html_url":"https://github.com/bitswired/website-to-knowledge-base","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitswired%2Fwebsite-to-knowledge-base","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitswired%2Fwebsite-to-knowledge-base/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitswired%2Fwebsite-to-knowledge-base/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bitswired%2Fwebsite-to-knowledge-base/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bitswired","download_url":"https://codeload.github.com/bitswired/website-to-knowledge-base/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228040863,"owners_count":17860211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-13T07:04:06.203Z","updated_at":"2026-03-17T23:50:04.405Z","avatar_url":"https://github.com/bitswired.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# AI-Powered Knowledge Base\n\n[Demo](https://user-images.githubusercontent.com/19983429/235894528-9ad791a8-f2f0-4ad8-a2fc-69f4ed0aa8d8.mp4)\n\n[YouTube Demo](https://youtu.be/qRsNQweVKj0)\n\nThis repository contains an AI-powered knowledge base that utilizes the LLMs model to answer questions based on a given website's content and **provide sources as links** to the relevant pages. \n\nThe system:\n1. Loads the website's content using a **sitemap**\n2. Split each web page into **chunks**\n4. **Embed** each chunk using a LLM (for now OpenAI) and store them in the **Chroma vector database\n5. Then it embeds the user query and run a similarity search using the Chroma database\n5. Finally it loads the similarity search results as context for a LLM (for now ChatGPT) to find relevant answers and citing the sources\n\nIt also provides a Streamlit-based web interface for an easy-to-use experience.\n\n## Files\n\n- `knowledge_base.py`: The main module that creates the KnowledgeBase class. This class is responsible for loading and processing the website content, creating the document index, and querying the LLM model for answers.\n- `app.py`: A Streamlit web application that provides a user interface for querying the AI-powered knowledge base.\n\n## Installation\n\n1. Clone the repository:\n\n```\ngit clone git@github.com:bitswired/website-to-knowledge-base.git\n```\n\n2. Instal the project with poetry:\n\n```\npoetry install\n```\n\n## Usage\n\n### Knowledge Base\n\nTo use the KnowledgeBase class, follow these steps:\n\n1. Import the KnowledgeBase class:\n\n```python\nfrom knowledge_base import KnowledgeBase\n```\n\n2. Instantiate the KnowledgeBase with the appropriate sitemap URL and pattern (optional):\n\n```python\nkb = KnowledgeBase(\n    sitemap_url=\"https://nextjs.org/sitemap.xml\",\n    pattern=\"docs/api-refe\",\n    chunk_size=8000,\n    chunk_overlap=3000,\n)\n```\n\n3. Ask a question:\n\n```python\nresult = kb.ask(\"How do I deploy my Next.js app?\")\nprint(result)\n```\n\n### Web Application\n\nTo run the Streamlit web application, execute the following command in your terminal:\n\n```\nstreamlit run app.py\n```\n\nThe web app will open in your default browser. Enter the URL to the website's sitemap, an optional filter pattern for the URLs, and your question. The AI-powered knowledge base will return an answer based on the content of the website.\n\n## Requirements\n\n- An API key for OpenAI's GPT-4 (see [OpenAI's API documentation](https://beta.openai.com/docs/) for details)\n\n## License\n\nThis project is licensed under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitswired%2Fwebsite-to-knowledge-base","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitswired%2Fwebsite-to-knowledge-base","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitswired%2Fwebsite-to-knowledge-base/lists"}