{"id":27294967,"url":"https://github.com/deliciousboy/llm-chatbot-backend","last_synced_at":"2025-08-03T20:35:29.446Z","repository":{"id":286371687,"uuid":"960813803","full_name":"DeliciousBoy/llm-chatbot-backend","owner":"DeliciousBoy","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-14T07:00:58.000Z","size":1384,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-14T08:21:43.108Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DeliciousBoy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-05T05:53:35.000Z","updated_at":"2025-05-14T07:01:02.000Z","dependencies_parsed_at":"2025-04-11T22:58:01.419Z","dependency_job_id":"73319d80-a10c-47b8-8f23-b13289f477d0","html_url":"https://github.com/DeliciousBoy/llm-chatbot-backend","commit_stats":null,"previous_names":["deliciousboy/llm-chatbot-backend"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeliciousBoy%2Fllm-chatbot-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeliciousBoy%2Fllm-chatbot-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeliciousBoy%2Fllm-chatbot-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeliciousBoy%2Fllm-chatbot-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DeliciousBoy","download_url":"https://codeload.github.com/DeliciousBoy/llm-chatbot-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeliciousBoy%2Fllm-chatbot-backend/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259053538,"owners_count":22798438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-11T22:57:59.827Z","updated_at":"2025-06-10T10:08:48.256Z","avatar_url":"https://github.com/DeliciousBoy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM Chatbot\n\n[![Powered by Kedro](https://img.shields.io/badge/powered_by-kedro-ffc900?logo=kedro)](https://kedro.org)\n\n## Overview\n\nA Retrieval-Augmented Generation (RAG) system for scraping website data, embedding text, and answering questions via LLM\n\n\n## How to install dependencies\n\nDeclare any dependencies in `requirements.txt` and `pyproject.toml` for `pip` installation.\n\n### clone the repository\n```bash\ngit clone https://github.com/DeliciousBoy/llm-chatbot-backend.git\ncd llm-chatbot-backend\n```\n\n### Installing `uv`\nthis project uses `uv` to manage virtual environments and dependencies for different Python versions. You can install `uv` run:\n\n```bash\ncurl -Ls https://astral.sh/uv/install.sh | sh\n```\nOr follow the instructions from the official GitHub repository: https://github.com/astral-sh/uv\nOnce installed, you can set up the environment with:\n\n\n### Install with `uv` (Recommended) `This project requires Python 3.11.11`\n```bash\nuv venv\nsource .venv/bin/activate # Or .venv/Scripts/activate for Windows\nuv pip install -r requirements.txt\nuv pip install -e .[dev, docs]\n```\nIf you prefer not to use uv, you can fall back to pip (see below).\n\n### Install with `pip` (Not recommended)\nThis is not recommended as it may lead to dependency conflicts, especially if you are using different Python versions.\n```bash\npython -m venv .venv\nsource .venv/bin/activate # Or .venv/Scripts/activate for Windows\npip install -r requirements.txt\npip install -e .[dev,docs]\n```\n\n## How to run Kedro pipeline\nThis project uses [Kedro](https://kedro.org) to organize data workflows into modular pipelines.\n\n### Avaliable pipelines\n\n\n| Pipeline Name      | Description                          |\n|--------------------|--------------------------------------|\n| `data_processing`  | Cleans and embeds text data into vectors |\n| `web_scraping`     | Asynchronously scrapes web content and stores it as raw data |\n\nEach pipeline is defined in `src/llm_chatbot_backend/pipelines/` and can be run individually or as a group. You can also run specific nodes within a pipeline.\n\n```bash\nkedro run # Run all pipelines\nkedro run --pipeline=web_scraping # Run web scraping pipeline\nkedro run --pipeline=data_processing # Run data processing pipeline\n```\n\n## Visualize Kedro pipeline\nYou can visualize the pipeline using Kedro's built-in visualization tool. This will generate a graph of the pipeline nodes and their dependencies.\n\n```bash\n kedro viz run --autoreload\n```\n## Running Scheduled Jobs\n\nThis project includes a scheduler using `APScheduler` to automate periodic tasks such as scraping data, generating embeddings, or updating indexes.\n\nTo start the scheduler, run:\n\n```bash\npython scheduler.py\n```\n\n## How to test your Kedro project\nthis project uses `pytest` to run test cases. You can run your tests with:\n\n```bash\npytest\n```\n\n## How to run chat interface\nThis project includes a Streamlit app for interacting with the chatbot. You can run the app with:\n\n```\nstreamlit run main.py\n```\nTo run the app locally, make sure the virtual environment is activated and dependencies are installed\n\n## Proejct Structure\nThis project follows the [Kedro](https://kedro.org) project layout with additional components for web scraping, vector embeddings, and an LLM chatbot interface via Streamlit.\n```\n📁llm-chatbot-backend/\n├── 📁conf/ # Kedro configuration files\n│ └── 📁base/\n│   └──📄catalog.yml # Dataset definitions (inputs/outputs for pipelines)\n│   └──📄parameters.yml # Project-level parameters for nodes/pipelines\n├── 📁data/ # raw/cleaned/embedded/chromadb\n├── 📁src/ # Source code (Kedro pipelines, modules)\n│ └── 📁llm_chatbot_backend/\n│   └── 📁datasets/ # Custom Kedro dataset classes\n│   |   └── 📄utf8_json.py # Custom JSON\n│   └── 📁pipelines/ # All Kedro pipelines\n│       └── 📁data_processing/\n│       |  └──📄nodes.py  # Data cleaning / embedding logic\n│       |  └──📄pipeline.py # Defines the data_processing pipeline\n│       └── 📁web_scraping/\n│          └──📄nodes.py # Async scraping logic\n│          └──📄pipeline.py # Defines the web_scraping pipeline\n├── 📁tests/ # Pytest test cases\n│   └── 📁pipelines/\n│       └── 📁data_processing/\n│       |   └──📄test_pipeline.py\n│       └── 📁web_scraping/\n|           └──📄test_pipeline.py\n├──📄main.py # Streamlit chat interface\\\n├──📄scheduler.py # Automate Web Scraping Task\n├──📄pyproject.toml # Project config \u0026 dependencies\n├──📄requirements.txt # Pip requirements\n├──📄uv.lock # uv dependency lockfile\n└──📄.env # Environment variables\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliciousboy%2Fllm-chatbot-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeliciousboy%2Fllm-chatbot-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeliciousboy%2Fllm-chatbot-backend/lists"}