{"id":16277533,"url":"https://github.com/c0sogi/llmchat","last_synced_at":"2025-04-07T06:07:26.047Z","repository":{"id":146237247,"uuid":"608255933","full_name":"c0sogi/LLMChat","owner":"c0sogi","description":"A full-stack Webui implementation of Large Language model, such as ChatGPT or LLaMA.","archived":false,"fork":false,"pushed_at":"2024-07-25T11:33:19.000Z","size":61154,"stargazers_count":258,"open_issues_count":3,"forks_count":45,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-11T18:55:24.165Z","etag":null,"topics":["chatbot","chatgpt","docker","docker-compose","fastapi","flutter","fullstack","langchain","llm","mysql","openai","python","redis","restapi","sqlalchemy","websocket","webui"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c0sogi.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-01T16:34:28.000Z","updated_at":"2024-10-10T21:49:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"c1cf77d5-a77b-4b62-9511-03e55cf0132e","html_url":"https://github.com/c0sogi/LLMChat","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0sogi%2FLLMChat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0sogi%2FLLMChat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0sogi%2FLLMChat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c0sogi%2FLLMChat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c0sogi","download_url":"https://codeload.github.com/c0sogi/LLMChat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247601447,"owners_count":20964864,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatbot","chatgpt","docker","docker-compose","fastapi","flutter","fullstack","langchain","llm","mysql","openai","python","redis","restapi","sqlalchemy","websocket","webui"],"created_at":"2024-10-10T18:55:18.934Z","updated_at":"2025-04-07T06:07:26.027Z","avatar_url":"https://github.com/c0sogi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLMChat 🎉\n\n👋 Welcome to the LLMChat repository, a full-stack implementation of an API server built with Python FastAPI, and a beautiful frontend powered by Flutter. \n💬 This project is designed to deliver a seamless chat experience with the advanced ChatGPT and other LLM models.\n🔝 Offering a modern infrastructure that can be easily extended when GPT-4's Multimodal and Plugin features become available. \n🚀 Enjoy your stay!\n\n## **Demo**\n---\n### **Enjoy the beautiful UI and rich set of customizable widgets provided by Flutter.**\n- It supports both `mobile` and `PC` environments. \n- `Markdown` is also supported, so you can use it to format your messages.\n\n---\n### Web Browsing\n+ #### **Duckduckgo**\n    You can use the Duckduckgo search engine to find relevant information on the web. Just activate the 'Browse' toggle button!\n    \n    Watch the demo video for full-browsing: https://www.youtube.com/watch?v=mj_CVrWrS08\n        \n\u003e ![Browse Web](app/contents/browsing_demo.png)\n\n---\n### Vector Embedding\n+ #### **Embed Any Text**\n    With the `/embed` command, you can store the text indefinitely in your own private vector database and query it later, anytime. If you use the `/share` command, the text is stored in a public vector database that everyone can share. Enabling `Query` toggle button or `/query` command helps the AI generate contextualized answers by searching for text similarities in the public and private databases. This solves one of the biggest limitations of language models: **memory**. \n\n+ #### **Upload Your PDF File**\n    You can embed PDF file by clicking `Embed Document` on the bottom left. In a few seconds, text contents of PDF will be converted to vectors and embedded to Redis cache.\n\u003e ![Upload Your PDF File](app/contents/upload_demo.gif)\n    \n\n---\n+ ### Change your chat model\n    You can change your chat model by dropdown menu. You can define whatever model you want to use in `LLMModels` which is located in `app/models/llms.py`. \n    \u003e ![Change your chat model](app/contents/model_selection_demo.png)\n---\n+ ### Change your chat title\n    You can change your chat title by clicking the title of the chat. This will be stored until you change or delete it!\n    \u003e ![Change your chat title](app/contents/edit_title_demo.png)\n---\n### 🦙 Local LLMs\n\n\u003e ![llama api](app/contents/llama_api.png)\n\nFor the local Llalam LLMs, it is assumed to work only in the local environment and uses the `http://localhost:8002/v1/completions` endpoint.  It continuously checks the status of the llama API server by connecting to `http://localhost:8002/health` once a second to see if a 200 OK response is returned, and if not, it automatically runs a separate process to create a the API server.\n\n#### **Llama.cpp**\n\nThe main goal of llama.cpp is to run the LLaMA model using `GGML` 4-bit quantization with plain C/C++ implementation without dependencies. You have to download GGML `bin` file from huggingface and put it in the `llama_models/ggml` folder, and define LLMModel in `app/models/llms.py`. There are few examples, so you can easily define your own model.\nRefer to the `llama.cpp` repository for more information: https://github.com/ggerganov/llama.cpp\n\n#### **Exllama**\n\nA standalone Python/C++/CUDA implementation of Llama for use with 4-bit `GPTQ` weights, designed to be fast and memory-efficient on modern GPUs. It uses `pytorch` and `sentencepiece` to run the model. It is assumed to work only in the local environment and at least one `NVIDIA CUDA GPU` is required. You have to download tokenizer, config, and GPTQ files from huggingface and put it in the `llama_models/gptq/YOUR_MODEL_FOLDER` folder, and define LLMModel in `app/models/llms.py`. There are few examples, so you can easily define your own model. Refer to the `exllama` repository for more detailed information: https://github.com/turboderp/exllama\n\n---\n\n## Key Features\n- **FastAPI** - High-performance `web framework` for building APIs with Python.\n- **Flutter** - `Webapp` frontend with beautiful UI and rich set of customizable widgets.\n- **ChatGPT** - Seamless integration with the `OpenAI API` for text generation and message management.\n- **LLAMA** - Suporting LocalLLM, `LlamaCpp` and `Exllama` models.\n- **WebSocket Connection** - `Real-time`, two-way communication with the ChatGPT, and other LLM models, with Flutter frontend webapp.\n- **Vectorstore** - Using `Redis` and `Langchain`, store and retrieve vector embeddings for similarity search. It will help AI to generate more relevant responses.\n- **Auto summarization** - Using Langchain's summarize chain, summarize the conversation and store it in the database. It will help saving a lot of tokens.\n- **Web Browsing** - Using `Duckduckgo` search engine, browse the web and find relevant information.\n- **Concurrency** - Asynchronous programming with `async`/`await` syntax for concurrency and parallelism.\n- **Security** - Token validation and authentication to keep API secure.\n- **Database** - Manage database connections and execute `MySQL` queries. Easily perform Create, Read, Update, and Delete actions, with `sqlalchemy.asyncio`\n- **Cache** - Manage cache connections and execute `Redis` queries with aioredis. Easily perform Create, Read, Update, and Delete actions, with `aioredis`.\n\n\n## Getting Started / Installation\n\nTo set up the on your local machine, follow these simple steps.\nBefore you begin, ensure you have `docker` and `docker-compose` installed on your machine. If you want to run the server without docker, you have to install `Python 3.11` additionally. Even though, you need `Docker` to run DB servers.\n\n### **1. Clone the repository**\n\nTo recursively clone the submodules to use `Exllama` or `llama.cpp` models, use the following command:\n```bash\ngit clone --recurse-submodules https://github.com/c0sogi/llmchat.git\n```\n\nYou only want to use core features(OpenAI), use the following command:\n```bash\ngit clone https://github.com/c0sogi/llmchat.git\n```\n\n### **2. Change to the project directory**\n\n```bash\ncd LLMChat\n```\n\n### **3. Create `.env` file**\nSetup an env file, referring to `.env-sample` file. Enter database information to create, OpenAI API Key, and other necessary configurations. Optionals are not required, just leave them as they are.\n\n\n### **4. To run the server**\nExecute these. It may take a few minutes to start the server for the first time:\n\n```bash\ndocker-compose -f docker-compose-local.yaml up\n```\n\n### **5. To stop the server**\n\n```bash\ndocker-compose -f docker-compose-local.yaml down\n```\n\n### **6. Enjoy it**\nNow you can access the server at `http://localhost:8000/docs` and the database at `db:3306` or `cache:6379`. You can also access the app at `http://localhost:8000/chat`.\n\n\n- **To run the server without docker**\nIf you want to run the server without docker, you have to install `Python 3.11` additionally. Even though, you need `Docker` to run DB servers. Turn off the API server already running with `docker-compose -f docker-compose-local.yaml down api`. Don't forget to run other DB servers on Docker! Then, run the following commands:\n\n    ```bash\n    python -m main\n    ```\n\n    Your Server should now be up and running on `http://localhost:8001` in this case.\n# License\n\nThis project is licensed under the [MIT License](LICENSE), which allows for free use, modification, and distribution, as long as the original copyright and license notice are included in any copy or substantial portion of the software.\n\n\n# Why FastAPI?\n\n🚀 `FastAPI` is a modern web framework for building APIs with Python. \n💪 It has high performance, easy to learn, fast to code, and ready for production. \n👍 One of the main features of `FastAPI` is that it supports concurrency and `async`/`await` syntax. \n🤝 This means that you can write code that can handle multiple tasks at the same time without blocking each other, especially when dealing with I/O bound operations, such as network requests, database queries, file operations, etc.\n\n\n# Why Flutter?\n\n📱 `Flutter` is an open-source UI toolkit developed by Google for building native user interfaces for mobile, web, and desktop platforms from a single codebase. \n👨‍💻 It uses `Dart`, a modern object-oriented programming language, and provides a rich set of customizable widgets that can adapt to any design.\n\n\n# WebSocket Connection\n\nYou can access `ChatGPT` or `LlamaCpp` through `WebSocket` connection using two modules: `app/routers/websocket` and `app/utils/chat/chat_stream_manager`. These modules facilitate the communication between the `Flutter` client and the Chat model through a WebSocket. With the WebSocket, you can establish a real-time, two-way communication channel to interact with the LLM.\n\n## Usage\n\nTo start a conversation, connect to the WebSocket route `/ws/chat/{api_key}` with a valid API key registered in the database. Note that this API key is not the same as OpenAI API key, but only available for your server to validate the user. Once connected, you can send messages and commands to interact with the LLM model. The WebSocket will send back chat responses in real-time. This websocket connection is established via Flutter app, which can accessed with endpoint `/chat`.\n\n## websocket.py\n\n`websocket.py` is responsible for setting up a WebSocket connection and handling user authentication. It defines the WebSocket route `/chat/{api_key}` that accepts a WebSocket and an API key as parameters.\n\nWhen a client connects to the WebSocket, it first checks the API key to authenticate the user. If the API key is valid, the `begin_chat()` function is called from the `stream_manager.py` module to start the conversation.\n\nIn case of an unregistered API key or an unexpected error, an appropriate message is sent to the client and the connection is closed.\n\n```python\n@router.websocket(\"/chat/{api_key}\")\nasync def ws_chat(websocket: WebSocket, api_key: str):\n    ...\n```\n\n## stream_manager.py\n\n`stream_manager.py` is responsible for managing the conversation and handling user messages. It defines the `begin_chat()` function, which takes a WebSocket, a user ID as parameters.\n\nThe function first initializes the user's chat context from the cache manager. Then, it sends the initial message history to the client through the WebSocket.\n\nThe conversation continues in a loop until the connection is closed. During the conversation, the user's messages are processed and GPT's responses are generated accordingly.\n\n```python\nclass ChatStreamManager:\n    @classmethod\n    async def begin_chat(cls, websocket: WebSocket, user: Users) -\u003e None:\n    ...\n```\n\n\n### Sending Messages to WebSocket\n\nThe `SendToWebsocket` class is used for sending messages and streams to the WebSocket. It has two methods: `message()` and `stream()`. The `message()` method sends a complete message to the WebSocket, while the `stream()` method sends a stream to the WebSocket.\n\n```python\nclass SendToWebsocket:\n    @staticmethod\n    async def message(...):\n        ...\n\n    @staticmethod\n    async def stream(...):\n        ...\n```\n\n### Handling AI Responses\n\nThe `MessageHandler` class also handles AI responses. The `ai()` method sends the AI response to the WebSocket. If translation is enabled, the response is translated using the Google Translate API before sending it to the client.\n\n```python\nclass MessageHandler:\n    ...\n    @staticmethod\n    async def ai(...):\n        ...\n```\n\n### Handling Custom Commands\n\nUser messages are processed using the `HandleMessage` class. If the message starts with `/`, such as `/YOUR_CALLBACK_NAME`. it is treated as a command and the appropriate command response is generated. Otherwise, the user's message is processed and sent to the LLM model for generating a response.\n\n\nCommands are handled using the `ChatCommands` class. It executes the corresponding callback function depending on the command. You can add new commands by simply adding callback in `ChatCommands` class from `app.utils.chat.chat_commands`.\n\n\n# 🌟Vector Embedding\n Using Redis for storing vector embeddings of conversations 🗨️ can aid the ChatGPT model 🤖 in several ways, such as efficient and fast retrieval of conversation context 🕵️‍♀️, handling large amounts of data 📊, and providing more relevant responses through vector similarity search 🔎.\n\nSome fun examples of how this could work in practice:\n- Imagine a user is chatting with ChatGPT about their favorite TV show 📺 and mentions a specific character 👤. Using Redis, ChatGPT could retrieve previous conversations where that character was mentioned and use that information to provide more detailed insights or trivia about that character 🤔.\n- Another scenario could be a user discussing their travel plans ✈️ with ChatGPT. If they mention a particular city 🌆 or landmark 🏰, ChatGPT could use vector similarity search to retrieve previous conversations that discussed the same location and provide recommendations or tips based on that context 🧳. \n- If a user mentions a particular cuisine 🍝 or dish 🍱, ChatGPT could retrieve previous conversations that discussed those topics and provide recommendations or suggestions based on that context 🍴.\n\n\n### 1. Embedding text using the `/embed` command\n\nWhen a user enters a command in the chat window like `/embed \u003ctext_to_embed\u003e`, the `VectorStoreManager.create_documents` method is called. This method converts the input text into a vector using OpenAI's `text-embedding-ada-002` model and stores it in the Redis vectorstore.\n\n```python\n@staticmethod\n@command_response.send_message_and_stop\nasync def embed(text_to_embed: str, /, buffer: BufferedUserContext) -\u003e str:\n    \"\"\"Embed the text and save its vectors in the redis vectorstore.\\n\n    /embed \u003ctext_to_embed\u003e\"\"\"\n    ...\n```\n\n### 2. Querying embedded data using the `/query` command\n\nWhen the user enters the `/query \u003cquery\u003e` command, the `asimilarity_search` function is used to find up to three results with the highest vector similarity to the embedded data in the Redis vectorstore. These results are temporarily stored in the context of the chat, which helps AI answer the query by referring to these data.\n\n```python\n@staticmethod\nasync def query(query: str, /, buffer: BufferedUserContext, **kwargs) -\u003e Tuple[str | None, ResponseType]:\n    \"\"\"Query from redis vectorstore\\n\n    /query \u003cquery\u003e\"\"\"\n    ...\n```\n\n### 3. Automatically embedding uploaded text files\n\nWhen running the `begin_chat` function, if a user uploads a file containing text (e.g., a PDF or TXT file), the text is automatically extracted from the file, and its vector embedding is saved to Redis.\n\n```python\n@classmethod\nasync def embed_file_to_vectorstore(cls, file: bytes, filename: str, collection_name: str) -\u003e str:\n    # if user uploads file, embed it\n    ...\n```\n\n### 4. `commands.py` functionality\n\nIn the `commands.py` file, there are several important components:\n\n- `command_response`: This class is used to set a decorator on the command method to specify the next action. It helps to define various response types, such as sending a message and stopping, sending a message and continuing, handling user input, handling AI responses, and more.\n- `command_handler`: This function is responsible for performing a command callback method based on the text entered by the user.\n- `arguments_provider`: This function automatically supplies the arguments required by the command method based on the annotation type of the command method.\n\n\n# 📝 Auto Summarization\n### There is a way to save tokens by adding a task to the LLM that summarizes the message. The auto summarization task is a crucial feature that enhances the efficiency of chatbot. Let's break down the functionality of this feature:\n\n\n1. **Task Triggering**: This feature is activated whenever a user types a message or the AI responds with a message. At this point, an automatic summarization task is generated to condense the text content.\n\n2. **Task Storage**: The auto-summarization task is then stored in the `task_list` attribute of the `BufferUserChatContext`. This serves as a queue for managing tasks linked to the user's chat context.\n\n3. **Task Harvesting**: Following the completion of a user-AI question and answer cycle by the `MessageHandler`, the `harvest_done_tasks` function is invoked. This function collects the results of the summarization task, making sure nothing is left out.\n\n4. **Summarization Application**: After the harvesting process, the summarized results replace the actual message when our chatbot is requesting answers from language learning models (LLMs), such as OPENAI and LLAMA_CPP. By doing so, we're able to send much more succinct prompts than the initial lengthy message.\n\n5. **User Experience**: Importantly, from the user's perspective, they only see the original message. The summarized version of the message is not shown to them, maintaining transparency and avoiding potential confusion.\n\n6. **Simultaneous Tasks**: Another key feature of this auto-summarization task is that it doesn't impede other tasks. In other words, while the chatbot is busy summarizing the text, other tasks can still be carried out, thereby improving the overall efficiency of our chatbot.\n\n### By default, summarize chain only works for messages of 512 tokens or more. This can be turned on/off and the threshold set in `ChatConfig`.\n\n\n\n# 📚 LLM Models\n\nThis repository contains different LLM models, defined in `llms.py`. Each LLM Model class inherit from the base class `LLMModel`. The `LLMModels` enum is a collection of these LLMs.\n\nAll operations are handled asynchronously without interupting the main thread. However, Local LLMs are not be able to handle multiple requests at the same time, as they are too computationally expensive. Therefore, a `Semaphore` is used to limit the number of requests to 1. \n\n\n## 📌 Usage\n\nThe default LLM model used by the user via `UserChatContext.construct_default` is `gpt-3.5-turbo`. You can change the default for that function.\n\n## 📖 Model Descriptions\n\n### 1️⃣ OpenAIModel\n\n`OpenAIModel` generates text asynchronously by requesting chat completion from the OpenAI server. It requires an OpenAI API key.\n\n### 2️⃣ LlamaCppModel\n\n`LlamaCppModel` reads a locally stored GGML model. The LLama.cpp GGML model must be put in the `llama_models/ggml` folder as a `.bin` file. For example, if you downloaded a q4_0 quantized model from \"https://huggingface.co/TheBloke/robin-7B-v2-GGML\",\nThe path of the model has to be \"robin-7b.ggmlv3.q4_0.bin\".\n\n### 3️⃣ ExllamaModel\n\n`ExllamaModel` read a locally stored GPTQ model. The Exllama GPTQ model must be put in the `llama_models/gptq` folder as a folder. For example, if you downloaded 3 files from \"https://huggingface.co/TheBloke/orca_mini_7B-GPTQ/tree/main\":\n\n- orca-mini-7b-GPTQ-4bit-128g.no-act.order.safetensors\n- tokenizer.model\n- config.json\n\nThen you need to put them in a folder.\nThe path of the model has to be the folder name. Let's say, \"orca_mini_7b\", which contains the 3 files.\n \n## 📝 Handling Exceptions\nHandle exceptions that may occur during text generation. If a `ChatLengthException` is thrown, it automatically performs a routine to re-limit the message to within the number of tokens limited by the `cutoff_message_histories` function, and resend it. This ensures that the user has a smooth chat experience regardless of the token limit.\n\n# Behind the WebSocket Connection...\n\nThis project aims to create an API backend to enable the large language model chatbot service. It utilizes a cache manager to store messages and user profiles in Redis, and a message manager to safely cache messages so that the number of tokens does not exceed an acceptable limit.\n\n## Cache Manager\n\nThe Cache Manager (`CacheManager`) is responsible for handling user context information and message histories. It stores these data in Redis, allowing for easy retrieval and modification. The manager provides several methods to interact with the cache, such as:\n\n- `read_context_from_profile`: Reads the user's chat context from Redis, according to the user's profile.\n- `create_context`: Creates a new user chat context in Redis.\n- `reset_context`: Resets the user's chat context to default values.\n- `update_message_histories`: Updates the message histories for a specific role (user, ai, or system).\n- `lpop_message_history` / `rpop_message_history`: Removes and returns the message history from the left or right end of the list.\n- `append_message_history`: Appends a message history to the end of the list.\n- `get_message_history`: Retrieves the message history for a specific role.\n- `delete_message_history`: Deletes the message history for a specific role.\n- `set_message_history`: Sets a specific message history for a role and index.\n\n## Message Manager\n\nThe Message Manager (`MessageManager`) ensures that the number of tokens in message histories does not exceed the specified limit. It safely handles adding, removing, and setting message histories in the user's chat context while maintaining token limits. The manager provides several methods to interact with message histories, such as:\n\n- `add_message_history_safely`: Adds a message history to the user's chat context, ensuring that the token limit is not exceeded.\n- `pop_message_history_safely`: Removes and returns the message history from the right end of the list while updating the token count.\n- `set_message_history_safely`: Sets a specific message history in the user's chat context, updating the token count and ensuring that the token limit is not exceeded.\n\n## Usage\n\nTo use the cache manager and message manager in your project, import them as follows:\n\n```python\nfrom app.utils.chat.managers.cache import CacheManager\nfrom app.utils.chat.message_manager import MessageManager\n```\n\nThen, you can use their methods to interact with the Redis cache and manage message histories according to your requirements.\n\nFor example, to create a new user chat context:\n\n```python\nuser_id = \"example@user.com\"  # email format\nchat_room_id = \"example_chat_room_id\"  # usually the 32 characters from `uuid.uuid4().hex`\ndefault_context = UserChatContext.construct_default(user_id=user_id, chat_room_id=chat_room_id)\nawait CacheManager.create_context(user_chat_context=default_context)\n```\n\nTo safely add a message history to the user's chat context:\n\n```python\nuser_chat_context = await CacheManager.read_context_from_profile(user_chat_profile=UserChatProfile(user_id=user_id, chat_room_id=chat_room_id))\ncontent = \"This is a sample message.\"\nrole = ChatRoles.USER  # can be enum such as ChatRoles.USER, ChatRoles.AI, ChatRoles.SYSTEM\nawait MessageManager.add_message_history_safely(user_chat_context, content, role)\n```\n\n\n# Middlewares\n\nThis project uses `token_validator` middleware and other middlewares used in the FastAPI application. These middlewares are responsible for controlling access to the API, ensuring only authorized and authenticated requests are processed.\n\n## Examples\n\nThe following middlewares are added to the FastAPI application:\n\n1. Access Control Middleware: Ensures that only authorized requests are processed.\n2. CORS Middleware: Allows requests from specific origins, as defined in the app configuration.\n3. Trusted Host Middleware: Ensures that requests are coming from trusted hosts, as defined in the app configuration.\n\n### Access Control Middleware\n\nThe Access Control Middleware is defined in the `token_validator.py` file. It is responsible for validating API keys and JWT tokens.\n\n#### State Manager\n\nThe `StateManager` class is used to initialize request state variables. It sets the request time, start time, IP address, and user token.\n\n#### Access Control\n\nThe `AccessControl` class contains two static methods for validating API keys and JWT tokens:\n\n1. `api_service`: Validates API keys by checking the existence of required query parameters and headers in the request. It calls the `Validator.api_key` method to verify the API key, secret, and timestamp.\n2. `non_api_service`: Validates JWT tokens by checking the existence of the 'authorization' header or 'Authorization' cookie in the request. It calls the `Validator.jwt` method to decode and verify the JWT token.\n\n#### Validator\n\nThe `Validator` class contains two static methods for validating API keys and JWT tokens:\n\n1. `api_key`: Verifies the API access key, hashed secret, and timestamp. Returns a `UserToken` object if the validation is successful.\n2. `jwt`: Decodes and verifies the JWT token. Returns a `UserToken` object if the validation is successful.\n\n#### Access Control Function\n\nThe `access_control` function is an asynchronous function that handles the request and response flow for the middleware. It initializes the request state using the `StateManager` class, determines the type of authentication required for the requested URL (API key or JWT token), and validates the authentication using the `AccessControl` class. If an error occurs during the validation process, an appropriate HTTP exception is raised.\n\n### Token\n\nToken utilities are defined in the `token.py` file. It contains two functions:\n\n1. `create_access_token`: Creates a JWT token with the given data and expiration time.\n2. `token_decode`: Decodes and verifies a JWT token. Raises an exception if the token is expired or cannot be decoded.\n\n### Params Utilities\n\nThe `params_utils.py` file contains a utility function for hashing query parameters and secret key using HMAC and SHA256:\n\n1. `hash_params`: Takes query parameters and secret key as input and returns a base64 encoded hashed string.\n\n### Date Utilities\n\nThe `date_utils.py` file contains the `UTC` class with utility functions for working with dates and timestamps:\n\n1. `now`: Returns the current UTC datetime with an optional hour difference.\n2. `timestamp`: Returns the current UTC timestamp with an optional hour difference.\n3. `timestamp_to_datetime`: Converts a timestamp to a datetime object with an optional hour difference.\n\n### Logger\n\nThe `logger.py` file contains the `ApiLogger` class, which logs API request and response information, including the request URL, method, status code, client information, processing time, and error details (if applicable). The logger function is called at the end of the `access_control` function to log the processed request and response.\n\n## Usage\n\nTo use the `token_validator` middleware in your FastAPI application, simply import the `access_control` function and add it as a middleware to your FastAPI instance:\n\n```python\nfrom app.middlewares.token_validator import access_control\n\napp = FastAPI()\n\napp.add_middleware(dispatch=access_control, middleware_class=BaseHTTPMiddleware)\n```\n\nMake sure to also add the CORS and Trusted Host middlewares for complete access control:\n\n```python\napp.add_middleware(\n    CORSMiddleware,\n    allow_origins=config.allowed_sites,\n    allow_credentials=True,\n    allow_methods=[\"*\"],\n    allow_headers=[\"*\"],\n)\n\napp.add_middleware(\n    TrustedHostMiddleware,\n    allowed_hosts=config.trusted_hosts,\n    except_path=[\"/health\"],\n)\n```\n\nNow, any incoming requests to your FastAPI application will be processed by the `token_validator` middleware and other middlewares, ensuring that only authorized and authenticated requests are processed.\n\n# Database Connection\n\nThis module `app.database.connection` provides an easy-to-use interface for managing database connections and executing SQL queries using SQLAlchemy and Redis. It supports MySQL, and can be easily integrated with this project.\n\n## Features\n\n- Create and drop databases\n- Create and manage users\n- Grant privileges to users\n- Execute raw SQL queries\n- Manage database sessions with async support\n- Redis caching support for faster data access\n\n\n## Usage\n\nFirst, import the required classes from the module:\n\n```python\nfrom app.database.connection import MySQL, SQLAlchemy, CacheFactory\n```\n\nNext, create an instance of the `SQLAlchemy` class and configure it with your database settings:\n\n```python\nfrom app.common.config import Config\n\nconfig: Config = Config.get()\ndb = SQLAlchemy()\ndb.start(config)\n```\n\nNow you can use the `db` instance to execute SQL queries and manage sessions:\n\n```python\n# Execute a raw SQL query\nresult = await db.execute(\"SELECT * FROM users\")\n\n# Use the run_in_session decorator to manage sessions\n@db.run_in_session\nasync def create_user(session, username, password):\n    await session.execute(\"INSERT INTO users (username, password) VALUES (:username, :password)\", {\"username\": username, \"password\": password})\n\nawait create_user(\"JohnDoe\", \"password123\")\n```\n\nTo use Redis caching, create an instance of the `CacheFactory` class and configure it with your Redis settings:\n\n```python\ncache = CacheFactory()\ncache.start(config)\n```\n\nYou can now use the `cache` instance to interact with Redis:\n\n```python\n# Set a key in Redis\nawait cache.redis.set(\"my_key\", \"my_value\")\n\n# Get a key from Redis\nvalue = await cache.redis.get(\"my_key\")\n```\nIn fact, in this project, the `MySQL` class does the initial setup at app startup, and all database connections are made with only the `db` and `cache` variables present at the end of the module. 😅 \n\nAll db settings will be done in `create_app()` in `app.common.app_settings`.\nFor example, the `create_app()` function in `app.common.app_settings` will look like this:\n\n```python\ndef create_app(config: Config) -\u003e FastAPI:\n    # Initialize app \u0026 db \u0026 js\n    new_app = FastAPI(\n        title=config.app_title,\n        description=config.app_description,\n        version=config.app_version,\n    )\n    db.start(config=config)\n    cache.start(config=config)\n    js_url_initializer(js_location=\"app/web/main.dart.js\")\n    # Register routers\n    # ...\n    return new_app\n```\n\n# Database CRUD Operations\n\nThis project uses simple and efficient way to handle database CRUD (Create, Read, Update, Delete) operations using SQLAlchemy and two module and path: `app.database.models.schema` and `app.database.crud`. \n\n## Overview\n\n### app.database.models.schema\n\nThe `schema.py` module is responsible for defining database models and their relationships using SQLAlchemy. It includes a set of classes that inherit from `Base`, an instance of `declarative_base()`. Each class represents a table in the database, and its attributes represent columns in the table. These classes also inherit from a `Mixin` class, which provides some common methods and attributes for all the models.\n\n#### Mixin Class\n\nThe Mixin class provides some common attributes and methods for all the classes that inherit from it. Some of the attributes include:\n\n- `id`: Integer primary key for the table.\n- `created_at`: Datetime for when the record was created.\n- `updated_at`: Datetime for when the record was last updated.\n- `ip_address`: IP address of the client that created or updated the record.\n\nIt also provides several class methods that perform CRUD operations using SQLAlchemy, such as:\n\n- `add_all()`: Adds multiple records to the database.\n- `add_one()`: Adds a single record to the database.\n- `update_where()`: Updates records in the database based on a filter.\n- `fetchall_filtered_by()`: Fetches all records from the database that match the provided filter.\n- `one_filtered_by()`: Fetches a single record from the database that matches the provided filter.\n- `first_filtered_by()`: Fetches the first record from the database that matches the provided filter.\n- `one_or_none_filtered_by()`: Fetches a single record or returns `None` if no records match the provided filter.\n\n### app.database.crud\nThe `users.py` and `api_keys.py` module contains a set of functions that perform CRUD operations using the classes defined in `schema.py`. These functions use the class methods provided by the Mixin class to interact with the database.\n\nSome of the functions in this module include:\n\n- `create_api_key()`: Creates a new API key for a user.\n- `get_api_keys()`: Retrieves all API keys for a user.\n- `get_api_key_owner()`: Retrieves the owner of an API key.\n- `get_api_key_and_owner()`: Retrieves an API key and its owner.\n- `update_api_key()`: Updates an API key.\n- `delete_api_key()`: Deletes an API key.\n- `is_email_exist()`: Checks if an email exists in the database.\n- `get_me()`: Retrieves user information based on user ID.\n- `is_valid_api_key()`: Checks if an API key is valid.\n- `register_new_user()`: Registers a new user in the database.\n- `find_matched_user()`: Finds a user with a matching email in the database.\n\n## Usage\n\nTo use the provided CRUD operations, import the relevant functions from the `crud.py` module and call them with the required parameters. For example:\n\n```python\nimport asyncio\nfrom app.database.crud.users import register_new_user, get_me, is_email_exist\nfrom app.database.crud.api_keys import create_api_key, get_api_keys, update_api_key, delete_api_key\n\nasync def main():\n    # `user_id` is an integer index in the MySQL database, and `email` is user's actual name\n    # the email will be used as `user_id` in chat. Don't confuse with `user_id` in MySQL\n\n    # Register a new user\n    new_user = await register_new_user(email=\"test@test.com\", hashed_password=\"...\")\n\n    # Get user information\n    user = await get_me(user_id=1)\n\n    # Check if an email exists in the database\n    email_exists = await is_email_exist(email=\"test@test.com\")\n\n    # Create a new API key for user with ID 1\n    new_api_key = await create_api_key(user_id=1, additional_key_info={\"user_memo\": \"Test API Key\"})\n\n    # Get all API keys for user with ID 1\n    api_keys = await get_api_keys(user_id=1)\n\n    # Update the first API key in the list\n    updated_api_key = await update_api_key(updated_key_info={\"user_memo\": \"Updated Test API Key\"}, access_key_id=api_keys[0].id, user_id=1)\n\n    # Delete the first API key in the list\n    await delete_api_key(access_key_id=api_keys[0].id, access_key=api_keys[0].access_key, user_id=1)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc0sogi%2Fllmchat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc0sogi%2Fllmchat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc0sogi%2Fllmchat/lists"}