{"id":17183576,"url":"https://github.com/nicholasgriffintn/llm-assistant","last_synced_at":"2026-04-10T15:51:31.788Z","repository":{"id":250451933,"uuid":"834513842","full_name":"nicholasgriffintn/llm-assistant","owner":"nicholasgriffintn","description":"This is a side project to build an assistant using llms.","archived":false,"fork":false,"pushed_at":"2024-12-12T23:50:10.000Z","size":200,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T04:39:11.710Z","etag":null,"topics":["cloudflare","cloudflare-ai","fastapi","langchain","llava","llm","mistral","mistral-nano","ollama"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nicholasgriffintn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-27T13:52:59.000Z","updated_at":"2024-12-12T23:50:14.000Z","dependencies_parsed_at":"2024-08-03T01:09:46.400Z","dependency_job_id":"ef1cc961-2964-4b02-b52e-302b77f1c4da","html_url":"https://github.com/nicholasgriffintn/llm-assistant","commit_stats":null,"previous_names":["nicholasgriffintn/llama-assistant","nicholasgriffintn/llm-assistant"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasgriffintn%2Fllm-assistant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasgriffintn%2Fllm-assistant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasgriffintn%2Fllm-assistant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nicholasgriffintn%2Fllm-assistant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nicholasgriffintn","download_url":"https://codeload.github.com/nicholasgriffintn/llm-assistant/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245394773,"owners_count":20608123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare","cloudflare-ai","fastapi","langchain","llava","llm","mistral","mistral-nano","ollama"],"created_at":"2024-10-15T00:40:58.640Z","updated_at":"2025-12-30T23:26:36.147Z","avatar_url":"https://github.com/nicholasgriffintn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## LLM Assistant\n\n\u003e Update: This funcationality has now moved to my new assistant project which is running in TypeScript and includes a range of features, not just article analysis and questions. https://github.com/nicholasgriffintn/assistant\n\n## Introduction\nThe goal of this project is to develop an advanced assistant using the Large Language Model (LLM) architecture, specifically leveraging the capabilities of the Mistral model.\n\n## Features\n\n- A frontend interface for users to interact with the assistant, manage the database and view past queries and responses.\n- A backend that leverages the Mixtral model and Ollama/ Cloudflare AI to generate responses to user queries.\n- [ ] A database to store user queries and responses for future reference. (maybe in the future).\n\n## Technologies\n\n- Python\n- FastAPI\n- Langchain\n- Mistral (mistral-nemo and mistral-7b-instruct)\n- LLava\n- Ollama\n- Cloudflare\n\n## Setup\n\n```bash\n# Clone the repository\ngit clone https://github.com/nicholasgriffin/llm-assistant.git\n# Navigate to the project directory\ncd llm-assistant\n# Create a virtual environment\npython3 -m venv venv\n# Activate the virtual environment\nsource venv/bin/activate\n# Install the required dependencies\npip install -r requirements.txt\n# Run the FastAPI server\nuvicorn app.main:app --reload --port 8080\n```\n\nThe app will be accessible at `http://localhost:8080`.\n\n## Setting up Ollama\n\nIf you haven't already, you will need to download and set up Ollama. You can find the instructions [here](https://github.com/ollama/ollama/blob/main/README.md#quickstart).\n\nAfter that, you will need to run the Mixtral model using the following command:\n\n```bash\nollama pull mistral-nemo\n```\n\nThis model is around 7GB, so make sure you have enough space on your machine, also note that this download will take some time.\n\nYou can find out more about the model [here](https://ollama.com/library/mixtral-nemo).\n\nYou can change the model being used by setting the `MODEL_NAME` environment variable.\n\n## Image to Text Generation\n\nIf an image (or multiple) is found in the provided query, the assistant will attempt to describe that image, this is configured to use the `llava` model by default, but you can change this by setting the `IMAGE_MODEL_NAME` environment variable.\n\nYou'll need to also pull the `llava` model using the following command in order for this to work:\n\n```bash\nollama pull llava\n```\n\nYou can find out more about the model [here](https://ollama.com/library/llava).\n\n## Using Cloudflare\n\nIf you'd like to use Cloudflare's AI service to generate responses instead, you can set the `USE_CLOUDFLARE` environment variable to `true`.\n\nYou'll also need to set the `CLOUDFLARE_API_TOKEN` environment variable to your Cloudflare API token and the `CLOUDFLARE_ACCOUNT_ID` environment variable to your Cloudflare account ID.\n\nYou can find out more about Cloudflare's AI service [here](https://developers.cloudflare.com/ai/).\n\n### Cloudflare Models\n\nOn Cloudflare, I have been using `@cf/mistral/mistral-7b-instruct-v0.1` as a replacement for the `mistral-nemo` model and `@cf/llava-hf/llava-1.5-7b-hf` as a replacement for the `llava` model.\n\n### Cloudflare AI Gateway\n\nThe Cloudflare AI Gateway is pretty cool, it allows you to track usage of CloudFlare AI, set rate limits for requests and, possibly the coolest thing, cache responses for a certain amount of time.\n\nYou can find out more about the Cloudflare AI Gateway [here](https://developers.cloudflare.com/ai-gateway/).\n\nTo use it, set the env variable `CLOUDFLARE_GATEWAY_ID` to the ID of your Cloudflare AI Gateway.\n\n## Prompt Engineering\n\nThe AI functions use a number of prompts to generate responses. These prompts are stored in the `prompts` folder as text files. You can add, remove or edit these prompts to change the behavior of the AI.\n\nIf you'd like to learn more about prompt engineering, you can read [this documentation](https://www.promptingguide.ai/models/mixtral) which talks about how to format prompts for mixtral (which we are using) and other models.\n\n## Testing\n\n```bash\n# Run the test suite\npytest -v\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasgriffintn%2Fllm-assistant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnicholasgriffintn%2Fllm-assistant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnicholasgriffintn%2Fllm-assistant/lists"}