{"id":16883968,"url":"https://github.com/DevXT-LLC/ezlocalai","last_synced_at":"2025-11-21T13:30:15.890Z","repository":{"id":197825989,"uuid":"699411012","full_name":"DevXT-LLC/ezlocalai","owner":"DevXT-LLC","description":"ezlocalai is an easy to set up local artificial intelligence server with OpenAI Style Endpoints.","archived":false,"fork":false,"pushed_at":"2025-02-14T02:40:46.000Z","size":156640,"stargazers_count":83,"open_issues_count":4,"forks_count":16,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-02-15T20:12:13.769Z","etag":null,"topics":["ai","artificial-intelligence","cuda","llamacpp","local"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DevXT-LLC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["Josh-XT"],"ko_fi":"joshxt"}},"created_at":"2023-10-02T15:22:42.000Z","updated_at":"2025-02-14T02:40:49.000Z","dependencies_parsed_at":"2023-10-05T07:47:06.980Z","dependency_job_id":"aac43e86-de43-4c47-a35d-837bfad0c948","html_url":"https://github.com/DevXT-LLC/ezlocalai","commit_stats":{"total_commits":734,"total_committers":5,"mean_commits":146.8,"dds":"0.016348773841961872","last_synced_commit":"2c267491605ad3c06ece7f839d934fb564dc383c"},"previous_names":["josh-xt/llamacpp-server","agixt/local-llm","agixt/ezlocalai","devxt-llc/ezlocalai","josh-xt/local-llm"],"tags_count":60,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevXT-LLC%2Fezlocalai","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevXT-LLC%2Fezlocalai/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevXT-LLC%2Fezlocalai/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DevXT-LLC%2Fezlocalai/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DevXT-LLC","download_url":"https://codeload.github.com/DevXT-LLC/ezlocalai/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239625941,"owners_count":19670783,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","artificial-intelligence","cuda","llamacpp","local"],"created_at":"2024-10-13T16:15:57.823Z","updated_at":"2025-11-21T13:30:15.823Z","avatar_url":"https://github.com/DevXT-LLC.png","language":"Jupyter Notebook","funding_links":["https://github.com/sponsors/Josh-XT","https://ko-fi.com/joshxt"],"categories":[],"sub_categories":[],"readme":"# ezlocalai\n\n[![GitHub](https://img.shields.io/badge/GitHub-ezLocalai-blue?logo=github\u0026style=plastic)](https://github.com/DevXT-LLC/ezlocalai) [![Dockerhub](https://img.shields.io/badge/Docker-ezlocalai-blue?logo=docker\u0026style=plastic)](https://hub.docker.com/r/joshxt/ezlocalai)\n\nezlocalai is an easy set up artificial intelligence server that allows you to easily run multimodal artificial intelligence from your computer. It is designed to be as easy as possible to get started with running local models. It automatically handles downloading the model of your choice and configuring the server based on your CPU, RAM, and GPU specifications. It also includes [OpenAI Style](https://pypi.org/project/openai/) endpoints for easy integration with other applications using ezlocalai as an OpenAI API proxy with any model. Additional functionality is built in for voice cloning text to speech and a voice to text for easy voice communication as well as image generation entirely offline after the initial setup.\n\n## Prerequisites\n\n- [Git](https://git-scm.com/downloads)\n- [Docker Desktop](https://docs.docker.com/docker-for-windows/install/) (Windows or Mac)\n- [CUDA Toolkit (May Need 12.4)](https://developer.nvidia.com/cuda-12-4-0-download-archive) (NVIDIA GPU only)\n\n\u003cdetails\u003e\n  \u003csummary\u003eAdditional Linux Prerequisites\u003c/summary\u003e\n\n- [Docker](https://docs.docker.com/get-docker/)\n- [Docker Compose](https://docs.docker.com/compose/install/)\n- [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) (NVIDIA GPU only)\n\n\u003c/details\u003e\n\n## Installation\n\n```bash\ngit clone https://github.com/DevXT-LLC/ezlocalai\ncd ezlocalai\n```\n\n### Environment Setup\n\nExpand Environment Setup if you would like to modify the default environment variables, otherwise skip to Usage. All environment variables are optional and have useful defaults. Change the default model that starts with ezlocalai in your `.env` file.\n\n\u003cdetails\u003e\n  \u003csummary\u003eEnvironment Setup (Optional)\u003c/summary\u003e\n\nNone of the values need modified in order to run the server. If you are using an NVIDIA GPU, I would recommend setting the `GPU_LAYERS` and `MAIN_GPU` environment variables. If you have multiple GPUs, especially different ones, you should set `TENSOR_SPLIT` to reflect the desired load balance (comma separated decimals totalling 1). If you plan to expose the server to the internet, I would recommend setting the `EZLOCALAI_API_KEY` environment variable for security. `THREADS` is set to your CPU thread count minus 2 by default, if this causes significant performance issues, consider setting the `THREADS` environment variable manually to a lower number.\n\nModify the `.env` file to your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.\n\nReplace the environment variables with your desired settings. Assumptions will be made on all of these values if you choose to accept the defaults.\n\n- `EZLOCALAI_URL` - The URL to use for the server. Default is `http://localhost:8091`.\n- `EZLOCALAI_API_KEY` - The API key to use for the server. If not set, the server will not require an API key when accepting requests.\n- `NGROK_TOKEN` - The ngrok token to use for the server. If not set, ngrok will not be used. Using ngrok will allow you to expose your ezlocalai server to the public with as simple as an API key. [Get your free NGROK_TOKEN here.](https://dashboard.ngrok.com/get-started/your-authtoken)\n- `DEFAULT_MODEL` - The default model to use when no model is specified. Use the Hugging Face path. Default is `TheBloke/phi-2-dpo-GGUF`.\n- `LLM_MAX_TOKENS` - The maximum number of tokens to use for the language model. If set to `0`, it will automatically use the max tokens for the model. Default is `0`.\n- `WHISPER_MODEL` - The model to use for speech-to-text. Default is `base.en`.\n- `AUTO_UPDATE` - Whether or not to automatically update ezlocalai. Default is `true`.\n- `THREADS` - The number of CPU threads ezlocalai is allowed to use. Default is 4.\n- `GPU_LAYERS` (Only applicable to NVIDIA GPU) - The number of layers to use on the GPU. Default is `0`. Your `GPU_LAYERS` will automatically determine a number of layers to use based on your GPU's memory if it is set to `-1` and you have an NVIDIA GPU. If it is set to `-2`, it will use the maximum number of layers requested by the model.\n- `MAIN_GPU` (Only applicable to NVIDIA GPU) - The GPU to use for the language model. Default is `0`.\n- `TENSOR_SPLIT` (Only applicable with multiple CUDA GPUs) - The allocation to each device in CSV format.\n- `IMG_ENABLED` - If set to true, models will choose to generate images when they want to based on the user input. **This is only available on GPU.** Default is `false`.\n- `SD_MODEL` - The stable diffusion model to use. Default is `stabilityai/sdxl-turbo`.\n- `VISION_MODEL` - The vision model to use. Default is None. Current options are `deepseek-ai/deepseek-vl-1.3b-chat` and `deepseek-ai/deepseek-vl-7b-chat`.\n\n\u003c/details\u003e\n\n## Usage\n\n### NVIDIA GPU\n\n```bash\ndocker-compose -f docker-compose-cuda.yml down\ndocker-compose -f docker-compose-cuda.yml build\ndocker-compose -f docker-compose-cuda.yml up\n```\n\n### CPU\n\n```bash\ndocker-compose down\ndocker-compose build\ndocker-compose up\n```\n\n## OpenAI Style Endpoint Usage\n\nOpenAI Style endpoints available at `http://\u003cYOUR LOCAL IP ADDRESS\u003e:8091/v1/` by default. Documentation can be accessed at that \u003chttp://localhost:8091\u003e when the server is running.\n\nFor examples on how to use the server to communicate with the models, see the [Examples Jupyter Notebook](tests.ipynb) once the server is running. We also have an [example to use in Google Colab](ezlocalai-ngrok.ipynb).\n\n## Demo UI\n\nYou can access the basic demo UI at \u003chttp://localhost:8502\u003e, or your local IP address with port 8502.\n\n## Workflow\n\n```mermaid\ngraph TD\n   A[app.py] --\u003e B[FASTAPI]\n   B --\u003e C[Pipes]\n   C --\u003e D[LLM]\n   C --\u003e E[STT]\n   C --\u003e F[CTTS]\n   C --\u003e G[IMG]\n   D --\u003e H[llama_cpp]\n   D --\u003e I[tiktoken]\n   D --\u003e J[torch]\n   E --\u003e K[faster_whisper]\n   E --\u003e L[pyaudio]\n   E --\u003e M[webrtcvad]\n   E --\u003e N[pydub]\n   F --\u003e O[TTS]\n   F --\u003e P[torchaudio]\n   G --\u003e Q[diffusers]\n   Q --\u003e J\n   A --\u003e R[Uvicorn]\n   R --\u003e S[ASGI Server]\n   A --\u003e T[API Endpoint: /v1/completions]\n   T --\u003e U[Pipes.get_response]\n   U --\u003e V{completion_type}\n   V --\u003e|completion| W[LLM.completion]\n   V --\u003e|chat| X[LLM.chat]\n   X --\u003e Y[LLM.generate]\n   W --\u003e Y\n   Y --\u003e Z[LLM.create_completion]\n   Z --\u003e AA[Return response]\n   AA --\u003e AB{stream}\n   AB --\u003e|True| AC[StreamingResponse]\n   AB --\u003e|False| AD[JSON response]\n   U --\u003e AE[Audio transcription]\n   AE --\u003e AF{audio_format}\n   AF --\u003e|Exists| AG[Transcribe audio]\n   AG --\u003e E\n   AF --\u003e|None| AH[Skip transcription]\n   U --\u003e AI[Audio generation]\n   AI --\u003e AJ{voice}\n   AJ --\u003e|Exists| AK[Generate audio]\n   AK --\u003e F\n   AK --\u003e AL{stream}\n   AL --\u003e|True| AM[StreamingResponse]\n   AL --\u003e|False| AN[JSON response with audio URL]\n   AJ --\u003e|None| AO[Skip audio generation]\n   U --\u003e AP[Image generation]\n   AP --\u003e AQ{IMG enabled}\n   AQ --\u003e|True| AR[Generate image]\n   AR --\u003e G\n   AR --\u003e AS[Append image URL to response]\n   AQ --\u003e|False| AT[Skip image generation]\n   A --\u003e AU[API Endpoint: /v1/chat/completions]\n   AU --\u003e U\n   A --\u003e AV[API Endpoint: /v1/embeddings]\n   AV --\u003e AW[LLM.embedding]\n   AW --\u003e AX[LLM.create_embedding]\n   AX --\u003e AY[Return embedding]\n   A --\u003e AZ[API Endpoint: /v1/audio/transcriptions]\n   AZ --\u003e BA[STT.transcribe_audio]\n   BA --\u003e BB[Return transcription]\n   A --\u003e BC[API Endpoint: /v1/audio/generation]\n   BC --\u003e BD[CTTS.generate]\n   BD --\u003e BE[Return audio URL or base64 audio]\n   A --\u003e BF[API Endpoint: /v1/models]\n   BF --\u003e BG[LLM.models]\n   BG --\u003e BH[Return available models]\n   A --\u003e BI[CORS Middleware]\n   BJ[.env] --\u003e BK[Environment Variables]\n   BK --\u003e A\n   BL[setup.py] --\u003e BM[ezlocalai package]\n   BM --\u003e BN[LLM]\n   BM --\u003e BO[STT]\n   BM --\u003e BP[CTTS]\n   BM --\u003e BQ[IMG]\n   A --\u003e BR[API Key Verification]\n   BR --\u003e BS[verify_api_key]\n   A --\u003e BT[Static Files]\n   BT --\u003e BU[API Endpoint: /outputs]\n   A --\u003e BV[Ngrok]\n   BV --\u003e BW[Public URL]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDevXT-LLC%2Fezlocalai","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDevXT-LLC%2Fezlocalai","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDevXT-LLC%2Fezlocalai/lists"}