{"id":27172801,"url":"https://github.com/genta-technology/kolosal","last_synced_at":"2025-04-09T10:30:20.059Z","repository":{"id":259141948,"uuid":"876403841","full_name":"Genta-Technology/Kolosal","owner":"Genta-Technology","description":"Kolosal AI is an OpenSource and Lightweight alternative to LM Studio to run LLMs 100% offline on your device.","archived":false,"fork":false,"pushed_at":"2025-04-08T16:59:31.000Z","size":86572,"stargazers_count":177,"open_issues_count":12,"forks_count":12,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-08T17:50:56.904Z","etag":null,"topics":["c","cpp","deepseek","gemma","gemma2","gemma3","gpt","llama","llama2","llama3","llamacpp","llava","llm","llms","localai","mistral","phi3","phi4","qwen","self-hosted"],"latest_commit_sha":null,"homepage":"https://kolosal.ai","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Genta-Technology.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["Genta-Technology"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"thanks_dev":null,"custom":null}},"created_at":"2024-10-21T23:04:43.000Z","updated_at":"2025-04-08T16:59:36.000Z","dependencies_parsed_at":"2025-02-08T23:19:57.377Z","dependency_job_id":"e5638fdd-30ae-4db2-9096-da2549700f35","html_url":"https://github.com/Genta-Technology/Kolosal","commit_stats":null,"previous_names":["genta-technology/kolosal-desktop","genta-technology/kolosal.cpp"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genta-Technology%2FKolosal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genta-Technology%2FKolosal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genta-Technology%2FKolosal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genta-Technology%2FKolosal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Genta-Technology","download_url":"https://codeload.github.com/Genta-Technology/Kolosal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020242,"owners_count":21034404,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c","cpp","deepseek","gemma","gemma2","gemma3","gpt","llama","llama2","llama3","llamacpp","llava","llm","llms","localai","mistral","phi3","phi4","qwen","self-hosted"],"created_at":"2025-04-09T10:30:19.052Z","updated_at":"2025-04-09T10:30:20.007Z","avatar_url":"https://github.com/Genta-Technology.png","language":"C++","readme":"## Kolosal AI\n\nhttps://github.com/user-attachments/assets/589cfb48-f806-493d-842b-3b6953b64e79\n\n**Kolosal AI** is an open-source desktop application designed to simplify the training and inference of large language models on your own device. It supports any CPU with **AVX2** instructions and also works with **AMD** and **NVIDIA** GPUs. Built to be lightweight (only ~20 MB compiled), **Kolosal AI** runs smoothly on most edge devices, enabling on-premise or on-edge AI solutions without heavy cloud dependencies.\n\n- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)\n- **Developer:** [Genta Technology](https://genta.tech)\n- **Community:** [Join our Discord](https://discord.gg/XDmcWqHmJP)\n\n### Key Features\n\n1. **Universal Hardware Support**  \n   - AVX2-enabled CPUs  \n   - AMD and NVIDIA GPUs\n\n2. **Lightweight \u0026 Portable**  \n   - Compiled size ~20 MB  \n   - Ideal for edge devices like Raspberry Pi or low-power machines\n\n3. **Wide Model Compatibility**  \n   - Supports popular models like **Mistral**, **LLaMA**, **Qwen**, and many more  \n   - Powered by the [Genta Personal Engine](https://genta.tech) built on top of [Llama.cpp](https://github.com/ggerganov/llama.cpp)\n\n4. **Easy Dataset Generation \u0026 Training**  \n   - Build custom datasets with minimal overhead  \n   - Train models using **UnsLOTH** or other frameworks  \n   - Deploy locally or as a server in just a few steps\n\n5. **On-Premise \u0026 On-Edge Focus**  \n   - Keeps data private on your own infrastructure  \n   - Lowers costs by avoiding expensive cloud-based solutions\n\n### Use Cases\n\n- **Local AI Inference:** Quickly run LLMs on your personal laptop or desktop for offline or on-premise scenarios.  \n- **Edge Deployment:** Bring large language models to devices with limited resources, ensuring minimal latency and improved privacy.  \n- **Custom Model Training:** Simplify the process of data preparation and model training without relying on cloud hardware.\n\n---\n\n## Credits \u0026 Attribution\n\nKolosal AI uses or references the following third-party projects, each licensed under their respective terms:\n\n- [Dear ImGui](https://github.com/ocornut/imgui) (MIT License)  \n- [llama.cpp](https://github.com/ggerganov/llama.cpp) (MIT License)  \n- [nativefiledialog-extended](https://github.com/btzy/nativefiledialog-extended) (zlib License)  \n- [nlohmann/json](https://github.com/nlohmann/json) (MIT License)  \n- [stb libraries](https://github.com/nothings/stb) (Public Domain or MIT License)  \n\nThese projects are distributed under their own licenses, separate from Kolosal AI. We are not affiliated with nor endorsed by the above entities.\n\n---\n\n## About Genta Technology\n\nWe are a small team of students passionate about addressing key concerns in AI such as **energy**, **privacy**, **on-premise**, and **on-edge** computing. Our flagship product is the **Genta Inference Engine**, which allows enterprises to deploy open-source models on their own servers, with **3-4x higher throughput**. This can reduce operational costs by up to **80%**, as a single server optimized by our engine can handle the workload of four standard servers.\n\n---\n\n### Get Involved\n\n1. **Clone the Repository**: [https://github.com/Genta-Technology/Kolosal]\n2. **Join the Community**: Ask questions, propose features, and discuss development on our [Discord](https://discord.gg/XDmcWqHmJP).  \n3. **Contribute**: We welcome pull requests, bug reports, feature requests, and any kind of feedback to improve **Kolosal AI**.\n\n---\n\n## How to Compile\n\n1. [Project Overview](#project-overview)\n2. [Directory Structure](#directory-structure)\n3. [Prerequisites](#prerequisites)\n4. [Cloning and Preparing the Repository](#cloning-and-preparing-the-repository)\n5. [Configuring the Project with CMake](#configuring-the-project-with-cmake)\n6. [Building the Application](#building-the-application)\n7. [Running the Application](#running-the-application)\n8. [Troubleshooting](#troubleshooting)\n\n---\n\n## Project Overview\n\n- **Name:** Kolosal AI (Desktop application target is `KolosalDesktop`)\n- **Language Standard:** C++17\n- **Build System:** [CMake](https://cmake.org/) (version 3.14 or higher)\n- **Dependencies** (automatically handled by the provided `CMakeLists.txt`, if placed in correct directories):\n  - OpenGL\n  - OpenSSL\n  - CURL\n  - GLAD\n  - Native File Dialog Extended\n  - genta-personal engine libraries (InferenceEngineLib, InferenceEngineLibVulkan)\n  - ImGui (provided in `external/imgui`)\n  - Other external libraries: `stb`, `nlohmann/json`, `icons`, etc.\n\n## Directory Structure\n\nA simplified look at the important folders/files:\n\n```\nKolosalAI/\n├─ cmake/\n│   └─ ucm.cmake                 # Utility script for static runtime linking\n├─ external/\n│   ├─ curl/                     # Pre-built or source for cURL\n│   ├─ glad/                     # GLAD loader\n│   ├─ genta-personal/           # genta-personal engine includes/libs\n│   ├─ imgui/                    # ImGui source\n│   ├─ nativefiledialog-extended # Native File Dialog Extended\n│   ├─ nlohmann/                 # JSON library\n│   ├─ stb/                      # stb (single-file) headers\n│   └─ fonts/                    # TrueType fonts\n├─ assets/\n│   ├─ logo.png\n│   └─ resource.rc               # Windows resource file\n├─ source/\n│   └─ main.cpp                  # Entry point for KolosalDesktop\n├─ include/\n│   └─ ... (additional headers)\n├─ models/\n│   └─ ... (model.json configuration files used by the inference engine to download, save, and load the model engine)\n├─ CMakeLists.txt\n├─ README.md                     # You are here!\n└─ ...\n```\n\n## Prerequisites\n\n1. **CMake 3.14 or above**  \n   Download from [https://cmake.org/download/](https://cmake.org/download/).\n2. **A C++17-compatible compiler**  \n   - For Windows, Visual Studio 2019/2022 (MSVC) or [MinGW-w64](https://www.mingw-w64.org/) with GCC 7+.\n   - For other platforms, an equivalent compiler supporting C++17.\n3. **Git** (optional, but recommended for cloning and submodule management).\n4. **OpenSSL**, **CURL**  \n   - On Windows, you can place the pre-built bins/headers inside `external/openssl` and `external/curl` (or anywhere you prefer, just ensure `CMakeLists.txt` sees them).\n5. **(Optional) Vulkan SDK** if you plan to use the Vulkan-based inference engine.\n\n## Cloning and Preparing the Repository\n\n1. **Clone the repository**:\n\n   ```bash\n   git clone https://github.com/Genta-Technology/Kolosal.git\n   cd KolosalAI\n   ```\n\n2. **(Optional) Update submodules**:  \n   If any external libraries are handled as Git submodules, initialize them:\n\n   ```bash\n   git submodule update --init --recursive\n   ```\n\n3. **Check external dependencies**:  \n   Ensure the `external` folder contains:\n   - `curl` with `include/`, `lib/`, and `bin/` (Windows).\n   - `openssl` or that OpenSSL is installed system-wide.\n   - The `genta-personal` engine in place if not fetched from elsewhere.\n\n4. **Folder structure verification**:  \n   Verify that folders like `nativefiledialog-extended`, `imgui`, etc., are present inside `external/`.\n\n## Configuring the Project with CMake\n\nYou can perform either an in-source or out-of-source build, but **out-of-source** is recommended. Below is an example of an out-of-source build:\n\n1. **Create a build folder**:\n\n   ```bash\n   mkdir build\n   cd build\n   ```\n\n2. **Run CMake**:  \n   By default, this will generate build files for your platform’s default generator (e.g., Visual Studio solution files on Windows, Makefiles on Linux, etc.):\n\n   ```bash\n   cmake -S .. -B . -DCMAKE_BUILD_TYPE=Release\n   ```\n\n   or explicitly (for Visual Studio multi-config):\n\n   ```bash\n   cmake -S .. -B . -G \"Visual Studio 17 2022\" -A x64\n   ```\n\n   - `-DDEBUG=ON` can be used if you want to build a debug version:\n\n     ```bash\n     cmake -S .. -B . -DCMAKE_BUILD_TYPE=Debug -DDEBUG=ON\n     ```\n\n3. **Check for any errors** during configuration, such as missing libraries or headers. Resolve them by installing or copying the required dependencies into the correct location.\n\n## Building the Application\n\nAfter successful configuration:\n\n- **On Windows with Visual Studio**:  \n  Open the generated `.sln` file inside `build/` and build the solution. Or build from the command line using:\n\n  ```bash\n  cmake --build . --config Release\n  ```\n\n- **On other platforms** (e.g., using Make or Ninja):\n\n  ```bash\n  cmake --build . --config Release\n  ```\n\n\u003e **Note**:  \n\u003e The `POST_BUILD` commands in `CMakeLists.txt` will copy the necessary DLLs, fonts, assets, and models into the final output folder (e.g., `build/Release/` or `build/Debug/`, depending on your generator).\n\n## Running the Application\n\n1. **Locate the output**:  \n   Once the build completes, you should find the executable (e.g., `KolosalDesktop.exe` on Windows) in a directory such as:\n   - `build/Release/` (Visual Studio).\n   - `build/` (single-config generators like Make).\n\n2. **Check for required files**:  \n   The post-build commands should have copied:\n   - **Fonts** (`/fonts` folder next to the exe).\n   - **Assets** (`/assets` folder next to the exe).\n   - **Models** (`/models` folder next to the exe).\n   - **OpenSSL** and **InferenceEngine** DLLs (Windows).\n   - **cURL** DLL(s) (Windows).\n\n   Make sure these folders and files are present in the same directory as `KolosalDesktop.exe`.\n\n3. **Double-click or run from terminal**:\n\n   ```bash\n   cd build/Release\n   ./KolosalDesktop.exe\n   ```\n\n4. **Enjoy Kolosal AI**!\n\n## Troubleshooting\n\n1. **OpenSSL or CURL not found**  \n   - Make sure you have them installed or placed in `external/openssl` and `external/curl` respectively.  \n   - Check environment variables like `OPENSSL_ROOT_DIR` or `CURL_ROOT_DIR` if needed.  \n   - Update `CMAKE_PREFIX_PATH` if you’re placing these libraries somewhere non-standard.\n\n2. **InferenceEngine libraries not found**  \n   - Verify the path `external/genta-personal/lib` actually contains `InferenceEngineLib.lib` or `InferenceEngineLibVulkan.lib` (on Windows).  \n   - Adjust `find_library` paths in `CMakeLists.txt` if your structure differs.\n\n3. **Missing Vulkan SDK**  \n   - If you plan to use the Vulkan-based inference engine, ensure Vulkan SDK is installed and available in your PATH or that CMake can find it.\n\n4. **ImGui not found**  \n   - Ensure `external/imgui` folder is not empty.  \n   - If you see compilation errors referencing ImGui headers, check that `target_include_directories` in `CMakeLists.txt` still points to the correct path.\n\n5. **Resource or Icon issues on non-Windows**  \n   - The `assets/resource.rc` file is Windows-specific. For Linux/macOS builds, you can comment out or remove references to `.rc` if they cause issues.\n\n6. **Runtime errors** due to missing DLLs or dynamic libraries  \n   - Confirm that the post-build step copies all required `.dll` files next to the executable.  \n   - On Linux/macOS, ensure `.so`/`.dylib` are in the library search path or same folder.\n","funding_links":["https://github.com/sponsors/Genta-Technology"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenta-technology%2Fkolosal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgenta-technology%2Fkolosal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenta-technology%2Fkolosal/lists"}