{"id":43116988,"url":"https://github.com/OutofAi/ChitChat","last_synced_at":"2026-02-11T16:01:33.296Z","repository":{"id":212001317,"uuid":"730446649","full_name":"OutofAi/ChitChat","owner":"OutofAi","description":"Modal LLM LLama.cpp based model deployment as part of series of Model as a Service (MaaS)","archived":false,"fork":false,"pushed_at":"2025-01-10T00:06:48.000Z","size":41,"stargazers_count":15,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-15T21:05:24.423Z","etag":null,"topics":["llamacpp","llm","llm-inference","machine-learning","mistral","mistral-7b","modelasservice","modeldeployment","openhermes","serverless"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OutofAi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-12T00:15:09.000Z","updated_at":"2025-10-06T09:22:49.000Z","dependencies_parsed_at":"2024-11-16T18:02:05.150Z","dependency_job_id":null,"html_url":"https://github.com/OutofAi/ChitChat","commit_stats":null,"previous_names":["outofai/chitchatsource"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OutofAi/ChitChat","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OutofAi%2FChitChat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OutofAi%2FChitChat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OutofAi%2FChitChat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OutofAi%2FChitChat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OutofAi","download_url":"https://codeload.github.com/OutofAi/ChitChat/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OutofAi%2FChitChat/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29336999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T16:00:30.228Z","status":"ssl_error","status_checked_at":"2026-02-11T16:00:25.398Z","response_time":97,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llamacpp","llm","llm-inference","machine-learning","mistral","mistral-7b","modelasservice","modeldeployment","openhermes","serverless"],"created_at":"2026-01-31T19:05:30.491Z","updated_at":"2026-02-11T16:01:33.291Z","avatar_url":"https://github.com/OutofAi.png","language":"Python","funding_links":["https://www.buymeacoffee.com/outofAI"],"categories":["LLMs/Multimodal Models"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OutofAi/ChitChatSource/assets/145302363/798510c4-c92f-47f3-8728-738f5b1333bc\" alt=\"logo\"\u003e\n\u003c/p\u003e\n\n\u003ctable style=\"border-collapse: collapse; width: 100%;\" border=\"1\" align=\"center\"\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 100%;\"\u003eGPU variation\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 100%;\"\u003e\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OutofAi/ChitChatSource/assets/145302363/08c3d21f-6d70-4e33-a3aa-a4c40a30ae6d\" alt=\"hello world\"\u003e\n\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003cp\u003eThis is the first part of a collection of templates we are working on for promoting the concept of Model as a Serivce (MaaS). Mainly revolving around using Firebase/Modal/Stripe. One of the user friendliest and cheapest way to deploy your model and creating inference endpoint API is \u003ca href=\"https://modal.com/\"\u003eModal\u003c/a\u003e. This example shows the simplicity of deploying Mistral 7B Instruct v0.1 - GGUF with only few lines of code and deploying it on Modal. But you can change it to any model that is supported by LLamacpp\u003c/p\u003e\n\u003chr /\u003e\n\u003cp\u003eFollow us on X for updates regarding the other templates\u003cbr /\u003e\u003ca href=\"https://twitter.com/OutofAi\"\u003ehttps://twitter.com/OutofAi\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eand also support our channel \u003cbr /\u003e\u003ca href=\"https://www.buymeacoffee.com/outofAI\"\u003ehttps://www.buymeacoffee.com/outofAI\u003c/a\u003e\u003c/p\u003e\n\u003chr /\u003e\n\u003ch2 dir=\"auto\" tabindex=\"-1\"\u003ePrerequisites\u003c/h2\u003e\n\u003cp\u003eMake sure you have created an account on \u003ca href=\"https://modal.com/\"\u003eModal.com\u003c/a\u003e and install the required Python packages\u003c/p\u003e\n\u003cpre\u003epip install modal\u003c/pre\u003e\n\u003cp\u003eThe next command will help you to automatically create a token and set everything up and log you in to simplify deployment\u003c/p\u003e\n\u003cpre\u003epython3 -m modal setup\u003c/pre\u003e\n\u003cp\u003eThis is all you need to be able to generate an endpoint.\u003c/p\u003e\n\u003ch2 dir=\"auto\" tabindex=\"-1\"\u003eDeploy\u003c/h2\u003e\n\u003cp\u003eThere are two examples avaiable here and depending on cost you can choose which one you like to deploy. We recommend deploying the cpu version first before attempting the gpu one. To deploy the model to create an inference endpoint API you only need to run this command.\u003c/p\u003e\n\u003cp\u003eCPU version:\u003c/p\u003e\n\u003cpre\u003emodal deploy chitchat-cpu.py\u003c/pre\u003e\n\u003cp\u003eGPU version (Running on T4):\u003c/p\u003e\n\u003cpre\u003emodal deploy chitchat-gpu.py\u003c/pre\u003e\n\u003cp\u003eAfter a successful process you will be given entrypoint link in this format\u003c/p\u003e\n\u003cpre\u003eCreated entrypoint: https://[ORG_NAME]--[NAME]-entrypoint.modal.run\u003c/pre\u003e\n\u003ch2 dir=\"auto\" tabindex=\"-1\"\u003eInference\u003c/h2\u003e\n\u003cp\u003eWe put together a website https://chitchatsource.com/ to simplify and enhance user experience, insert the provided link in previous step on that page to run inference on your model.\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OutofAi/ChitChatSource/assets/145302363/79a79b25-5d5b-4e81-b972-b49cc472de66\" alt=\"ChitChat-Settings\"\u003e\n\u003c/p\u003e\n\n\u003cp\u003eAfter saving your deployment link you should be able to run inference on the model. You can use this website for running local FastAPI inference endpoint as well. You just need to make sure the formating and parameters expected matches the one provided in this example. I will do a different repository related to that.\u003c/p\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOutofAi%2FChitChat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOutofAi%2FChitChat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOutofAi%2FChitChat/lists"}