{"id":13455974,"url":"https://github.com/herval/cliobot","last_synced_at":"2025-04-13T19:14:39.182Z","repository":{"id":215286938,"uuid":"737589082","full_name":"herval/cliobot","owner":"herval","description":"open source bot framework for generative AI on telegram, discord, whatsapp \u0026 more","archived":false,"fork":false,"pushed_at":"2024-03-20T19:20:14.000Z","size":298,"stargazers_count":108,"open_issues_count":2,"forks_count":10,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-27T09:52:27.384Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/herval.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-31T16:37:23.000Z","updated_at":"2025-02-23T22:07:41.000Z","dependencies_parsed_at":"2024-01-23T03:42:59.662Z","dependency_job_id":"24964503-0d70-43cc-a535-5789498af0d9","html_url":"https://github.com/herval/cliobot","commit_stats":null,"previous_names":["herval/cliobot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herval%2Fcliobot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herval%2Fcliobot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herval%2Fcliobot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/herval%2Fcliobot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/herval","download_url":"https://codeload.github.com/herval/cliobot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248766769,"owners_count":21158301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T08:01:14.278Z","updated_at":"2025-04-13T19:14:39.161Z","avatar_url":"https://github.com/herval.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Cliobot - multimodal generative AI bot for chat platforms\n\n![Clio Bot](clio.jpg)\n\nCliobot is a modular bot platform for generative AI agents. It's goal is to provide a simple, easy to use and extendable\nplatform for running generative AI agents that can handle audio, video, text and images, on any chat platform.\n\nIt can be easily extend it to use multiple APIs and services, from Stable Diffusion to OpenAI, and you can run it on\nyour own device or deploy it online.\n\nIt comes with Telegram support and multiuser handling out of the box, and minimal dependencies.\n\nImportant: This repo is a work in progress - I'm porting over code from a startup I was working on, so it's still a bit\nrough and subject to multiple rewrites.\n\n## The Basic\n\nCliobot has two main working modes: __command mode__ or __LLM mode__\n\nIn __command mode__, you interact by using __slash commands__ (messages starting with a /). It comes with a set default\nof slash commands and you can easily create yor own.\n\n[WIP] In __LLM mode__, the bot works like chatgpt \u0026 other multimodal chatbots out there: it follows a configurable\nsystem prompt that defines its core behavior and can use functions to perform actions (including executing code or\nbrowsing the web).\n\nNotice both modes use the same command definitions, so the only difference between them is a tradeoff between more\nnatural language interpretation versus cost (since running GPT4 \u0026 other models can get expensive quickly).\n\n### Running the bot\n\n1. Install all dependencies with:\n\n```\npoetry install\n```\n\n2. Rename `config.example.yml` to `config.yml` and set the appropriate variables you want.\n\n3. Run the bot using the following command:\n\n```\npoetry run python fullbot.py\n```\n\nThe codebase includes other examples, such as a simple chat-only bot that uses OpenAI's API to respond to messages (`chatbot.py`). Documentation for these examples is still a work in progress.\n\n\n### Running tests\n```\npoetry run pytest\n```\n\n## Built-in commands\n\nCliobot comes with a set of built-in commands that you can use out of the box. You can also easily add your own!\n\n### /image\n\nGenerates an image from a text prompt.\n\nBuilt-in implementations: DALL-E 3, any image model hosted on Replicate.com.\n\n### /describe [WIP]\n\nDescribe an image using text.\n\nBuilt-in implementations: OpenAI GPT4V, Ollama (Llava, etc), any image to text model hosted on Replicate.com.\n\n### /transcribe\n\nTranscribes an audio file into text.\n\nBuilt-in implementations: OpenAI Whisper-1\n\n### /ask\n\nAsk a question to an LLM agent. This doesn't take any conversation context.\n\nBuilt-in implementations: GPT-4 or any model supported by [Ollama](https://github.com/jmorganca/ollama) running in\nserver mode, any LLM hosted on Replicate.com.\n\n### /chat [WIP]\n\nChat with an LLM agent, including a backlog of context\n\n## Command syntax\n\nCliobot uses a simple prompt parsing system (common across apps such as Midjourney \u0026 others). It's based on the\nfollowing format:\n\n```\n/\u003ccommand\u003e \u003ctext prompt\u003e? [--\u003cparam_name\u003e \u003cvalue\u003e]+\n```\n\nEach command handler is defined as a pydantic model, and the parameters are automatically parsed and validated.\n\nWhen a certain command requires multiple inputs, such as image, the bot will ask for them in sequence, then run the\ncommand after you provide all the inputs.\n\nAn example of a command using the default dalle3 image generation command would be as follows:\n\n```\n/image a giant hamster in space --size 1024x1024 --model dalle3\n```\n\n## Installing\n\nRunning a bot locally is simple:\n\n- Clone this repo\n- Setup the python env\n- Rename `config.example.yml` to `config.yml` and set the appropriate variables you want.\n- Install all dependencies with:\n\n```\npython -m venv create venv\nsource venv/bin/activate\npip install -r requirements.txt\n```\n\n### Running Cliobot on Telegram\n\nThe bare minimum you'll need is an API Token for a Telegram bot. Please refer to\nthe [official documentation](https://core.telegram.org/bots/tutorial#obtain-your-bot-token) for how to obtain an API\ntoken. It should look like this: `4839574812:AAFD39kkdpWt3ywyRZergyOLMaJhac60qc`\n\nOnce you get a token, change your `config.yml` to include the following session:\n\n```\nbot:\n  platform: telegram\n  token: \"1234567890:ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890\"\n```\n\nThen, run the bot using the following command:\n\n```\nsource venv/bin/activate\npython app.py\n```\n\n## Running with Docker\n\nIf you don't have Python on your system or just prefer to keep things simple, you can run Cliobot using Docker too:\n\n```\ndocker build -t cliobot .\ndocker run -it --rm -v $(pwd)/data:/content/data -v $(pwd)/.env:/content/.env -v $(pwd)/config.yml:/content/config.yml cliobot\n```\n\n## Using Automatic1111 WebUI as a backend\n\nYou can plug in [Automatic1111 WebUI](https://github.com/AUTOMATIC1111/stable-diffusion-webui) and use it as a backend for image generation! To do so, you'll need to set the following variables on your config.yml:\n\n```\nwebui:\n    endpoint: http://localhost:7860\n    auth: user:pass\n```\n\nNotice you'll need to start webui with the `--api` flag. The `auth` field is optional (you can leave it blank if you don't use API authentication). For more information on how to use the API, please refer to the [official documentation](https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/API).\n\n### Supported operations\n\nYou can use any Stable Diffusion model that's installed along webui with the `/image` command. The following is an example using all the supported parameters:\n\n\n```\n/image a hamster in space --negative cartoon, drawing, illustration --model sdxl1.0 --steps 20 --sampler 'DPM++ 2M SDE' --cfg 7 --seed 1234 --steps 50 --width 1024 --height 1024 --batchcount 1 --batchsize 4\n```\n\n\n## Configuring OpenAI\n\nTo use OpenAI models (gpt, dalle3, whisper, etc), include the following in your config.yml:\n\n```\nopenai:\n  endpoints:\n    - api_key: sk-....\n      api_type: open_ai\n      base_url: https://api.openai.com/v1/\n\n    - api_key: xxx\n      api_type: azure\n      api_version: 2023-10-01-preview\n      base_url: https://xxx.openai.azure.com\n      model: gpt4\n      kind: gpt-4\n\n    - api_key: xxx\n      api_type: azure\n      api_version: 2\n      base_url: https://xxx.openai.azure.com\n      model: embeddings\n      kind: embeddings\n\n    - api_key: xxx\n      api_type: azure\n      api_version: 2023-12-01-preview\n      base_url: https://xxx.openai.azure.com\n      model: dalle3\n      kind: dall-e-3\n\n    - api_key: xxx\n      api_type: azure\n      api_version: 2023-12-01-preview\n      base_url: https://xxx.openai.azure.com\n      model: whisper1\n      kind: whisper-1\n``` \n\nNotice that for Azure deployments, you'll need to set one entry per model\nkind (`dall-e-3`, `whisper-1`, `embeddings`, `gpt-4`). The API key can be the same for all of them.\n\n## Configuring Ollama\n\nIn order to use any LLM via Ollama, simply include the following in your config.yml:\n\n```\nollama:\n  endpoint: http://localhost:11434\n  models:\n    - llama2\n```\n\nEach model on the `models` list will be exposed as a model on the bot. You can then use it by using the `/ask` command:\n\n```\n/ask what's the meaning of life? --model llama2\n```\n\n## Configuring Replicate\n\nYou can use any model hosted on [Replicate](https://replicate.com/) by mapping it out on your config.yml. The mapping is\na bit more involved than other models, since you need to map out each parameter. Here's a complete example using SDXL\nhosted on Replicate:\n\n```\nreplicate:\n  api_token: xxx\n  endpoints:\n    - model: 'sdxl'\n      kind: 'image'\n      version: 'stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b'\n      params:\n        prompt:\n          kind: str\n          required: true\n        negative_prompt:\n          alias: no\n          kind: str\n        width:\n          kind: int\n          default: 1024\n        height:\n          kind: int\n          default: 1024\n        num_outputs:\n          alias: num\n          kind: int\n          default: 1\n        num_inference_steps:\n          alias: steps\n          kind: int\n          default: 25\n        guidance_scale:\n            alias: cfg\n            kind: float\n            default: 7.5\n        prompt_strength:\n            alias: ps\n            kind: float\n            default: 0.8\n        seed:\n            kind: int\n        apply_watermark:\n            alias: watermark\n            kind: bool\n            default: true\n        scheduler:\n            kind: str\n            default: 'KarrasDPM'\n        refine:\n            kind: str\n            alias: refiner\n            default: 'no_refiner'\n            value_map:\n              no: no_refiner\n              expert: expert_ensemble_refiner\n              base: base_image_refiner\n        refine_steps:\n            kind: int\n            alias: rs\n```\n\nWith the above config, you'll be able to generate images using the following command:\n\n```\n/image photo of a giant hamster in space --model sdxl --no illustration, cartoon, drawing --width 1280 --num 4 --steps 50 --rs 8 --refiner expert\n```\n\nNotice the parameter names on your slash command will match the param name on the config, _or_ an optional `alias`. This\nallows you to use shorter parameter names on your commands (eg typing out `--no` instead of `--negative_prompt`).\n\n## Built-in extensions\n\nThese are all deactivated by default, but easily enabled:\n\n- Sentry.io support for error reporting/tracking\n- Automatic message translation using Google Translate API\n- Utilization metrics using MixPanel\n- S3 for file storage\n\n## Features\n\n- OpenAI API support for DALL-E, GPT-3, GPT-4 and Whisper, including Azure support and multiple API keys\n- Ollama support for any LLM model (including image to text)\n- Support for any model hosted on Replicate.com\n- Multiuser support\n- File storage support (local \u0026 S3)\n- Automatic message translation using Google Translate API\n- Persistent preferences to reduce repetitive prompt parameters\n\n## Running on K8s\n\nTODO\n\n## Writing plugins\n\nTODO\n\n## Planned features\n\n- Discord integration\n- Whatsapp integration\n- Stable Diffusion\n- StableHorde processing\n\n## TODO\n\n- RAG mode\n- chat history\n- Finish the LLM mode\n- save generated images to storage\n- save uploads\n- i18n support\n- img2txt commands\n- llama implementation\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fherval%2Fcliobot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fherval%2Fcliobot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fherval%2Fcliobot/lists"}