{"id":25109882,"url":"https://github.com/glific/open-llm","last_synced_at":"2025-04-22T07:25:37.092Z","repository":{"id":189159231,"uuid":"680151138","full_name":"glific/Open-LLM","owner":"glific","description":"LLM leveraging OpenAI API to be used in Glific via webhooks","archived":false,"fork":false,"pushed_at":"2024-05-13T06:57:31.000Z","size":51199,"stargazers_count":4,"open_issues_count":5,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-29T14:06:45.508Z","etag":null,"topics":["llm","openai","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/glific.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-18T13:14:56.000Z","updated_at":"2024-10-13T21:31:28.000Z","dependencies_parsed_at":"2023-09-25T06:57:15.807Z","dependency_job_id":"e83d9ede-2d24-46ec-a887-90668caa840f","html_url":"https://github.com/glific/Open-LLM","commit_stats":null,"previous_names":["glific/open-llm"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glific%2FOpen-LLM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glific%2FOpen-LLM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glific%2FOpen-LLM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/glific%2FOpen-LLM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/glific","download_url":"https://codeload.github.com/glific/Open-LLM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249837179,"owners_count":21332405,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","openai","python"],"created_at":"2025-02-08T00:32:36.597Z","updated_at":"2025-04-20T01:29:32.189Z","avatar_url":"https://github.com/glific.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Open-LLM\n\nThis project aims to take Glific interactions to the next level by seamlessly integrating OpenAI's Language Model to generate contextually accurate responses from the custom knowledge base. This will be a standalone repository within the Glific ecosystem and will be hosted separately. Through the use of Glific webhooks, this will be connected to the Glific instance for better contextual response through the chatbot\n\n## Architecture\n\n![Diagram of overall chain](diagram-of-overall-chain.png)\n\n## Development\n\n### Prerequisites\n\n#### Python \u0026 pip\n\nMake sure you have Python and pip installed. You can check this by running:\n\n```bash\npython --version\npip --version\n```\n\n#### Pipenv\n\nThis project uses [pipenv](https://pipenv.pypa.io/en/latest/) for dependency management. To install pipenv, run:\n\n```bash\npip install pipenv --user\n```\n\n## Getting Started\n\n### .env\n\nRun `cp .env.example .env` file and add your OpenAI key.\n\n### Start database\n\nWe are using Postgres to store conversation history. You can start the database using docker-compose:\n\n```bash\ndocker-compose up\n```\n\n### Project dependencies\n\nThis project uses Python 3.9 and virtual environments:\n\nCreate a virtualenv:\n\n```bash\npython -m venv venv\n```\n\nActivate the virtualenv:\n\n```bash\nsource venv/bin/activate\n```\n\nInstall the dependencies, run:\n\n```bash\npip install -r requirements.txt\n```\n\nRun migration\n\n```bash\npython manage.py migrate\n```\n\n### Start the server\n\n```bash\npython manage.py runserver\n```\n\n### Seeding\n\nTo seed the database with a sample organization, open a python shell (`python manage.py shell`) and run the following command:\n\n```python\nfrom llm.models import Organization\nOrganization.objects.create(\n  name=\"Myna Mahila\",\n  system_prompt=\"I want you to act as a chatbot for providing tailored sexual and reproductive health advice to women in India. You represent an organization called The Myna Mahila Foundation (mynamahila.com), an Indian organization which empowers women by encouraging discussion of taboo subjects such as menstruation, and by setting up workshops to produce low-cost sanitary protection to enable girls to stay in school. In India, majority of girls report not knowing about menstruation before their first period. This is because of limited access to unbiased information due to stigma, discrimination, and lack of resources. The information you provide needs to be non-judgmental, confidential, accurate, and tailored to those living in urban slums. Your response should be in the same language as the user's input.\",\n  api_key=\"sk_ABC123\",\n)\n```\n\nTo make requests to the API with the organization's API key, use the following command:\n\n```bash\ncurl -X POST -H \"Authorization: sk_ABC123\" -H \"Content-Type: application/json\" -d '{\"system_prompt\":\"You are a chatbot that formats your responses as poetry.\"}' http://localhost:8000/api/system_prompt\n```\n\nTo upload a file from `llm/data/sources/*` and ultimately create embeddings out of it, use the following command:\n\n```bash\ncurl -X POST -H \"Authorization: sk_ABC123\" -H \"Content-Type: multipart/form-data\" -F \"file=@llm/data/sources/ANXIETY.docx.pdf\" http://localhost:8000/api/upload\n```\n\nFor testing and convenience, running the `upload_docs.sh` script will upload all the files in `llm/data/sources/*` for embeddings to be created out of them.\n\n```bash\n./upload_docs.sh\n```\n\n### Query\n\nFor testing the LLM, we are using [HTTPie](https://httpie.io) because of its succint syntax. You may alternatively use curl or Postman or your favorite HTTP client for this.\n\nYou can query the LLM using the following command:\n\n```bash\ncurl -X POST -H \"Authorization: sk_ABC123\" -H \"Content-Type: application/json\" -d '{\"prompt\": \"Peshab ki jagah se kharash ho rahi hai\"}' http://localhost:8000/api/chat\n```\n\nThis will return a JSON endpoint with the LLM response as well as a session id.\n\n```json\n{\n  \"answer\": \"Aapki samasya ke liye dhanyavaad. Yah peshab ke samay kharash ki samasya ho sakti hai. Isko urinary tract infection (UTI) kaha jata hai. Urinary tract infection utpann hone ka mukhya karan antarik infection ho sakta hai.\",\n  \"chat_history\": [],\n  \"session_id\": \"uhh0pq\"\n}\n```\n\nTo ask a fellow up question, you can use the session id returned in the previous response:\n\n```bash\ncurl -X POST -H \"Authorization: sk_ABC123\" -H \"Content-Type: application/json\" -d '{\"prompt\":\"Peshab ki jagah kharash hai\",\"session_id\":\"uhh0pq\"}' http://127.0.0.1:8000/api/chat\n```\n\n```json\n{\n  \"answer\": \"aapki samasya ke liye dhanyavad. Yah peshaab ki jagah me kharash ho sakti hai. Isko urinary tract infection (UTI) kaha jata hai. UTI utpann hone ka mukhya karan aantarik infection ho sakta hai.\",\n  \"chat_history\": [\n    [\n      [\"content\", \"Peshab ki jagah se kharash ho rahi hai\"],\n      [\"additional_kwargs\", {}],\n      [\"type\", \"human\"],\n      [\"example\", false]\n    ],\n    [\n      [\n        \"content\",\n        \"Aapki samasya ke liye dhanyavaad. Yah peshab ke samay kharash ki samasya ho sakti hai. Isko urinary tract infection (UTI) kaha jata hai. Urinary tract infection utpann hone ka mukhya karan antarik infection ho sakta hai.\"\n      ],\n      [\"additional_kwargs\", {}],\n      [\"type\", \"ai\"],\n      [\"example\", false]\n    ]\n  ],\n  \"session_id\": \"uhh0pq\"\n}\n```\n\nThe default model used is [`gpt-3.5-turbo`](https://platform.openai.com/docs/models/gpt-3-5) but you can specify a different GPT model by passing a `gpt_model` parameter in the request body.\n\n```bash\ncurl -X POST -H \"Authorization: sk_ABC123\" -H \"Content-Type: application/json\" -d '{\"prompt\":\"Mujhe peshab ki jagah pe kharash ho rahi hai\",\"gpt_model\":\"gpt-3.5-turbo-16k\"}' http://127.0.0.1:8000/api/chat\n```\n\n```json\n{\n  \"answer\": \"Aapki samasya ke liye dhanyavaad. Yah peshaab karne ke samay kharash ki samasya ho sakti hai. Isko urinary tract infection (UTI) kaha jata hai. UTI utpann hone ka mukhya karan aantrik infection ho sakta hai.\",\n  \"chat_history\": [],\n  \"session_id\": \"cfbQXg\"\n}\n```\n\n### Evaluation\n\nTo use an secondary LLM to evaluate the output of the primary LLM, users can set the evaluator prompt at the organization level:\n\nREFERENCE G-EVAL PROMPT:\n\n```\nYou will be given one summary written for a news article.\nYour task is to rate the summary on one metric.\nPlease make sure you read and understand these instructions carefully. Please keep this\ndocument open while reviewing, and refer to it as needed.\nEvaluation Criteria:\nCoherence (1-5) - the collective quality of all sentences. We align this dimension with\nthe DUC quality question of structure and coherence whereby ”the summary should be\nwell-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.”\nEvaluation Steps:\n1. Read the news article carefully and identify the main topic and key points.\n2. Read the summary and compare it to the news article. Check if the summary covers the main\ntopic and key points of the news article, and if it presents them in a clear and logical order.\n3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest\nbased on the Evaluation Criteria.\nExample:\nSource Text:\n{{Document}}\nSummary:\n{{Summary}}\nEvaluation Form (scores ONLY):\n- Coherence:\n```\n\n```sh\ncurl --request POST \\\n  --url http://localhost:8000/api/evaluator_prompt \\\n  --header 'Authorization: sk_ABC123' \\\n  --header 'Content-Type: application/json' \\\n  --data '{\n  \"evaluator_prompts\": {\n    \"friendliness\": \"You will be given a response from a womens health chatbot.\\nYour task is to rate the response on one metric.\\nPlease make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.\\n\\nEvaluation Criteria:\\nFriendliness (1-10) - how friendly the response is to an audience of women who traditionally do not have open conversations about health issues due to stigma.\\n\\nEvaluation Steps:\\n1. Read the chatbot response in detail.\\n2. Analyze the tone and word choice of the response.\\n3. Considering the women are talking to this chatbot in private, assess whether the response is considerate of their situation.\\n4. Assign an integer score for friendliness on a scale of 1 to 10, where 1 is the lowest and 10 is the highest based on the Evaluation Criteria.\\n\\nExample:\\nUser Question:\\n{{Question}}\\nChatbot Response:\\n{{Response}}\\n\\nEvaluation Form (scores ONLY):\\n- Friendliness:\",\n    \"factuality\": \"You will be given a response from a women'\\''s health chatbot. Your task is to rate the factual accuracy of the response. Please be certain you have understood these instructions, keep this document open during the process for reference as needed.\\nEvaluation Criteria:\\nFactuality (1-10) - assess the degree to which the given response accurately reflects current, scientifically-backed knowledge in the field of women'\\''s health.\\nEvaluation Steps:\\n1. Carefully review the chatbot response.\\n2. Compare and contrast the information in the response with your knowledge and/or verified sources.\\n3. Assess the correct presentation of facts and utilization of credible, scientific information.\\n4. Based on the Evaluation Criteria, assign a integer score for factuality on a scale of 1 to 10. 1 indicates entirely inaccurate, whilst 10 suggests perfect factual accuracy.\\nExample:\\nUser Question:\\n{{Question}}\\nChatbot Response:\\n{{Response}}\\nEvaluation Form (scores ONLY):\\nFactuality:\"\n  }\n}'\n```\n\nTo run the evaluator, you simply call the chat endpoint with the `evaluator` parameter set to `true`:\n\n```sh\ncurl -X POST -H 'Authorization: sk_ABC123' -H 'Content-Type: application/json' -d '{\"evaluate\": true, \"prompt\": \"Peshab ki jagah se kharash ho rahi hai\"}' http://localhost:8000/api/chat\n```\n\nThe response will include your specific evaluation scores in a `evaluation_scores` key in the response:\n\n```json\n{\n  \"answer\": \"Aapki samasya ke liye dhanyavaad. Yah peshab ke samay kharash ki samasya ho sakti hai. Isko urinary tract infection (UTI) kaha jata hai. Urinary tract infection utpann hone ka mukhya karan antarik infection ho sakta hai.\",\n  \"chat_history\": [],\n  \"evaluation_scores\": {\n    \"coherence\": 4,\n    \"consistency\": 3\n  },\n  \"session_id\": \"uhh0pq\"\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglific%2Fopen-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fglific%2Fopen-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fglific%2Fopen-llm/lists"}