{"id":18950228,"url":"https://github.com/salesforce/botsim","last_synced_at":"2025-04-15T21:23:37.430Z","repository":{"id":66001645,"uuid":"522753466","full_name":"salesforce/botsim","owner":"salesforce","description":" BotSIM - a data-efficient end-to-end Bot SIMulation toolkit for evaluation, diagnosis, and improvement of commercial chatbots","archived":false,"fork":false,"pushed_at":"2023-06-12T21:53:24.000Z","size":19455,"stargazers_count":114,"open_issues_count":8,"forks_count":8,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-09T23:04:42.103Z","etag":null,"topics":["bot-testing","chatbot","dialog-simulator","dialogflow-chatbot","dialogue-systems","liveagent","paraphrase-generation","salesforce-bot","task-oriented-dialogue","testing"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salesforce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null}},"created_at":"2022-08-09T00:49:04.000Z","updated_at":"2024-11-28T20:41:14.000Z","dependencies_parsed_at":"2023-09-24T16:57:25.099Z","dependency_job_id":null,"html_url":"https://github.com/salesforce/botsim","commit_stats":{"total_commits":37,"total_committers":2,"mean_commits":18.5,"dds":0.08108108108108103,"last_synced_commit":"e382eabba2a40287c5c7e7cd46788c3fb59f1db3"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fbotsim","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fbotsim/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fbotsim/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salesforce%2Fbotsim/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salesforce","download_url":"https://codeload.github.com/salesforce/botsim/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249155173,"owners_count":21221553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bot-testing","chatbot","dialog-simulator","dialogflow-chatbot","dialogue-systems","liveagent","paraphrase-generation","salesforce-bot","task-oriented-dialogue","testing"],"created_at":"2024-11-08T13:21:50.572Z","updated_at":"2025-04-15T21:23:37.403Z","avatar_url":"https://github.com/salesforce.png","language":"Jupyter Notebook","readme":"\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"docs/_static/logo.png\" width=\"400\"/\u003e\n    \u003cbr\u003e\n\u003cp\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://github.com/salesforce/botsim/releases\"\u003e\u003cimg alt=\"Latest Release\" src=\"https://img.shields.io/github/release/salesforce/botsim.svg\" /\u003e\u003c/a\u003e\n  \u003ca href=\"https://opensource.salesforce.com/botsim/index.html\"\u003e\n  \u003cimg alt=\"docs\" src=\"https://github.com/salesforce/botsim/actions/workflows/docs.yaml/badge.svg\"/\u003e\n  \u003ca href=\"https://opensource.org/licenses/BSD-3-Clause\"\u003e\n  \u003cimg alt=\"license\" src=\"https://img.shields.io/badge/License-BSD_3--Clause-blue.svg\"/\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003ca href=\"https://arxiv.org/abs/2211.11982\"\u003eSystem Description\u003c/a\u003e,\n\u003ca href=\"https://arxiv.org/abs/2211.15916\"\u003eTechnical Report\u003c/a\u003e,\n\u003ca href=\"https://salesforce-botsim.herokuapp.com/\"\u003eDemo\u003c/a\u003e,\n\u003ca href=\"https://opensource.salesforce.com/botsim//latest/index.html\"\u003eDocumentation\u003c/a\u003e,\n\u003ca href=\"https://blog.salesforceairesearch.com/botsim/\"\u003eBlog\u003c/a\u003e\n\u003c/div\u003e\n\n\n# BotSIM:  An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems\n\n\n## Table of Contents\n1. [Introduction](#introduction)\n2. [Installation](#installation)\n3. [Getting Started](#getting-started)\n4. [Tutorials](#tutorial)\n4. [Documentation](#documentation)\n5. [System Demo Paper and Technical Report](#system-demo-paper-and-technical-report)\n\n\n## Introduction\nBotSIM is an open-source Bot SIMulation toolkit for large-scale data-efficient end-to-end evaluation, diagnosis and remediation of commercial task-oriented dialog systems (chatbots) across multiple bot platforms. \nThis toolkit aims to provide bot developers and practitioners with a one-stop solution to accelerate bot development and evaluation, reduce cost and time-to-market.\nAs a modular framework, BotSIM can be extended by bot developers to support new bot platforms. As a toolkit, BotSIM offers an easy-to-use App and a suite of command line tools for bot practitioners to readily apply BotSIM's \"generation-simulation-remediation\" pipeline. \n\nKey features of BotSIM include:\n\n- **Multi-stage bot evaluation**: BotSIM can be used for both pre-deployment testing and potentially post-deployment performance monitoring.\n- **Data-efficient dialogue generation**: Equipped with a deep network based paraphrasing model, BotSIM can generate an extensive set of test intent queries from the limited number of input intent utterances, which can be used to evaluate the bot intent model at scale.\n- **End-to-end bot evaluation via dialogue simulation**: Through automatic chatbot simulation, BotSIM can identify existing issues of the bot and evaluate both the natural language understanding (NLU) performance and the end-to-end dialogue performance such as goal completion rates.\n- **Bot health report dashboard**: The bot health report dashboard presents a multi-granularity top-down view of bot performance. Together with the analytical tools, they help bot practitioners quickly diagnose and troubleshoot their bots.\n- **Easy extension to new bot platform**: BotSIM can be easily extended to support new bot platforms. BotSIM currently supports  [Salesforce Einstein BotBuilder](https://help.salesforce.com/s/articleView?id=sf.bots_service_intro.htm\u0026type=5) and [Google DialogFlow CX](https://cloud.google.com/dialogflow/cx/docs/basics)\n\n\n## Prerequisite \nTo use BotSIM, users need to have API access to the following supported bot platforms:\n- Salesforce EinsteinBotBuilder LiveAgent API: `org ID`, `button ID`, `deployment Id`, `chat endpoint`\n- Google DialogFlow CX API: `location ID`, `project ID`, `agent ID` and JSON API token\n\n## Installation\n1. (Optional) Creating conda environment\n```bash\n   conda create -n botsim python=3.9\n   conda activate botsim\n```\n\n2. Cloning and building dependencies\n``` bash\n   git clone https://github.com/salesforce/botsim.git\n   cd botsim\n   pip install -r requirements.txt\n```\n\n## Getting Started\n\n### Streamlit Web App\nThe most straightforward way of getting started with BotSIM is the Streamlit Web App. \nThe multi-page App is developed to guide users to leverage BotSIM's \"generation-simulation-remediation\" pipeline for evaluation, diagnosis and remediation of their bots.     \n\u003cp align=\"center\" width=\"100%\"\u003e\n    \u003cimg width=\"100%\" src=\"docs/_static/BotSIM_health_reports_readme.png\"\u003e\n\u003c/p\u003e\n\n\nThe following commands can be used to start the Streamlit Web App locally:\n```bash\n  export PYTHONPATH=./:$PYTHONPATH\n  export DATABASE_URL=\"db/botsim_sqlite_demo.db\"\n  streamlit run botsim/streamlit_app/app.py\n```\nThe App can also be deployed to GCP for access to GPU resources. Detailed deployment instructions are given in the [GCP Deployment](https://opensource.salesforce.com/botsim//latest/advanced_usage.html#gcp-deployment) section of the documentation.\n\n### Command Line Tools\nAlternatively, users can also deep-dive to learn more about BotSIM's system components through the command line tools. Details are given in the [tutorial section](https://opensource.salesforce.com/botsim//latest/tutorials.html#botsim-command-line-tools) of the [code documentation](https://opensource.salesforce.com/botsim//latest/index.html#).\n\n## Tutorial\nWe provide the following tutorials in the [code documentation](https://opensource.salesforce.com/botsim//latest/tutorials.html). \n- [Streamlit Web App](https://opensource.salesforce.com/botsim//latest/tutorials.html#streamlit-web-app)\n- [Command Line Tools](https://opensource.salesforce.com/botsim//latest/tutorials.html#botsim-command-line-tools)\n- [Bot Health Dashboard Navigation](https://opensource.salesforce.com/botsim//latest/dashboard.html)\n- [Applying Remedidation Suggestions](https://opensource.salesforce.com/botsim//latest/dashboard.html#apply-intent-model-remediation-suggestions)\n\n## Documentation \nFor more details of the [system components](https://opensource.salesforce.com/botsim/latest/deep_dive.html) and [advanced usages](https://opensource.salesforce.com/botsim//latest/advanced_usage.html), please refer to the [code documentation](https://opensource.salesforce.com/botsim//latest/index.html#).\nWe welcome the contribution from the open-source community to improve BotSIM. To support new bot platforms, please also follow the [guidelines](https://opensource.salesforce.com/botsim/latest/advanced_usage.html) detailed in the code documentation.\n\n## System Demo and Technical Report\nA dashboard demo of the Streamlit App can be accessed [here](https://salesforce-botsim.herokuapp.com/).\nYou can find more details regarding BotSIM designs in our technical report:\n```\n@article{guangsen2022-botsim-tr,\n  author    = {Guangsen Wang and Junnan Li and Shafiq Joty and Steven Hoi},\n  title     = {BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems},\n  year      = {2022},\n  url       = {https://arxiv.org/abs/2211.15916},\n  archivePrefix = {arXiv},\n}\n```\nThe detailed system descriptions are presented in the following EMNLP 2022 system demo paper:\n```\n@article{guangsen2022-botsim-demo,\n  author    = {Guangsen Wang and Samson Tan and Shafqi Joty and Gang Wu and Jimmy Au and Steven Hoi},\n  title     = {BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems},\n  year      = {2022},\n  url       = {https://arxiv.org/abs/2211.11982},\n  archivePrefix = {arXiv},\n}\n```\nIf you're using BotSIM in your research or applications, please cite the papers.\n## Contact Us\nFeel free to contact botsim@salesforce.com for any comments, issues or suggestions.\n\n## License\n[BSD 3-Clause License](LICENSE.txt)\n","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fbotsim","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalesforce%2Fbotsim","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalesforce%2Fbotsim/lists"}