{"id":13704448,"url":"https://github.com/aiverify-foundation/moonshot","last_synced_at":"2026-02-05T09:06:26.792Z","repository":{"id":212447820,"uuid":"731489151","full_name":"aiverify-foundation/moonshot","owner":"aiverify-foundation","description":"Moonshot - A simple and modular tool to evaluate and red-team any LLM application.","archived":false,"fork":false,"pushed_at":"2026-02-03T08:37:57.000Z","size":235730,"stargazers_count":308,"open_issues_count":2,"forks_count":59,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-02-03T16:42:09.760Z","etag":null,"topics":["benchmarking","evaluation-framework","llm","red-teaming","trustworthy-ai"],"latest_commit_sha":null,"homepage":"https://aiverify-foundation.github.io/moonshot/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aiverify-foundation.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-14T07:43:03.000Z","updated_at":"2026-02-02T10:46:49.000Z","dependencies_parsed_at":"2025-01-13T11:37:31.836Z","dependency_job_id":"117626bc-913a-457c-94e7-9b9391ebeeb3","html_url":"https://github.com/aiverify-foundation/moonshot","commit_stats":{"total_commits":1159,"total_committers":35,"mean_commits":"33.114285714285714","dds":0.8317515099223468,"last_synced_commit":"d9234bf9ff3673dd88d6007a2e7e5003cea1a5e0"},"previous_names":["moonshot-admin/moonshot","aiverify-foundation/moonshot"],"tags_count":25,"template":false,"template_full_name":null,"purl":"pkg:github/aiverify-foundation/moonshot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiverify-foundation%2Fmoonshot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiverify-foundation%2Fmoonshot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiverify-foundation%2Fmoonshot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiverify-foundation%2Fmoonshot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aiverify-foundation","download_url":"https://codeload.github.com/aiverify-foundation/moonshot/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aiverify-foundation%2Fmoonshot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29117917,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-05T05:31:32.482Z","status":"ssl_error","status_checked_at":"2026-02-05T05:31:29.075Z","response_time":65,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","evaluation-framework","llm","red-teaming","trustworthy-ai"],"created_at":"2024-08-02T21:01:09.939Z","updated_at":"2026-02-05T09:06:26.787Z","avatar_url":"https://github.com/aiverify-foundation.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n\n![Moonshot Logo](https://github.com/aiverify-foundation/moonshot/raw/main/misc/aiverify-moonshot-logo.png)\n\n**Version 0.7.6**\n\nA simple and modular tool to evaluate any LLM-based AI systems.\n\n[![Python 3.11](https://img.shields.io/badge/python-3.11-green)](https://www.python.org/downloads/release/python-3111/)\n\n\n\u003c/div\u003e\n\n## 🎯 Motivation\n\nDeveloped by the [AI Verify Foundation](https://aiverifyfoundation.sg/), [Moonshot](https://aiverifyfoundation.sg/project-moonshot/) is a tool to bring Benchmarking and Red-Teaming together to help AI developers, compliance teams evaluate LLM-based Apps and LLMs.\n\n\u003c/br\u003e\n\n## 🚀 Why Moonshot\n\nIn the rapidly evolving landscape of Generative AI, ensuring safety, reliability, and performance of LLM applications is paramount. Moonshot addresses this critical need by providing a unified platform for:\n- \u003cb\u003eBenchmark Tests:\u003c/b\u003e Systematically test LLM Apps or LLMs across critical trust \u0026 safety risks using a wide array of open-source benchmark dataset and metrics, including guided workflows to implement \u003cb\u003eIMDA's Starter Kit for LLM-based App Testing\u003c/b\u003e.\n- \u003cb\u003eRed Team Attacks:\u003c/b\u003e Proactively identify vulnerabilities and potential misuse scenarios in your LLM applications through streamlined adversarial prompting.\n\n\u003c/br\u003e\n\n## 🔑 Key Features\n\n- \u003cb\u003eUser-friendly Interfaces:\u003c/b\u003e Interact with Moonshot via an intuitive Web UI for visual insights, and an interactive Command Line Interface (CLI) for quick operations.\n- \u003cb\u003eComprehensive Benchmarking:\u003c/b\u003e\n  - [View list of available datasets available](https://aiverify-foundation.github.io/moonshot/resources/datasets/)\n  - Test for \u003cb\u003ePerformance\u003c/b\u003e (e.g., accuracy, BLEU)\n  - Ensure \u003cb\u003eTrust \u0026 Safety\u003c/b\u003e e.g., bias, toxicity, hallucination)\n  - Utilize built-in workflow to implement IMDA's Starter Kit for LLM-based App Testing. [View available pre-built Cookbooks](https://aiverify-foundation.github.io/moonshot/resources/cookbooks/)\n- \u003cb\u003ePowerful Red-Teaming:\u003c/b\u003e\n  - [View list of available attack modules](https://aiverify-foundation.github.io/moonshot/resources/attack_modules/)\n  - Simplify adversarial prompt generation using algorithmic strategies or generative LLM to uncover potential misuse.\n  - Leverage prompt templates, context strategies, and automated attack modules.\n- \u003cb\u003eCustomizable Recipes:\u003c/b\u003e Build your own benchmark tests with custom datasets (input-target pairs), prompt templates (optional), evaluation metric, and grading scales. [View available pre-built Recipes](https://aiverify-foundation.github.io/moonshot/resources/recipes/)\n- \u003cb\u003eInsightful Reporting:\u003c/b\u003e Use our HTML reports with interactive charts for clear visualization of test results, and download detailed raw JSON results for deeper programmatic analysis.\n- \u003cb\u003eExtensible \u0026 Modular:\u003c/b\u003e Designed for easy extension and integration with new LLM applications, benchmarks, and attack techniques.\n\n\u003c/br\u003e\n\n# Getting Started\n\nMoonshot can be used through several interfaces:\n- User-friendly Web UI - [Web UI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)\n- Interactive Command Line Interface - [CLI User Guide](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)\n- Seamless Integration into your MLOps workflow via Moonshot Library APIs or Moonshot Web APIs - [Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook), [Web API Docs](https://aiverify-foundation.github.io/moonshot/api_reference/web_api_swagger/)\n\n\u003c/br\u003e\n\n## 💻 Let's Go!\n\nThis section will guide you through getting Moonshot up and running.\n\n\u003c/br\u003e\n\n### ✅ Prerequisites\n1. \u003cb\u003ePython:\u003c/b\u003e [Version 3.11](https://www.python.org/downloads/) is required. \n\n2. \u003cb\u003eGit Version Control:\u003c/b\u003e [Git](https://github.com/git-guides/install-git) is essential for cloning the repository.\n\n3. \u003cb\u003e(Optional) Virtual Environment:\u003c/b\u003e Highly recommended to manage dependencies.\n\n    ```\n    # Create a virtual environment\n    python -m venv venv\n\n    # Activate the virtual environment\n    source venv/bin/activate\n    ```\n4. If you plan to install our Web UI, you will also need [Node.js version 20.11.1 LTS](https://nodejs.org/en/blog/release/v20.11.1) and above\n\u003c/br\u003e\n\n### ⬇️ Installation\n\nYou can install Moonshot in various ways depending on your needs\n\n\u003cb\u003e1. Using `pip` (Recommended for most users)\u003c/b\u003e\n\n```\n# Install Project Moonshot's Python Library, which includes Moonshot's full functionalities (Library APIs, CLI and Web APIs)\npip install \"aiverify-moonshot[all]\"\n\n# Clone and install test assets and Web UI\npython -m moonshot -i moonshot-data -i moonshot-ui\n```\n⚠️ You will need to have test assets from [moonshot-data](https://github.com/aiverify-foundation/moonshot-data) before you can run any tests.\n\n🖼️ If you plan to install our Web UI, you will also need [moonshot-ui](https://github.com/aiverify-foundation/moonshot-ui)\n\nCheck out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for more details.\n\n\u003c/br\u003e\n\n\u003cb\u003e2. From Source Code (For developers and contributors)\u003c/b\u003e\n\n```\n# To install from source code (Full functionalities)\ngit clone git@github.com:aiverify-foundation/moonshot.git\ncd moonshot\npip install -r requirements.txt\n```\nIf you have installation issues, please take a look at the [Troubleshooting Guide](https://aiverify-foundation.github.io/moonshot/faq/).\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eOther installation options\u003c/b\u003e\u003c/summary\u003e\nHere's a summary of other installation commands available:\n\n```\n# To install Moonshot library APIs only\npip install aiverify-moonshot\n\n# To install Moonshot library APIs and Web APIs only\npip install \"aiverify-moonshot[web-api]\"\n\n# To install Moonshot library APIs and CLI only\npip install \"aiverify-moonshot[cli]\"\n```\nCheck out our [Installation Guide](https://aiverify-foundation.github.io/moonshot/getting_started/quick_install/) for more details.\n\u003c/details\u003e\n\n\u003c/br\u003e\n\n### 🏃‍♀️ Run Moonshot\n\n#### Running the Web UI\n```\npython -m moonshot web\n```\nOpen [http://localhost:3000/](http://localhost:3000/) in a browser and you should see this homepage:\n\n![Moonshot UI Home](https://github.com/aiverify-foundation/moonshot/raw/main/misc/ui-homepage.png)\n\nRefer to this [guide](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/moonshot_interface/homepage/) to discover the rich features available in Moonshot Web UI\n\n\u003c/br\u003e\n\n#### Running the Interactive CLI\n```\npython -m moonshot cli interactive\n```\n![Moonshot cli](https://github.com/aiverify-foundation/moonshot/raw/main/misc/cli-homepage.png)\n\nRefer to this [Command List](https://aiverify-foundation.github.io/moonshot/user_guide/cli/cli_command_list/) to discover the list of CLI commands for Moonshot\n\n\u003c/br\u003e\u003c/br\u003e\n\n# 📚 Documentation \u0026 User Guides\n\nFor detailed information on configuring, using, and extending Moonshot, please refer to our comprehensive documentation:\n\n#### Guides for Moonshot Web UI\n- [Getting Started with Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/user_guide/web_ui/web_ui_guide/)\n- [Creating Your Custom Cookbook via Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/create_cookbook/)\n- [Creating Your Custom Connector Endpoint via Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/create_endpoint/)\n- [Running Benchmark Test on Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/getting_started/first_test/)\n- [Running Red Teaming on Moonshot Web UI](https://aiverify-foundation.github.io/moonshot/tutorial/web-ui/redteam/)\n\n#### Guides for Moonshot Interactive CLI\n- [Getting Started with Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/user_guide/cli/connecting_endpoints/)\n- [Creating Your Custom Benchmark Tests for Your RAG Apps via Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/create_benchmark_tests/)\n- [Creating Your Custom Connector Endpoint via Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/create_endpoint/)\n- [Running Benchmark Test on Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/run_benchmark_tests/)\n- [Running Red Teaming on Moonshot Interactive CLI](https://aiverify-foundation.github.io/moonshot/tutorial/cli/run_red_teaming/)\n\n#### For Users Interested in trying Out Moonshot using Jupyter Notebook\n- [Moonshot Library Python Notebook Examples](https://github.com/aiverify-foundation/moonshot/tree/main/examples/jupyter-notebook)\n\n#### \n\n\u003c/br\u003e\n\n## 🤝 Contribution\n\nMoonshot is an open-source project, and we welcome contributions from the community! Whether fixing a bug, adding a new feature, improving documentation, or suggesting an enhancement, your efforts are highly valued.\n\nPlease refer to our [Contributor Guide](https://aiverify-foundation.github.io/moonshot/contributing/) for details on how to get started.\n\n\u003c/br\u003e\n\n## ✨ Project Status\n\nMoonshot is currently in beta. We are actively developing new features, improving existing ones, and enhancing stability. We encourage you to try it out and provide feedback!\n\n\u003c/br\u003e\n\n## 📜 License\n\nMoonshot is released under the [Apache Software License 2.0](https://www.apache.org/licenses/LICENSE-2.0.txt)\n\n\u003c/br\u003e\n","funding_links":[],"categories":["Tools","AI Safety Tools","LLM Development and Optimization"],"sub_categories":["LLM Evaluation","AI Security Tools","LLM Testing and Evaluation"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiverify-foundation%2Fmoonshot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faiverify-foundation%2Fmoonshot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faiverify-foundation%2Fmoonshot/lists"}