https://github.com/ksm26/automated-testing-for-llmops
Create a continuous integration (CI) workflow for testing LLMs applications in an effective way.
https://github.com/ksm26/automated-testing-for-llmops
automated-testing circleci continuous-integration continuous-integration-workflow data-evaluation large-language-models llmops llms model-based-evaluations role-based-evaluations software-testing
Last synced: about 1 year ago
JSON representation
Create a continuous integration (CI) workflow for testing LLMs applications in an effective way.
- Host: GitHub
- URL: https://github.com/ksm26/automated-testing-for-llmops
- Owner: ksm26
- Created: 2024-02-13T13:03:28.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-02-14T10:31:53.000Z (over 2 years ago)
- Last Synced: 2024-02-14T14:58:50.571Z (over 2 years ago)
- Topics: automated-testing, circleci, continuous-integration, continuous-integration-workflow, data-evaluation, large-language-models, llmops, llms, model-based-evaluations, role-based-evaluations, software-testing
- Language: Jupyter Notebook
- Homepage: https://www.deeplearning.ai/short-courses/automated-testing-llmops/
- Size: 1.15 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# 🚀 [Automated Testing for LLMOps](https://www.deeplearning.ai/short-courses/automated-testing-llmops/)
💻 Welcome to the "Automated Testing for LLMOps" course! Instructed by Rob Zuber, CTO at CircleCI, this course will teach you how to create a continuous integration (CI) workflow for evaluating your Large Language Model (LLM) applications at every change, enabling faster, safer, and more efficient application development.
**Course Website**: 📚[deeplearning.ai](https://www.deeplearning.ai/short-courses/automated-testing-llmops/)
## Course Summary
In this course, you will learn the importance of systematic testing in LLM application development and how to implement a continuous integration workflow to catch issues early. Here's what you can expect to learn and experience:
1. 📋 **Robust LLM Evaluations**: Write robust evaluations covering common problems like hallucinations, data drift, and harmful or offensive output.
2. ⚙️ **Continuous Integration Workflow**: Build a CI workflow to automatically evaluate every change to your LLM application.
3. 🔄 **Orchestrating CI Workflow**: Orchestrate your CI workflow to run specific evaluations at different stages of development.
## Key Points
- 🧪 Learn how LLM-based testing differs from traditional software testing and implement rules-based testing to assess your LLM application.
- 📝 Build model-graded evaluations to test your LLM application using an evaluation LLM.
- 🔄 Automate your evaluations (rules-based and model-graded) using continuous integration tools from CircleCI.
## About the Instructor
🌟 **Rob Zuber** is the CTO at CircleCI, bringing extensive expertise in software development and continuous integration to guide you through automating testing for LLMOps.
🔗 To enroll in the course or for further information, visit [deeplearning.ai](https://www.deeplearning.ai/short-courses/).