https://github.com/thisisartium/continuous-alignment-testing
CAT AI - very much work in progress
https://github.com/thisisartium/continuous-alignment-testing
Last synced: 5 months ago
JSON representation
CAT AI - very much work in progress
- Host: GitHub
- URL: https://github.com/thisisartium/continuous-alignment-testing
- Owner: thisisartium
- License: mit
- Created: 2025-02-22T00:31:58.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2025-04-07T18:40:17.000Z (8 months ago)
- Last Synced: 2025-04-07T19:40:25.808Z (8 months ago)
- Language: Jupyter Notebook
- Homepage:
- Size: 6.25 MB
- Stars: 15
- Watchers: 9
- Forks: 9
- Open Issues: 14
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-continuous-ai - Continuous alignment testing - A framework for implementing Continuous Alignment Testing (CAT) for LLM-based applications. (Categories / Continuous AI Engineering)
README
[![Python Tests][python-tests-badge]][python-tests-link]
[![CAT][cat-badge]][cat-link]
# Continuous Alignment Testing (CAT) Framework
A framework for implementing Continuous Alignment Testing (CAT) for LLM-based applications.
## Overview
CAT provides the infrastructure to:
- Run and track automated tests against an AI application.
- Store and analyze AI test results over iterations.
- Monitor reliability changes in an AI application as prompts, models, code, and data evolve.
- Integrate AI tests into the CI pipeline.
- Build the AI application as an API service using Python and use any language to invoke it.
- Ensure AI applications are reliable while maintaining creativity.
- Iterate on prompts and code while measuring improvements.
## Example Apps
- [Team Recommender - AI assembles team of developers](examples/team_recommender/readme.md)
- you next idea?
## Documentation
[Github Pages][github-pages-index]
Edit Pages in [docs folder](docs) to automatically deploy: [![github-pages-badge]][github-pages-deploy-link]
## Wiki
[Wiki Pages][wiki]
[python-tests-badge]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/python-tests.yml/badge.svg
[python-tests-link]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/python-tests.yml
[cat-badge]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/cat-test-examples.yml/badge.svg
[cat-link]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/cat-test-examples.yml
[wiki]: https://github.com/thisisartium/continuous-alignment-testing/wiki
[github-pages-index]: https://thisisartium.github.io/continuous-alignment-testing/
[github-pages-source]: https://github.com/thisisartium/continuous-alignment-testing/tree/main/docs
[github-pages-badge]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/pages/pages-build-deployment/badge.svg
[github-pages-deploy-link]: https://github.com/thisisartium/continuous-alignment-testing/actions/workflows/pages/pages-build-deployment