https://github.com/Agenta-AI/agenta

The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
https://github.com/Agenta-AI/agenta

human-annotation langchain large-language-models llama-index llm llm-evaluation llm-framework llm-tools llmops llms prompt-engineering prompt-management prompt-toolkit rag rag-evaluation

Last synced: about 1 month ago
JSON representation

The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.

Host: GitHub
URL: https://github.com/Agenta-AI/agenta
Owner: Agenta-AI
License: mit
Created: 2023-04-26T09:54:28.000Z (almost 2 years ago)
Default Branch: main
Last Pushed: 2024-10-29T11:55:12.000Z (6 months ago)
Last Synced: 2024-10-29T12:44:36.611Z (6 months ago)
Topics: human-annotation, langchain, large-language-models, llama-index, llm, llm-evaluation, llm-framework, llm-tools, llmops, llms, prompt-engineering, prompt-management, prompt-toolkit, rag, rag-evaluation
Language: Python
Homepage: http://www.agenta.ai
Size: 130 MB
Stars: 1,242
Watchers: 20
Forks: 184
Open Issues: 57
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY.md

Awesome Lists containing this project

awesome-llmops - agenta - AI/agenta.svg?style=flat-square) | (LLMOps / Observability)
awesome-llm-eval - agenta - AI/agenta.svg?style=social) | 用于构建强大LLM应用的LLMOps平台。轻松尝试和评估不同提示、模型和工作流，以构建稳健的应用程序。 | (LLMOps / Popular-LLM)
jimsghstars - Agenta-AI/agenta - The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place. (Python)
awesome - Agenta-AI/agenta - The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place. (Python)
awesome-production-machine-learning - Agenta - AI/agenta.svg?style=social) - Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and (LLM observability and tracing). (Deployment and Serving)
awesome-hacking-lists - Agenta-AI/agenta - The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place. (Python)
awesome-generative-ai-data-scientist - Agenta - source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place. | [Documentation](https://docs.agenta.ai/) | (LLMOps)

README

        


  

      

        

        

        

      

  



  

    Documentation |

    Website |

    Slack

  



    The Open source LLMOps Platform 

  Prompt playground, prompt management, evaluation, and observability





  

  

    

  

  

    

  

  

  

    

  

    

  





    

        

    

    

        

    

    

        

    



  




  

      

        

        

        

      

  






  


      


        

          

            

          

        

    








---



  Documentation •

  Changelog •

  Website •

  Agenta Cloud



---

## What is Agenta?

Agenta is a platform for building production-grade LLM applications. It helps **engineering and product teams** create reliable LLM apps faster.

Agenta provides end-to-end tools for the entire LLMOps workflow:  building (**LLM playground**, **evaluation**), deploying (**prompt and configuration management**), and monitoring (**LLM observability and tracing**).

## Features

- **Prompt Playground**: Experiment, iterate on prompts, and compare outputs from over 50 LLM models side by side ([docs](https://docs.agenta.ai/prompt-management/using-the-playground?utm_source=github&utm_medium=referral&utm_campaign=readme))

- **Custom Workflows**: Build a playground for any custom LLM workflow, such as RAG or agents. Enable all the team to easily iterate on its parameters and evaluate it from the web UI.

- **LLM evaluation**: Run evaluation suite from the webUI using predefined evaluators like LLM-as-a-judge, RAG evaluators, or custom code evaluators. ([docs](https://docs.agenta.ai/evaluation/overview?utm_source=github&utm_medium=referral&utm_campaign=readme))

- **Human evaluation**: Collaborate with subject matter experts for human annotation evaluation, including A/B testing and annotating golden test sets.

- **Prompt Management**: Version your prompts and manage them across different environments ([docs](https://docs.agenta.ai/prompt-management/overview?utm_source=github&utm_medium=referral&utm_campaign=readme), [quick start](https://docs.agenta.ai/prompt-management/quick-start?utm_source=github&utm_medium=referral&utm_campaign=readme))

- **LLM Tracing**: Observe and debug your apps with integrations to most providers and frameworks ([docs](https://docs.agenta.ai/observability/overview?utm_source=github&utm_medium=referral&utm_campaign=readme), [quick start](https://docs.agenta.ai/observability/quickstart?utm_source=github&utm_medium=referral&utm_campaign=readme))

- **LLM Monitoring**: Track cost and latency and compare different deployments.

  

## Getting Started 

### Agenta Cloud:

The easiest way to get started is through Agenta Cloud. It is free to signup, and comes with a generous free-tier.

  

      

        

        

        

      

  

  

   

### Self-hosting Agenta

1. Clone Agenta:

```bash

git clone https://github.com/Agenta-AI/agenta && cd agenta

```

2. Edit `hosting/docker-compose/oss/.env.oss.gh` and add your LLM provider API keys.

3. Start Agenta services:

```bash

docker compose -f hosting/docker-compose/oss/docker-compose.gh.yml --env-file hosting/docker-compose/oss/.env.oss.gh --profile with-web up -d

```

4. Access Agenta at `http://localhost`.

For deploying on a remote host, or using different ports refers to our [self-hosting](https://docs.agenta.ai/self-host/host-locally?utm_source=github&utm_medium=referral&utm_campaign=readme) and [remote deployment documentation](https://docs.agenta.ai/self-host/host-remotely?utm_source=github&utm_medium=referral&utm_campaign=readme).

## Disabling Anonymized Tracking

By default, Agenta automatically reports anonymized basic usage statistics. This helps us understand how Agenta is used and track its overall usage and growth. This data does not include any sensitive information. To disable anonymized telemetry set `TELEMETRY_ENABLED` to `false` in your `.env` file.

## Contributing

We warmly welcome contributions to Agenta. Feel free to submit issues, fork the repository, and send pull requests.

We are usually hanging in our Slack. Feel free to [join our Slack and ask us anything](https://join.slack.com/t/agenta-hq/shared_invite/zt-2yewk6o2b-DmhyA4h_lkKwecDtIsj1AQ)

Check out our [Contributing Guide](https://docs.agenta.ai/misc/contributing/getting-started?utm_source=github&utm_medium=referral&utm_campaign=readme) for more information.

### Contributors ✨

[![All Contributors](https://img.shields.io/badge/all_contributors-49-orange.svg?style=flat-square)](#contributors-)

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

  

    

      
_{Sameh Methnani}
💻 📖

      
_{Suad Suljovic}
💻 🎨 🧑‍🏫 👀

      
_burtenshaw
💻

      
_Abram
💻 📖

      
_{Israel Abebe}
🐛 🎨 💻

      
_{Master X}
💻

      
_corinthian
💻 🎨

    

    

      
_{Pavle Janjusevic}
🚇

      
_{Kaosi Ezealigo}
🐛 💻

      
_{Alberto Nunes}
🐛

      
_{Maaz Bin Khawar}
💻 👀 🧑‍🏫

      
_{Nehemiah Onyekachukwu Emmanuel}
💻 💡 📖

      
_{Philip Okiokio}
📖

      
_{Abhinav Pandey}
💻

    

    

      
_{Ramchandra Warang}
💻 🐛

      
_{Biswarghya Biswas}
💻

      
_{Uddeepta Raaj Kashyap}
💻

      
_{Nayeem Abdullah}
💻

      
_{Kang Suhyun}
💻

      
_Yoon
💻

      
_{Kirthi Bagrecha Jain}
💻

    

    

      
_Navdeep
💻

      
_{Rhythm Sharma}
💻

      
_{Osinachi Chukwujama}
💻

      
_莫尔索
📖

      
_{Agunbiade Adedeji}
💻

      
_{Emmanuel Oloyede}
💻 📖

      
_{Dhaneshwarguiyan}
💻

    

    

      
_{Priyanshu Prajapati}
📖

      
_Raviteja
💻

      
_Arijit
💻

      
_Yachika9925
📖

      
_Aldrin
⚠️

      
_{seungduk.kim.2304}
💻

      
_{Andrei Dragomir}
💻

    

    

      
_diego
💻

      
_brockWith
💻

      
_{Dennis Zelada}
💻

      
_{Romain Brucker}
💻

      
_{Heon Heo}
💻

      
_{Drew Reisner}
💻

      
_{Ikko Eltociear Ashimine}
📖

    

    

      
_{Vishal Vanpariya}
💻

      
_{Youcef Boumar}
📖

      
_LucasTrg
💻 🐛

      
_{Ashraf Chowdury}
🐛 💻

      
_jp-agenta
💻 🐛

      
_{Mr Unhappy}
🐛 🚇

      
_{Moreno Bonaventura}
🐛

    

  

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind are welcome!