{"id":31936969,"url":"https://github.com/future-agi/agent-opt","last_synced_at":"2025-10-14T07:54:55.659Z","repository":{"id":318521757,"uuid":"1062359288","full_name":"future-agi/agent-opt","owner":"future-agi","description":"Agent Optimization Library","archived":false,"fork":false,"pushed_at":"2025-10-07T16:32:21.000Z","size":1158,"stargazers_count":13,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-07T18:34:52.731Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/future-agi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-23T06:43:18.000Z","updated_at":"2025-10-07T17:18:19.000Z","dependencies_parsed_at":"2025-10-07T18:35:02.662Z","dependency_job_id":null,"html_url":"https://github.com/future-agi/agent-opt","commit_stats":null,"previous_names":["future-agi/agent-opt"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/future-agi/agent-opt","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/future-agi%2Fagent-opt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/future-agi%2Fagent-opt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/future-agi%2Fagent-opt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/future-agi%2Fagent-opt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/future-agi","download_url":"https://codeload.github.com/future-agi/agent-opt/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/future-agi%2Fagent-opt/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018186,"owners_count":26086307,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T07:54:54.824Z","updated_at":"2025-10-14T07:54:55.653Z","avatar_url":"https://github.com/future-agi.png","language":"Python","funding_links":[],"categories":["Optimizations","LLM Optimization","Frameworks"],"sub_categories":["Profiling"],"readme":"\u003cdiv align=\"center\"\u003e\n\n# 🎯 Agent Opt\n\n**Automated Workflow Optimization with State-of-the-Art Algorithms**  \nBuilt by [Future AGI](https://futureagi.com) | [Docs](https://docs.futureagi.com) | [Platform](https://app.futureagi.com)\n\n---\n\n### 🚀 Try it Now\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1XjOUlEwUk-S0nl8dBb16iZHgr-XpXsnp?usp=sharing)\n\n\u003c/div\u003e\n\n---\n\n## 🚀 Overview\n\n**agent-opt** is a comprehensive Python SDK for optimizing prompts through iterative refinement. Powered by state-of-the-art optimization algorithms and flexible evaluation strategies from our [`ai-evaluation`](https://github.com/future-agi/ai-evaluation) library, agent-opt helps you discover the best prompts for your LLM workflows automatically.\n\n- 🧬 **Smart Optimization**: 6 proven algorithms from random search to genetic evolution\n- 📊 **Flexible Evaluation**: Heuristic metrics, LLM-as-a-judge, and platform integration\n- ⚡ **Easy Integration**: Works with any LLM through LiteLLM\n- 🔧 **Extensible Design**: Clean abstractions for custom optimizers and evaluators\n\n---\n\n## 🎨 Features\n\n### 🧬 Multiple Optimization Algorithms\n\nChoose from 6 battle-tested optimization strategies:\n\n| Algorithm           | Best For                    | Key Feature                                   |\n| ------------------- | --------------------------- | --------------------------------------------- |\n| **Random Search**   | Quick baselines             | Simple random variations                      |\n| **Bayesian Search** | Few-shot optimization       | Intelligent hyperparameter tuning with Optuna |\n| **ProTeGi**         | Gradient-based refinement   | Textual gradients for iterative improvement   |\n| **Meta-Prompt**     | Teacher-driven optimization | Uses powerful models to analyze and rewrite   |\n| **PromptWizard**    | Multi-stage refinement      | Mutation, critique, and refinement pipeline   |\n| **GEPA**            | Complex solution spaces     | Genetic Pareto evolutionary optimization      |\n\n### 📊 Flexible Evaluation\n\nAll evaluation backends powered by FutureAGI's [`ai-evaluation`](https://github.com/future-agi/ai-evaluation) library:\n\n- ✅ **Heuristic Metrics**: BLEU, ROUGE, embedding similarity, and more\n- 🧠 **LLM-as-a-Judge**: Custom criteria with any LLM provider\n- 🎯 **FutureAGI Platform**: 50+ pre-built evaluation templates\n- 🔌 **Custom Metrics**: Build your own evaluation logic\n\n### 🔧 Easy Integration\n\n- Works with **any LLM** through LiteLLM (OpenAI, Anthropic, Google, etc.)\n- Simple Python API with sensible defaults\n- Comprehensive logging and progress tracking\n- Clean separation of concerns\n\n---\n\n## 📦 Installation\n\n```bash\npip install agent-opt\n```\n\n**Requirements:**\n\n- Python \u003e= 3.10\n- ai-evaluation \u003e= 0.1.9\n- gepa \u003e= 0.0.17\n- litellm \u003e= 1.35.2\n- optuna \u003e= 3.6.1\n\n---\n\n## 🧑‍💻 Quick Start\n\n```python\nfrom fi.opt.generators import LiteLLMGenerator\nfrom fi.opt.optimizers import BayesianSearchOptimizer\nfrom fi.opt.datamappers import BasicDataMapper\nfrom fi.opt.base.evaluator import Evaluator\nfrom fi.evals.metrics import BLEUScore\n\n# 1. Set up your dataset\ndataset = [\n    {\n        \"context\": \"Paris is the capital of France\",\n        \"question\": \"What is the capital of France?\",\n        \"answer\": \"Paris\"\n    },\n    # ... more examples\n]\n\n# 2. Configure the evaluator\nmetric = BLEUScore()\nevaluator = Evaluator(metric)\n\n# 3. Set up data mapping\ndata_mapper = BasicDataMapper(\n    key_map={\n        \"response\": \"generated_output\",\n        \"expected_response\": \"answer\"\n    }\n)\n\n# 4. Choose and configure an optimizer\noptimizer = BayesianSearchOptimizer(\n    inference_model_name=\"gpt-4o-mini\",\n    teacher_model_name=\"gpt-4o\",\n    n_trials=10\n)\n\n# 5. Run optimization\ninitial_prompt = \"Given the context: {context}, answer the question: {question}\"\nresult = optimizer.optimize(\n    evaluator=evaluator,\n    data_mapper=data_mapper,\n    dataset=dataset,\n    initial_prompts=[initial_prompt]\n)\n\n# 6. Get the best prompt\nprint(f\"Best Score: {result.final_score:.4f}\")\nprint(f\"Best Prompt: {result.best_generator.get_prompt_template()}\")\n```\n\n---\n\n## 🏗️ Core Components\n\n### 🤖 Generators\n\nGenerators execute prompts and return responses. Use `LiteLLMGenerator` for seamless integration with any LLM provider.\n\n```python\nfrom fi.opt.generators import LiteLLMGenerator\n\ngenerator = LiteLLMGenerator(\n    model=\"gpt-4o-mini\",\n    prompt_template=\"Summarize this text: {text}\"\n)\n```\n\n---\n\n### 📊 Evaluators\n\nEvaluators score generated outputs using various strategies:\n\n#### **Heuristic Metrics**\n\n```python\nfrom fi.opt.base.evaluator import Evaluator\nfrom fi.evals.metrics import BLEUScore\n\nevaluator = Evaluator(metric=BLEUScore())\n```\n\n#### **LLM-as-a-Judge**\n\n```python\nfrom fi.evals.llm import LiteLLMProvider\nfrom fi.evals.metrics import CustomLLMJudge\n\n# LLM provider used by the judge\nprovider = LiteLLMProvider()\n\n# Create custom LLM judge metric\ncorrectness_judge_config = {\n    \"name\": \"correctness_judge\",\n    \"grading_criteria\": '''You are evaluating an AI's answer to a question.\n    The score must be 1.0 if the 'response' is semantically equivalent to the\n    'expected_response' (the ground truth). The score should be 0.0 if incorrect.\n    Partial credit is acceptable.'''\n}\n\n# Instantiate the judge and pass to evaluator\ncorrectness_judge = CustomLLMJudge(\n    provider=provider,\n    config=correctness_judge_config,\n    model=\"gemini/gemini-2.5-flash\",\n    temperature=0.4\n)\nevaluator = Evaluator(metric=correctness_judge)\n```\n\n#### **FutureAGI Platform**\n\nAccess 50+ pre-built evaluation templates:\n\n```python\nevaluator = Evaluator(\n    eval_template=\"summary_quality\",\n    eval_model_name=\"turing_flash\",\n    fi_api_key=\"your_key\",\n    fi_secret_key=\"your_secret\"\n)\n```\n\n---\n\n### 🗺️ Data Mappers\n\nData mappers transform your data into the format expected by evaluators:\n\n```python\nfrom fi.opt.datamappers import BasicDataMapper\n\nmapper = BasicDataMapper(\n    key_map={\n        \"output\": \"generated_output\",  # Maps generator output\n        \"input\": \"question\",            # Maps from dataset\n        \"ground_truth\": \"answer\"        # Maps from dataset\n    }\n)\n```\n\n---\n\n## ⚙️ Optimization Algorithms\n\n### 🔍 Bayesian Search\n\nUses Optuna for intelligent hyperparameter optimization of few-shot example selection.\n\n```python\nfrom fi.opt.optimizers import BayesianSearchOptimizer\n\noptimizer = BayesianSearchOptimizer(\n    min_examples=2,\n    max_examples=8,\n    n_trials=20,\n    inference_model_name=\"gpt-4o-mini\",\n    teacher_model_name=\"gpt-4o\"\n)\n```\n\n**Best for:** Few-shot prompt optimization with automatic example selection\n\n---\n\n### 🎯 ProTeGi\n\nGradient-based prompt optimization that iteratively refines prompts through error analysis.\n\n```python\nfrom fi.opt.optimizers import ProTeGi\nfrom fi.opt.generators import LiteLLMGenerator\n\nteacher = LiteLLMGenerator(\n    model=\"gpt-4o\",\n    prompt_template=\"{prompt}\"\n)\noptimizer = ProTeGi(\n    teacher_generator=teacher,\n    num_gradients=4,\n    beam_size=4\n)\n```\n\n**Best for:** Iterative refinement with textual gradients\n\n---\n\n### 🧠 Meta-Prompt\n\nUses a powerful teacher model to analyze performance and rewrite prompts.\n\n```python\nfrom fi.opt.optimizers import MetaPromptOptimizer\n\noptimizer = MetaPromptOptimizer(\n    teacher_generator=teacher,\n    num_rounds=5\n)\n```\n\n**Best for:** Leveraging powerful models for prompt refinement\n\n---\n\n### 🧬 GEPA (Genetic Pareto)\n\nEvolutionary optimization using the GEPA library for complex solution spaces.\n\n```python\nfrom fi.opt.optimizers import GEPAOptimizer\n\noptimizer = GEPAOptimizer(\n    reflection_model=\"gpt-5\",\n    generator_model=\"gpt-4o-mini\"\n)\n```\n\n**Best for:** Multi-objective optimization with genetic algorithms\n\n---\n\n### 🪄 PromptWizard\n\nMulti-stage optimization with mutation, critique, and refinement.\n\n```python\nfrom fi.opt.optimizers import PromptWizardOptimizer\n\noptimizer = PromptWizardOptimizer(\n    teacher_generator=teacher,\n    mutate_rounds=3,\n    refine_iterations=2\n)\n```\n\n**Best for:** Comprehensive multi-phase optimization pipeline\n\n---\n\n### 🎲 Random Search\n\nSimple baseline that tries random prompt variations.\n\n```python\nfrom fi.opt.optimizers import RandomSearchOptimizer\n\noptimizer = RandomSearchOptimizer(\n    generator=generator,\n    teacher_model=\"gpt-4o\",\n    num_variations=5\n)\n```\n\n**Best for:** Quick baselines and sanity checks\n\n---\n\n## 🔧 Advanced Usage\n\n### 🎨 Custom Evaluation Metrics\n\nCreate custom heuristic metrics by extending `BaseMetric`:\n\n```python\nfrom fi.evals.metrics.base_metric import BaseMetric\n\nclass CustomMetric(BaseMetric):\n    @property\n    def metric_name(self):\n        return \"your_custom_metric\"\n\n    def compute_one(self, inputs):\n        # Your evaluation logic here\n        score = your_scoring_logic(inputs)\n        return score\n```\n\n---\n\n### 📝 Logging Configuration\n\n```python\nfrom fi.opt.utils import setup_logging\nimport logging\n\nsetup_logging(\n    level=logging.INFO,\n    log_to_console=True,\n    log_to_file=True,\n    log_file=\"optimization.log\"\n)\n```\n\n---\n\n### 🏗️ Custom Prompt Builders\n\nFor complex prompt construction:\n\n```python\ndef custom_prompt_builder(base_prompt: str, few_shot_examples: List[str]) -\u003e str:\n    examples = \"\\n\\n\".join(few_shot_examples)\n    return f\"{base_prompt}\\n\\nExamples:\\n{examples}\"\n\noptimizer = BayesianSearchOptimizer(\n    prompt_builder=custom_prompt_builder\n)\n```\n\n---\n\n## 🔑 Environment Setup\n\n### API Keys\n\nSet up your API keys for LLM providers and FutureAGI:\n\n```bash\nexport OPENAI_API_KEY=\"your_openai_key\"\nexport GEMINI_API_KEY=\"your_gemini_key\"  # If using Gemini\nexport FI_API_KEY=\"your_futureagi_key\"\nexport FI_SECRET_KEY=\"your_futureagi_secret\"\n```\n\nOr use a `.env` file:\n\n```\nOPENAI_API_KEY=your_openai_key\nFI_API_KEY=your_futureagi_key\nFI_SECRET_KEY=your_futureagi_secret\n```\n\n---\n\n## 📚 Examples \u0026 Tutorials\n\n🎯 **Complete Example**: Check out `examples/FutureAGI_Agent_Optimizer.ipynb` for a comprehensive walkthrough!\n\n---\n\n## 📁 Project Structure\n\n```\nsrc/fi/opt/\n├── base/              # Abstract base classes\n├── datamappers/       # Data transformation utilities\n├── generators/        # LLM generator implementations\n├── optimizers/        # Optimization algorithms\n├── utils/             # Helper utilities\n└── types.py           # Type definitions\n```\n\n---\n\n## 🔌 Related Projects\n\n- 🧪 [ai-evaluation](https://github.com/future-agi/ai-evaluation): Comprehensive LLM evaluation framework with 50+ metrics\n- 🚦 [traceAI](https://github.com/future-agi/traceAI): Add tracing \u0026 observability to your optimized workflows\n\n---\n\n## 🗺️ Roadmap\n\n- [x] **Core Optimization Algorithms**\n- [x] **ai-evaluation Integration**\n- [x] **LiteLLM Support**\n- [x] **Bayesian Optimization**\n- [x] **ProTeGi \u0026 Meta-Prompt**\n- [x] **GEPA Integration**\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! To report issues, suggest features, or contribute improvements:\n\n1. Open a [GitHub issue](https://github.com/future-agi/agent-opt/issues)\n2. Submit a pull request\n3. Join our community discussions\n\n---\n\n## 💬 Support\n\nFor questions and support:\n\n📧 **Email**: support@futureagi.com  \n📚 **Documentation**: [docs.futureagi.com](https://docs.futureagi.com)  \n🌐 **Platform**: [app.futureagi.com](https://app.futureagi.com)\n\n---\n\n\u003cdiv align=\"center\"\u003e\n\nBuilt with ❤️ by [Future AGI](https://futureagi.com)\n\n\u003c/div\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffuture-agi%2Fagent-opt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffuture-agi%2Fagent-opt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffuture-agi%2Fagent-opt/lists"}