{"id":19888631,"url":"https://github.com/WecoAI/aideml","last_synced_at":"2025-05-02T17:32:21.495Z","repository":{"id":231561544,"uuid":"781524183","full_name":"WecoAI/aideml","owner":"WecoAI","description":"AIDE: the state-of-the-art machine learning engineer agent, generating machine learning solution code from natural language descriptions.","archived":false,"fork":false,"pushed_at":"2024-11-03T14:41:44.000Z","size":331,"stargazers_count":538,"open_issues_count":4,"forks_count":51,"subscribers_count":19,"default_branch":"main","last_synced_at":"2024-11-03T15:28:37.029Z","etag":null,"topics":["ai","data-science","llm","machine-learning"],"latest_commit_sha":null,"homepage":"https://www.weco.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/WecoAI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-03T14:45:49.000Z","updated_at":"2024-11-03T14:41:48.000Z","dependencies_parsed_at":"2024-11-03T15:23:42.703Z","dependency_job_id":"b2498586-be93-4cd9-936c-f5910b8acd74","html_url":"https://github.com/WecoAI/aideml","commit_stats":null,"previous_names":["wecoai/aideml"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WecoAI%2Faideml","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WecoAI%2Faideml/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WecoAI%2Faideml/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/WecoAI%2Faideml/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/WecoAI","download_url":"https://codeload.github.com/WecoAI/aideml/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224324468,"owners_count":17292521,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","data-science","llm","machine-learning"],"created_at":"2024-11-12T18:07:45.838Z","updated_at":"2025-05-02T17:32:21.482Z","avatar_url":"https://github.com/WecoAI.png","language":"Python","funding_links":[],"categories":["AutoML","其他_机器学习与深度学习","🔬 Autonomous Research \u0026 Self-Improving Agents","🔧 Projects","Code AI","AutoML Agents","Model Development \u0026 Experiment Tracking","Orchestration","🤖 Research Agents \u0026 Autonomous Workflows","Research","🛠️ General-purpose descendants","Table of Open-Source AI Agents Projects"],"sub_categories":["General-Purpose Descendants","Research Assessment","Vector search","Evolutionary Framework","Autonomous Research Systems (2023-2025 Breakthroughs)"],"readme":"\u003ch1 align=\"center\"\u003eAIDE: The Machine Learning Engineer Agent\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n          📑 \u003ca href=\"https://arxiv.org/abs/2502.13138\"\u003ePaper\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp📝 \u003ca href=\"https://www.weco.ai/blog/technical-report\"\u003eBlog\u003c/a\u003e\u0026nbsp\u0026nbsp | \u0026nbsp\u0026nbsp🌐 \u003ca href=\"https://www.aide.ml\"\u003eProject\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://www.python.org/downloads/release/python-3100/\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3.10+-blue.svg\" alt=\"Python 3.10+\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/aideml/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/aideml?color=blue\" alt=\"PyPI\"\u003e\u003c/a\u003e\u0026ensp;\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"License: MIT\"\u003e\u003c/a\u003e\u0026ensp;\n  \u003ca href=\"https://twitter.com/WecoAI\"\u003e\u003cimg src=\"https://img.shields.io/twitter/follow/WecoAI?style=social\" alt=\"Twitter Follow\"\u003e\u003c/a\u003e\u0026ensp;\n  \u003ca href=\"https://pepy.tech/projects/aideml\"\u003e\u003cimg src=\"https://static.pepy.tech/badge/aideml\" alt=\"PyPI Downloads\"\u003e\u003c/a\u003e\u0026ensp;\n\u003c/p\u003e\n\nAIDE is an LLM agent that generates solutions for machine learning tasks just from natural language descriptions of the task.\n\nIn our own benchmark composed of over 60 Kaggle data science competitions, AIDE demonstrated impressive performance, surpassing 50% of Kaggle participants on average.\n\nOpenAI's [MLE-bench](https://arxiv.org/pdf/2410.07095), a benchmark composed of 75 Kaggle machine learning tasks, shows that AIDE achieved four times more medals compared to the runner-up agent architecture.\n\nMETR's [RE-Bench](https://arxiv.org/pdf/2411.15114) shows that AIDE is not only capable at machine learning tasks but generalizes to the AI R\u0026D tasks such as optimizing low level Triton kernels and finetuning GPT-2 for QA, even surpassing the performance of human experts.\n\nMore specifically, AIDE has the following features:\n\n1. **Instruct with Natural Language**: Describe your problem or additional requirements and expert insights, all in natural language.\n2. **Deliver Solution in Source Code**: AIDE will generate Python scripts for the **tested** machine learning pipeline. Enjoy full transparency, reproducibility, and the freedom to further improve the source code!\n3. **Iterative Optimization**: AIDE iteratively runs, debugs, evaluates, and improves the ML code, all by itself.\n4. **Visualization**: We also provide tools to visualize the solution tree produced by AIDE for a better understanding of its experimentation process. This gives you insights not only about what works but also what doesn't.\n\n# How to Use AIDE?\n\n## Running AIDE via the Web UI\n\n\nhttps://github.com/user-attachments/assets/1da42853-fe36-45e1-b6a2-852f88470af6\n\n\nWe have developed a user-friendly Web UI using Streamlit to make it even easier to interact with AIDE.\n\n### Prerequisites\n\nEnsure you have installed the development version of AIDE and its dependencies as described in the [Development](#development) section.\n\n### Running the Web UI\n\nNavigate to the `aide/webui` directory and run the Streamlit application:\n\n```bash\ncd aide/webui\nstreamlit run app.py\n```\n\nAlternatively, you can run it from the root directory:\n\n```bash\nstreamlit run aide/webui/app.py\n```\n\n### Using the Web UI\n\n1. **API Key Configuration**: In the sidebar, input your OpenAI API key or Anthropic API key and click \"Save API Keys\".\n\n2. **Input Data**:\n   - You can either **upload your dataset files** (`.csv`, `.txt`, `.json`, `.md`) using the \"Upload Data Files\" feature.\n   - Or click on \"Load Example Experiment\" to use the example house prices dataset.\n\n3. **Define Goal and Evaluation Criteria**:\n   - In the \"Goal\" text area, describe what you want the model to achieve (e.g., \"Predict the sales price for each house\").\n   - In the \"Evaluation Criteria\" text area, specify the evaluation metric (e.g., \"Use the RMSE metric between the logarithm of the predicted and observed values.\").\n\n4. **Configure Steps**:\n   - Use the slider to set the number of steps (iterations) for the experiment.\n\n5. **Run the Experiment**:\n   - Click on \"Run AIDE\" to start the experiment.\n   - Progress and status updates will be displayed in the \"Results\" section.\n\n6. **View Results**:\n   - **Tree Visualization**: Explore the solution tree to understand how AIDE experimented and optimized the models.\n   - **Best Solution**: View the Python code of the best solution found.\n   - **Config**: Review the configuration used for the experiment.\n   - **Journal**: Examine the detailed journal entries for each step.\n\n\n## Running AIDE via the Command Line\n\n### Setup\n\nMake sure you have `Python\u003e=3.10` installed and run:\n\n```bash\npip install -U aideml\n```\n\nAlso install `unzip` to allow the agent to autonomously extract your data.\n\nSet up your OpenAI (or Anthropic) API key:\n\n```bash\nexport OPENAI_API_KEY=\u003cyour API key\u003e\n# or\nexport ANTHROPIC_API_KEY=\u003cyour API key\u003e\n# or\nexport OPENROUTER_API_KEY=\u003cyour API key\u003e\n```\n\nTo run AIDE:\n\n```bash\naide data_dir=\"\u003cpath to your data directory\u003e\" goal=\"\u003cdescribe the agent's goal for your task\u003e\" eval=\"\u003c(optional) describe the evaluation metric the agent should use\u003e\"\n```\n\nFor example, to run AIDE on the example [house price prediction task](https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques/data):\n\n```bash\naide data_dir=\"example_tasks/house_prices\" goal=\"Predict the sales price for each house\" eval=\"Use the RMSE metric between the logarithm of the predicted and observed values.\"\n```\n\nOptions:\n\n- `data_dir` (required): A directory containing all the data relevant for your task (`.csv` files, images, etc.).\n- `goal`: Describe what you want the models to predict in your task, for example, \"Build a time series forecasting model for bitcoin close price\" or \"Predict sales price for houses\".\n- `eval`: The evaluation metric used to evaluate the ML models for the task (e.g., accuracy, F1, Root-Mean-Squared-Error, etc.).\n\nAlternatively, you can provide the entire task description as a `desc_str` string, or write it in a plaintext file and pass its path as `desc_file` ([example file](aide/example_tasks/house_prices.md)).\n\n```bash\naide data_dir=\"my_data_dir\" desc_file=\"my_task_description.txt\"\n```\n\nThe result of the run will be stored in the `logs` directory.\n\n- `logs/\u003cexperiment-id\u003e/best_solution.py`: Python code of the _best solution_ according to the validation metric.\n- `logs/\u003cexperiment-id\u003e/journal.json`: A JSON file containing the metadata of the experiment runs, including all the code generated in intermediate steps, plan, evaluation results, etc.\n- `logs/\u003cexperiment-id\u003e/tree_plot.html`: You can open it in your browser. It contains a visualization of the solution tree, which details the experimentation process of finding and optimizing ML code. You can explore and interact with the tree visualization to view what plan and code AIDE comes up with in each step.\n\nThe `workspaces` directory will contain all the files and data that the agent generated.\n\n### Advanced Usage\n\nTo further customize the behavior of AIDE, some useful options might be:\n\n- `agent.code.model=...` to configure which model the agent should use for coding (default is `gpt-4-turbo`).\n- `agent.steps=...` to configure how many improvement iterations the agent should run (default is 20).\n- `agent.search.num_drafts=...` to configure the number of initial drafts the agent should generate (default is 5).\n\nYou can check the [`config.yaml`](aide/utils/config.yaml) file for more options.\n\n### Using Local LLMs\n\nAIDE supports using local LLMs through OpenAI-compatible APIs. Here's how to set it up:\n\n1. Set up a local LLM server with an OpenAI-compatible API endpoint. You can use:\n   - [Ollama](https://github.com/ollama/ollama)\n   - or similar solutions.\n\n2. Configure your environment to use the local endpoint:\n\n   ```bash\n   export OPENAI_BASE_URL=\"http://localhost:11434/v1\"  # For Ollama\n   export OPENAI_API_KEY=\"local-llm\"  # Can be any string if your local server doesn't require authentication\n   ```\n\n3. Update the model configuration in your AIDE command or config. For example, with Ollama:\n\n   ```bash\n   # Example with house prices dataset\n   aide agent.code.model=\"qwen2.5\" agent.feedback.model=\"qwen2.5\" report.model=\"qwen2.5\" \\\n       data_dir=\"example_tasks/house_prices\" \\\n       goal=\"Predict the sales price for each house\" \\\n       eval=\"Use the RMSE metric between the logarithm of the predicted and observed values.\"\n   ```\n\n## Using AIDE in Python\n\nUsing AIDE within your Python script/project is easy. Follow the setup steps above, and then create an AIDE experiment like below and start running:\n\n```python\nimport aide\nexp = aide.Experiment(\n    data_dir=\"example_tasks/bitcoin_price\",  # replace this with your own directory\n    goal=\"Build a time series forecasting model for bitcoin close price.\",  # replace with your own goal description\n    eval=\"RMSLE\"  # replace with your own evaluation metric\n)\n\nbest_solution = exp.run(steps=10)\n\nprint(f\"Best solution has validation metric: {best_solution.valid_metric}\")\nprint(f\"Best solution code: {best_solution.code}\")\n```\n\n## Development\n\nTo install AIDE for development, clone this repository and install it locally:\n\n```bash\ngit clone https://github.com/WecoAI/aideml.git\ncd aideml\npip install -e .\n```\n\n### Running the Web UI in Development Mode\n\nEnsure that you have all the required development dependencies installed. Then, you can run the Web UI as follows:\n\n```bash\ncd aide/webui\nstreamlit run app.py\n```\n\n## Using AIDE with Docker\n\nYou can also run AIDE using Docker:\n\n1. **Build the Docker Image**:\n\n   ```bash\n   docker build -t aide .\n   ```\n\n2. **Run AIDE with Docker** (example with house prices task):\n\n   ```bash\n   # Set custom workspace and logs location (optional)\n   export WORKSPACE_BASE=$(pwd)/workspaces\n   export LOGS_DIR=$(pwd)/logs\n\n   docker run -it --rm \\\n             -v \"${LOGS_DIR:-$(pwd)/logs}:/app/logs\" \\\n             -v \"${WORKSPACE_BASE:-$(pwd)/workspaces}:/app/workspaces\" \\\n             -v \"$(pwd)/aide/example_tasks:/app/data\" \\\n             -e OPENAI_API_KEY=\"your-actual-api-key\" \\\n             aide \\\n             data_dir=/app/data/house_prices \\\n             goal=\"Predict the sales price for each house\" \\\n             eval=\"Use the RMSE metric between the logarithm of the predicted and observed values.\"\n   ```\n\nYou can customize the location of workspaces and logs by setting environment variables before running the container:\n\n- `WORKSPACE_BASE`: Sets the base directory for AIDE workspaces (default: `$(pwd)/workspaces`)\n- `LOGS_DIR`: Sets the directory for AIDE logs (default: `$(pwd)/logs`)\n\nContribution guide will be available soon.\n\n## Algorithm Description\n\nAIDE's problem-solving approach is inspired by how human data scientists tackle challenges. It starts by generating a set of initial solution drafts and then iteratively refines and improves them based on performance feedback. This process is driven by a technique we call Solution Space Tree Search.\n\nAt its core, Solution Space Tree Search consists of three main components:\n\n- **Solution Generator**: This component proposes new solutions by either creating novel drafts or making changes to existing solutions, such as fixing bugs or introducing improvements.\n- **Evaluator**: The evaluator assesses the quality of each proposed solution by running it and comparing its performance against the objective. This is implemented by instructing the LLM to include statements that print the evaluation metric and by having another LLM parse the printed logs to extract the evaluation metric.\n- **Base Solution Selector**: The solution selector picks the most promising solution from the explored options to serve as the starting point for the next iteration of refinement.\n\nBy repeatedly applying these steps, AIDE navigates the vast space of possible solutions, progressively refining its approach until it converges on the optimal solution for the given data science problem.\n\n![Tree Search Visualization](https://github.com/WecoAI/aideml/assets/8918572/2401529c-b97e-4029-aed2-c3f376f54c3c)\n\n# Citation\n\nIf you use AIDE in your work, please cite the following paper:\n```bibtex\n@article{aide2025,\n      title={AIDE: AI-Driven Exploration in the Space of Code}, \n      author={Zhengyao Jiang and Dominik Schmidt and Dhruv Srikanth and Dixing Xu and Ian Kaplan and Deniss Jacenko and Yuxiang Wu},\n      year={2025},\n      eprint={2502.13138},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2502.13138}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWecoAI%2Faideml","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FWecoAI%2Faideml","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FWecoAI%2Faideml/lists"}