{"id":23587651,"url":"https://github.com/thunlp/ProactiveAgent","last_synced_at":"2025-08-30T04:31:11.781Z","repository":{"id":265077173,"uuid":"859954124","full_name":"thunlp/ProactiveAgent","owner":"thunlp","description":"A LLM-based Agent that predict its tasks proactively.","archived":false,"fork":false,"pushed_at":"2025-08-22T08:57:28.000Z","size":2936,"stargazers_count":411,"open_issues_count":5,"forks_count":33,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-22T10:39:54.270Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thunlp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-09-19T15:02:33.000Z","updated_at":"2025-08-22T08:57:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"478d9bbe-ea73-4fdc-920e-455a22d5f7ee","html_url":"https://github.com/thunlp/ProactiveAgent","commit_stats":null,"previous_names":["thunlp/proactiveagent"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/thunlp/ProactiveAgent","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunlp%2FProactiveAgent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunlp%2FProactiveAgent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunlp%2FProactiveAgent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunlp%2FProactiveAgent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thunlp","download_url":"https://codeload.github.com/thunlp/ProactiveAgent/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thunlp%2FProactiveAgent/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272805293,"owners_count":24995909,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-30T02:00:09.474Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-27T05:01:25.936Z","updated_at":"2025-08-30T04:31:11.764Z","avatar_url":"https://github.com/thunlp.png","language":"Python","funding_links":[],"categories":["Python","A01_文本生成_文本对话"],"sub_categories":["大语言对话模型及数据"],"readme":"\u003cdiv align= \"center\"\u003e\n    \u003ch1\u003e Proactive Agent \u003c/h1\u003e\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\n![Dialogues](https://img.shields.io/badge/Current\\_Event\\_Size-6790-red?style=flat-square)\n\u003c/div\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#overview\"\u003eOverview\u003c/a\u003e •\n  \u003ca href=\"#data\"\u003eData Release\u003c/a\u003e •\n  \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e •\n  \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2410.12361\"\u003ePaper\u003c/a\u003e •\n  \u003ca href=\"https://huggingface.co/YancyLee/ProactiveAgent/tree/main\"\u003eModel\u003c/a\u003e\n\u003c/p\u003e\n\n\u003c/div\u003e\n\n## News\n\n- **[2025/03/21]**: The model for Proactive Agent and Reward Agent is released! Check our model at [here](https://huggingface.co/YancyLee/ProactiveAgent/tree/main)!\n- **[2025/01/22]**: Our paper was accepted by ICLR 2025.\n\nThis project (Proactive Agent) aims to construct a fully active agent, who may anticipate user's requirements and take the initiative, offering assistance and suggesting actions without explicit requests from user. We achieve this by developing a data collection and generation pipeline, building an automatic evaluator and training agent within data generated. For now, we provide the whole collection and generation pipeline, the datasets, and the corresponding evaluation scripts, and the prompts to finetune LLM for proactive agent.\n\n*Read this in [中文](README_zh.md).*\n*Read this in [日本語](README_ja.md).*\n\n## Overview\n\n✨Here is an overview of the whole process of Proactive Agent.\n\n\u003cbr\u003e\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/overall_pipeline.png\" width=\"800px\"\u003e\n\u003c/div\u003e\n\u003cbr\u003e\n\n\n✨✨Features:\n- **Environment Sensing**: We provide scripts to collect environment scenes and user activities through Activity Watcher, and recommend tasks automatically based on the model.\n- **Assistance Annotation**: We provide a platform to annotate the response generated by the proactive agent, which is a good way to align the result with human annotators.\n- **Dynamic Generation**: We provide a dynamic pipeline to generate new data, the feedback from user could affect events afterwards.\n- **Construction Pipeline**: We provide a generation pipeline consist of **Environment Gym**, **Proactive Agent** and **Reward Model**, where our Reward Model reaches a `0.918` F1 score on the test set.\n\nA demo is also provided to show the performance of our agent.\n\nhttps://github.com/user-attachments/assets/81e56660-727f-4ba8-b26c-d28b580a8b0a\n\nIn the future, we will continually improve the data quality and increase the coverage of real-world scenarios.\n\n## Data\n\n👐Proactive Agent is intended for coding, writing and daily life scenarios only at present and should not be constructed as reflecting the opinions or views of the creators, owners, or contributors of this dataset. It is distributed under Apache License 2.0. Below is the statistics of the data:\n\n| Settings | Coding | Writing | Daily Life | Total |\n|:-------------:|:------:|:-----:|:-----:|:-----:|\n|   Inst.Num    | 46 |  46 | 44 | 136 |\n|   Events Num  | 2275 |  2354 | 2161 | 6790 |\n\nAll the training instances for the Proactive Agent were generated from our [GYM](gym/README.md).\nWe utilize the [Activity Watcher](https://activitywatch.net/) to collect the human traces across all the scenes, and annotate a test set to validate the effectiveness of the Proactive Agent.\nMore details about the data collection and annotation can be found [here](dataset/README.md).\n\n## 📦 Installation\n\nClone this repository and navigate to the proactive demand sensing agent folder\n```bash\ngit clone git@github.com:thunlp/ProactiveAgent\ncd ProactiveAgent\n```\n\nInstall Package\n```bash\nconda create -n activeagent python=3.10\nconda activate activeagent\npip install -r requirements.txt\n```\n\n### Install Activity Watcher\n\n- You can go to the [Official Website](https://activitywatch.net/downloads/) to download the main app based on your operating system.\n- An extension for chrome is at `./agent/resource/aw-watcher-web.zip`. To download this extension, you will have to download the file and unzip it.\n  - For Edge users, go to `edge://extensions/` site, open developer mode and load the extension by clicking `load unpacked`.`\n  - For Google Chrome users, go to `chrome://extensions/` site, open developer mode and select `load unpacked` to load the unziped extension.\n  - This Extension is not tested under `Safari`.\n- There is an official extension for vscode user, you may download it from the [marketplace](https://marketplace.visualstudio.com/items?itemName=activitywatch.aw-watcher-vscode) or search for `aw-watcher-vscode` in the extensions in your vscode and install it.\n\nTo check whether the installation is complete, please open your browser and go to `http://localhost:5600/#/timeline` to check if there are four traces displaying in the window(`afk`,`vscode`,`window`,`web`).\n\n\n## 🚀 Usage\n\n### Configuration\nYou should first configure the `private.toml` file. The example is given in `example_config.toml`:\n\n```bash\ncp example_config.toml private.toml\n```\n\nYou should change the `default_completions_model`, `api_key` and `base_url` to your own settings.\n\n\n### Running the Proactive Agent\nTo experience our proactive agent, you will first enter folder `./agent` and then follow the instructions in [here](agent/README.md).\n\n\n### Connect the Reward Model\nTo improve the experience with the Proactive Agent, you can use our built reward model to filter the message from the Proactive Agent.\nHere are steps to connect the reward model with the Proactive Agent.\n__TO BE UPDATE__\n\n\n### Interact with the Proactive Agent\nOur agent will try to make a proposal by creating a toast on the window, to interact with the proactive agent, you may choose:\n- Accept the proposal: you will click on the toast body(Windows) or click the button(MacOS) to let the agent know you accept his idea, the agent will make relavent actions in return.\n- Reject the proposal: you will click on the dismiss button(the x on the top right of the toast) to let the agent know you **reject** the proposal, the agent will try to propose in some other way for next turn.\n- Ignore the proposal: you will do nothing, the agent will remove the toast for some time depending on the time interval, doing nothing will make the agent know that you are busy and ignored the proposal, the agent will try to make less proposal in the following turns.\n\n\n## 📊 Model Results\nTo automatic evaluate the performance of the Proactive Agent, we build a reward model based on our annotated data to judge the performance of the Proactive Agent.\nOur reward model reaches a `0.918` F1 score on the test set, which is a good indicator of the performance of the Proactive Agent.\n\n### Reward Model Experiments Results\nWe test the agreement between the reward model and human annotators on the test set:\n- **Missed-Needed (MN)**: The scenario when the user needs help but the agent does not provide help.\n- **Non-Response(NR)**: The scenario when the user does not need help and the agent does not prompt any help.\n- **Correct-Detection(CD)**: The scenario when the user needs help and the agent provides help.\n- **False-Alarm(FA)**: The scenario when the user does not need help but the agent prompts help.\n\nWe compare the judgement of the reward model with the human annotators.\nWe compare the performance of different LLMs and our model on the [test set](eval/README.md#reward-model-evaluation).\nThe results are as follows:\n\n|                         | GPT-4o  | GPT-4o-mini | LLaMa 3.1 8b | LLaMa 3.1 70b  | ours    |\n|-------------------------|---------|-------------|--------------|----------------|---------|\n| Missed-Need (MN)        | 0.0333  | 0.5667      | 0.8000       | 0.3333         | 0.8000  |\n| Non-Response (NR)       | 1.0000  | 0.5667      | 0.3000       | 0.8333         | 0.8667  |\n| Correct-Detection (CD)  | 1.0000  | 0.8667      | 0.9667       | 1.0000         | 1.0000  |\n| False-Alarm (FA)        | 0.0000  | 0.3333      | 0.1333       | 0.0667         | 1.0000  |\n| Accuracy                | 0.5083  | 0.5833      | 0.5500       | 0.5583         | **0.9167**  |\n| Precision               | 0.5042  | 0.5658      | 0.5429       | 0.5340         | **0.9032**  |\n| Recall                  | 1.0000  | 0.7167      | 0.6333       | 0.9167         | **0.9333**  |\n| F1                      | 0.6704  | 0.6324      | 0.5846       | 0.6748         | **0.9180**  |\n\n\n### Proactive Agent Experiments Results\nIn current experiments, we evaluate the performance of the Proactive Agent with our Reward Model.\nWe define the following metrics:\n- **True Positive(TP)**: Instances where the proactive agent correctly predicts a task that the reward model subsequently accepts.\n- **False Positive(FP)**: Instances where the proactive agent predicts a task that the reward model does not accept.\n- **True Negative(TN)**: Instances where the proactive agent correctly refrains from predicting a task,and the reward model also does not accept any task.\n- **False Negative(FN)**: Instances where the proactive agent fails to predict a task that the reward model would have accepted if proposed.\n\nWe report the performance of the Proactive Agent on the test set of the [ProactiveBench](eval/README.md).\n\n| Model                  | Recall  | Precision | Accuracy | False-Alarm | F1-Score  |\n|:----------------------:|:-------:|:---------:|:--------:|:-----------:|:---------:|\n| GPT-4o-mini            | 100.00% | 35.28%    | 36.12%   | 64.73%      | 52.15%    |\n| GPT-4o                 | 98.11%  | 48.15%    | 49.78%   | 51.85%      | 64.60%    |\n| Claude-3.5-Sonnet      | 97.89%  | 45.37%    | 49.78%   | 54.63%      | 62.00%    |\n| LLaMA-3.1-8B           | 98.86%  | 38.16%    | 39.06%   | 61.84%      | 55.06%    |\n| LLaMA-3.1-8B-Proactive | 99.06%  | 49.76%    | 52.86%   | 50.24%      | 66.25%    |\n| Qwen2-7B               | 98.02%  | 44.00%    | 43.61%   | 56.00%      | 60.74%    |\n| Qwen2-7B-Proactive     | 100.00% | 49.78%    | 50.66%   | 50.22%      | 66.47%    |\n\n\n\n## Citation\nIf you find this project useful in your research, please consider citing it:\n```\n@misc{2024,\n  author = {THUNLP},\n  title = {ProactiveAgent},\n  year = {2024},\n  publisher = {GitHub},\n  journal = {GitHub Repository},\n  howpublished = {\\url{https://github.com/thunlp/ProactiveAgent}}\n}\n```\n\n## Friendly Links\n- [ChatDev](https://github.com/openbmb/ChatDev)\n- [Activity Watcher](https://activitywatch.net/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp%2FProactiveAgent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp%2FProactiveAgent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp%2FProactiveAgent/lists"}