{"id":20064468,"url":"https://github.com/xlang-ai/spider2-v","last_synced_at":"2025-04-30T08:20:28.059Z","repository":{"id":245097106,"uuid":"784033470","full_name":"xlang-ai/Spider2-V","owner":"xlang-ai","description":"[NeurIPS 2024] Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?","archived":false,"fork":false,"pushed_at":"2024-08-26T17:41:17.000Z","size":140285,"stargazers_count":120,"open_issues_count":0,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-30T14:21:33.026Z","etag":null,"topics":["agent","artificial-intelligence","code-generation","data-science-and-engineering","gui","llm","multi-modal","natural-language-processing","vlm"],"latest_commit_sha":null,"homepage":"https://spider2-v.github.io","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xlang-ai.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-09T03:48:55.000Z","updated_at":"2025-03-26T07:16:51.000Z","dependencies_parsed_at":"2024-11-13T13:56:28.470Z","dependency_job_id":null,"html_url":"https://github.com/xlang-ai/Spider2-V","commit_stats":null,"previous_names":["xlang-ai/spider2-v"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2FSpider2-V","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2FSpider2-V/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2FSpider2-V/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xlang-ai%2FSpider2-V/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xlang-ai","download_url":"https://codeload.github.com/xlang-ai/Spider2-V/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251666608,"owners_count":21624339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","artificial-intelligence","code-generation","data-science-and-engineering","gui","llm","multi-modal","natural-language-processing","vlm"],"created_at":"2024-11-13T13:46:23.072Z","updated_at":"2025-04-30T08:20:28.022Z","avatar_url":"https://github.com/xlang-ai.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/spider2v-overview.png\" alt=\"Spider2-V\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://spider2-v.github.io/\"\u003e🌏 Website\u003c/a\u003e •\n  \u003ca href=\"https://arxiv.org/abs/2407.10956\"\u003e📝 Paper\u003c/a\u003e •\n  \u003ca href=\"https://github.com/xlang-ai/Spider2-V/tree/main/evaluation_examples\"\u003e🔍 Data\u003c/a\u003e •\n  \u003ca href=\"https://spider2-v.github.io/explorer.html\"\u003e📸 Task Viewer\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n    \u003ca href=\"https://img.shields.io/badge/PRs-Welcome-red\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/PRs-Welcome-red\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://img.shields.io/github/last-commit/xlang-ai/Spider2-V?color=green\"\u003e\n        \u003cimg src=\"https://img.shields.io/github/last-commit/xlang-ai/Spider2-V?color=green\"\u003e\n    \u003c/a\u003e\n    \u003ca href=\"https://opensource.org/licenses/Apache-2.0\"\u003e\n        \u003cimg src=\"https://img.shields.io/badge/License-Apache%202.0-blue.svg\"\u003e\n    \u003c/a\u003e\n    \u003cbr/\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/overall.png\" alt=\"Tasks\"\u003e\n\u003c/p\u003e\n\n## 📢 Updates\n\n- 2024-07-16: We released our [environment and dataset](https://github.com/xlang-ai/Spider2-V), [website](https://spider2-v.github.io/) and [arxiv paper](https://arxiv.org/abs/2407.10956). Check it out!\n\n## 🏗️ Installation\n### 💾 On Your Desktop or Server\nThe executable environment~(desktop Ubuntu 22.04 LTS) is based on our previous work [OSWord](https://github.com/xlang-ai/OSWorld). Please install the pip dependencies and virtual machine first.\n\n1. First, clone this repository and `cd` into it. Then, install the dependencies listed in `requirements.txt`. It is recommended that you use the latest version of `conda` to manage the environment, but you can also choose to manually install the dependencies. Please ensure that the version of Python is \u003e= 3.11.\n```bash\n# Clone the Spider2-V repository\ngit clone https://github.com/xlang-ai/Spider2-V\n\n# Change directory into the cloned repository\ncd Spider2-V\n\n# Optional: Create a Conda environment for Spider2-V\nconda create -n spider2v python=3.11\nconda activate spider2v\n\n# Install required dependencies\npip install -r requirements.txt\n```\n2. Install [VMware Workstation Pro](https://www.vmware.com/products/workstation-pro/workstation-pro-evaluation.html) (for systems with Apple Chips, you should install [VMware Fusion](https://www.vmware.com/go/getfusion)) and append the `vmrun` command into system path.  The installation process can refer to [How to install VMware Worksation Pro](./INSTALL_VMWARE.md). Verify the successful installation by running the following:\n```bash\nvmrun -T ws list\n```\nIf the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines.\n\u003e **Note:** We will also support using [VirtualBox](https://www.virtualbox.org/) in the near future if you have issues with VMware Pro.\n\n3. Install other dependencies like Playwright.\n```bash\nplaywright install chromium\n```\nAll set! Now, you/agent can interact with the environment via the [Quick Start](#-quick-start) below.\n\n\n### 🚀 Quick Start\nRun the following minimal example to interact with the environment:\n- It will automatically download the prepared virtual machine from Huggingface (for [arm64](https://huggingface.co/datasets/xlangai/ubuntu_spider2v/blob/main/ubuntu-arm.zip) and for [x86](https://huggingface.co/datasets/xlangai/ubuntu_spider2v/blob/main/ubuntu-x86.zip)) into `./vm_data`, configure the environment, and take one snapshot called `init_state` for you.\n```python\nfrom desktop_env.envs.desktop_env import DesktopEnv\n\n# feel free to change the example!\n# task instruction: Help me materialize the asset top10_story_ids in this dagster project in the UI. Do NOT materialize other assets.\nexample_path = 'evlauation_examples/examples/dagster/22ef9058-6188-422a-9c12-e6934e4ed936/22ef9058-6188-422a-9c12-e6934e4ed936.json'\nwith open(example_path, 'r') as infile:\n    example = json.load(infile)\n\nenv = DesktopEnv(action_space=\"pyautogui\")\n\nobs = env.reset(task_config=example)\nprint(f'Task instruction: {example[\"instruction\"]}')\nobs, reward, done, info = env.step(\"pyautogui.rightClick()\")\ninput('Now, you can finish the task in the virtual machine manually and Press ENTER to evaluate ...')\nscore = env.evaluate()\nprint(f'Evaluation score: {float(score):.1f}')\nenv.close()\n```\nYou will see all the logs of the system running normally, including the successful creation of the environment, completion of setup, and successful execution of a `rightClick` action on the screen, which means you are ready to continue.\n\n## 🧪 Experiments\n\nA quick glance on the success rates of state-of-the-art VLM agents.\n\n| VLM            | Abstract | Verbose | Account | Non-Account | Overall  |\n| -------------- | -------- | ------- | ------- | ----------- | -------- |\n| Claude-3-Opus  | 5.3      | 10.9    | 5.9     | 9.3         | 8.1      |\n| Gemini-Pro-1.5 | 6.1      | 12.1    | 8.8     | 9.3         | 9.1      |\n| GPT-4o         | 11.3     | 16.2    | 10.6    | 15.6        | 13.8     |\n| GPT-4V         | 11.3     | 16.6    | 11.2    | 15.4        | **14.0** |\n\n### 🕸️ Different Data Splits\nThe entire task set contains $494$ examples (`evaluation_examples/test_all.json`) covering $20$ applications. There are different categories to split the entire task set into subsets, depending on:\n- whether the task instruction is *verbose* or *abstract* (`evaluation_examples/test_verbose.json` and `evaluation_examples/test_abstract.json`)\n- whether a real account is needed (`evaluation_examples/test_account.json` and `evaluation_examples/test_non_account.json`)\n- etc.\n\nNote that, *verbose* instruction means we explicitly tell the agent how to complete the task step-by-step. If you want to test examples which require real accounts (e.g., Snowflake and Google BigQuery), please register relevant accounts and fill credentials into template files under folder `evaluation_examples/settings` first (see [Account Guideline](ACCOUNT_GUIDELINE.md) for more step-by-step details).\n\nWith respect to the task format, see [Task Format](evaluation_examples/README.md#task-format).\n\n### 👷 Human Validation\nIf you want to check more examples manually in the virtual machine on a GUI screen, which includes:\n1. reseting the environment;\n2. completing the task in the virtual machine by yourself;\n3. invoking the customized evaluation function after completion,\nyou can run the following interactive script:\n- if the `--path_to_vm` argument is not specified, the script will automatically find available virtual machines under folder `./vm_data`. If not found, it will download our prepared VM from Huggingface (for [arm64](https://huggingface.co/datasets/xlangai/ubuntu_spider2v/blob/main/ubuntu-arm.zip) and for [x86](https://huggingface.co/datasets/xlangai/ubuntu_spider2v/blob/main/ubuntu-x86.zip)) into `./vm_data` and use it\n- if the `--snapshot` argument is not specified, it will use snapshot with name `init_state` by default\n- if the `--example` argument is not specified, it will test on one example `evaluation_examples/test_one.json` by default\n```bash\npython run_spider2v_human.py --snapshot init_state --example evaluation_examples/test_non_account.json\n```\n\n### 💻 Agent Baselines\nIf you wish to run the baseline agent used in our paper, you can execute the following command as an example under the GPT4-o pyautogui+SoM+ExecutionFeedback+RAG setting:\n\nSet **OPENAI_API_KEY** environment variable with your API key:\n```bash\nexport OPENAI_API_KEY='changme'\n```\n\n```bash\npython run_spider2v_agent.py --snapshot init_state \\\n    --model gpt-4o-2024-05-13 \\\n    --headless \\\n    --action_space pyautogui \\\n    --observation_space som \\\n    --execution_feedback \\\n    --rag \\\n    --result_dir ./results \\\n    --example evaluation_examples/test_non_account.json\n```\nThe results, which include screenshots, a11y trees, actions, and video recordings of the agent's task completion, will be saved in the `./results` directory.\n\n\u003e If you want to use other models, you can revise the code file `mm_agents/agent.py`.\n\n\n## ❓ FAQ\n### What is the username and password for the virtual machines?\nThe username and password for the virtual machines are as follows:\n- **Ubuntu:** `user` / `password`\n\n### How to tackle task examples requiring accounts?\n\nSee [Account Guideline](ACCOUNT_GUIDELINE.md)\n\n### How can I configure a proxy for the VM if I'm behind a GFW?\n\nSee [Proxy Guideline](PROXY_GUIDELINE.md).\n\n### Where to download the documents for retrieval?\nThe preprocessed documents can be downloaded from [GoogleDrive](https://drive.usercontent.google.com/download?id=1aGaHXDkBeoUZ9EOIPj7iIRFra_2FjJoZ\u0026export=download\u0026authuser=0\u0026confirm=t). See [Documents](evaluation_examples/documents/README.md) for details.\n\n### What are the running times and costs?\n| Setting | Expected Time | Average Prompt Tokens | Average Completion Tokens | Average Cost |\n| --------| ------------- | --------------------- | ------------------------- | ------------ |\n| GPT-4o  | 8.7m         | 0.32M                 | 1.6K                      | $1.64        |\n| GPT-4V  | 9.1m         | 0.32M                 | 1.8K                      | $3.29        |\n\n- action_space = pyautogui, observation_space = SoM\n- history_trajectory_length = 3, max_steps = 15, RAG = True\n- prompt tokens include image pixels. Calculated per task in June 2024.\n\n## 📄 Citation\nIf you find this benchmark useful, please consider citing our work:\n```\n@misc{Spider2-V,\n      title={Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?}, \n      author={Ruisheng Cao and Fangyu Lei and Haoyuan Wu and Jixuan Chen and Yeqiao Fu and Hongcheng Gao and Xinzhuang Xiong and Hanchong Zhang and Yuchen Mao and Wenjing Hu and Tianbao Xie and Hongshen Xu and Danyang Zhang and Sida Wang and Ruoxi Sun and Pengcheng Yin and Caiming Xiong and Ansong Ni and Qian Liu and Victor Zhong and Lu Chen and Kai Yu and Tao Yu},\n      year={2024},\n      eprint={2407.10956},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2407.10956}, \n}\n```\n\n```\n@misc{OSWorld,\n      title={OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}, \n      author={Tianbao Xie and Danyang Zhang and Jixuan Chen and Xiaochuan Li and Siheng Zhao and Ruisheng Cao and Toh Jing Hua and Zhoujun Cheng and Dongchan Shin and Fangyu Lei and Yitao Liu and Yiheng Xu and Shuyan Zhou and Silvio Savarese and Caiming Xiong and Victor Zhong and Tao Yu},\n      year={2024},\n      eprint={2404.07972},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlang-ai%2Fspider2-v","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxlang-ai%2Fspider2-v","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxlang-ai%2Fspider2-v/lists"}