🌏 Website • 📝 Paper • 🔍 Data • 📸 Task Viewer

📢 Updates

• 2024-07-16: We released our environment and dataset, website and arxiv paper. Check it out!

🏗️ Installation

💾 On Your Desktop or Server

The executable environment~(desktop Ubuntu 22.04 LTS) is based on our previous work OSWord. Please install the pip dependencies and virtual machine first.

1. First, clone this repository and cd into it. Then, install the dependencies listed in requirements.txt. It is recommended that you use the latest version of conda to manage the environment, but you can also choose to manually install the dependencies. Please ensure that the version of Python is >= 3.11.

# Clone the Spider2-V repository
git clone https://github.com/xlang-ai/Spider2-V
# Change directory into the cloned repository
cd Spider2-V
# Optional: Create a Conda environment for Spider2-V
conda create -n spider2v python=3.11
conda activate spider2v
# Install required dependencies
pip install -r requirements.txt

2. Install VMware Workstation Pro (for systems with Apple Chips, you should install VMware Fusion) and append the vmrun command into system path. The installation process can refer to How to install VMware Worksation Pro. Verify the successful installation by running the following:

vmrun -T ws list

If the installation along with the environment variable set is successful, you will see the message showing the current running virtual machines.

Note: We will also support using VirtualBox in the near future if you have issues with VMware Pro.

3. Install other dependencies like Playwright.

playwright install chromium

All set! Now, you/agent can interact with the environment via the Quick Start below.

🚀 Quick Start

Run the following minimal example to interact with the environment:

• It will automatically download the prepared virtual machine from Huggingface (for arm64 and for x86) into ./vm_data, configure the environment, and take one snapshot called init_state for you.

from desktop_env.envs.desktop_env import DesktopEnv
# feel free to change the example!
# task instruction: Help me materialize the asset top10_story_ids in this dagster project in the UI. Do NOT materialize other assets.
example_path = 'evlauation_examples/examples/dagster/22ef9058-6188-422a-9c12-e6934e4ed936/22ef9058-6188-422a-9c12-e6934e4ed936.json'
with open(example_path, 'r') as infile:
    example = json.load(infile)
env = DesktopEnv(action_space="pyautogui")
obs = env.reset(task_config=example)
print(f'Task instruction: {example["instruction"]}')
obs, reward, done, info = env.step("pyautogui.rightClick()")
input('Now, you can finish the task in the virtual machine manually and Press ENTER to evaluate ...')
score = env.evaluate()
print(f'Evaluation score: {float(score):.1f}')
env.close()

You will see all the logs of the system running normally, including the successful creation of the environment, completion of setup, and successful execution of a rightClick action on the screen, which means you are ready to continue.

🧪 Experiments

A quick glance on the success rates of state-of-the-art VLM agents.

🕸️ Different Data Splits

The entire task set contains