--- base_model: - Qwen/Qwen2.5-VL-32B-Instruct datasets: - xlangai/AgentNet - xlangai/aguvis-stage1 - xlangai/aguvis-stage2 - osunlp/UGround-V1-Data language: - en license: mit metrics: - code_eval - accuracy pipeline_tag: image-text-to-text tags: - VLM - Computer-Use-Agent - OS-Agent - GUI - Grounding library_name: transformers ---

OpenCUA-32B

๐ŸŒ Website ๐Ÿ“ Paper ๐Ÿ’ป Code
# ๐Ÿš€ vLLM Serve (Recommended) We recommend using vLLM for production deployment. Requires **vllm>=0.12.0** with `--trust-remote-code`. ```bash # OpenCUA-32B (4 GPUs, tensor parallel) vllm serve xlangai/OpenCUA-32B \ --trust-remote-code \ --tensor-parallel-size 4 \ --served-model-name opencua-32b \ --host 0.0.0.0 \ --port 8000 ``` Adjust `--tensor-parallel-size` and `--gpu-memory-utilization` based on your hardware configuration. --- # Introduction
OpenCUA models (OpenCUA-7B, OpenCUA-32B, and OpenCUA-72B) are end-to-end computer-use foundation models that can produce executable actions in the computer environments with great planning and grounding capabilities. They are based on the Qwen2.5-VL model family. With the help of OpenCUA framework, our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-72B achieves an average success rate of **45.0%** on [OSWorld-Verified](https://os-world.github.io/), establishing a new state-of-the-art (SOTA) among open-source models. OpenCUA-72B also has strong grounding ability, achieving 37.3% (SOTA) on [UI-Vision](https://arxiv.org/abs/2504.07981) and 60.8% on [ScreenSpot-Pro](https://arxiv.org/abs/2504.07981).
## ๐Ÿ“ข Updates - 2026-01-17: ๐ŸŽ‰ **vLLM now fully supports OpenCUA-7B, OpenCUA-32B, and OpenCUA-72B!** Thanks to the [Meituan EvoCUA Team](https://github.com/meituan) for their contributions to vLLM integration. ### Key Features - **Superior Computer-Use Capablity**: Able to execute multi-step computer-use actions with effective planning and reasoning - **Multi-OS Support**: Trained on demonstrations across Ubuntu, Windows, and macOS - **Visual Grounding**: Strong GUI element recognition and spatial reasoning capabilities - **Multi-Image Context**: Processes up to 3 screenshot history for better context understanding - **Reflective Reasoning**: Enhanced with reflective long Chain-of-Thought that identifies errors and provides corrective reasoning # Performance ### Online Agent Evaluation OpenCUA models achieves strong performance on **[OSWorld-Verified](https://os-world.github.io/)**. OpenCUA-72B achieves the best performance among all open-source models with an average success rate of 45.0%, establishing a new state-of-the-art (SOTA).
| **Model** | **15 Steps** | **50 Steps** | **100 Steps** | |-------------------------------|:--------:|:--------:|:---------:| | **Proprietary** | | | | | OpenAI CUA | 26.0 | 31.3 | 31.4 | | Seed 1.5-VL | 27.9 | โ€” | 34.1 | | Claude 3.7 Sonnet | 27.1 | 35.8 | 35.9 | | Claude 4 Sonnet | 31.2 | 43.9 | 41.5 | | **Open-Source** | | | | | Qwen 2.5-VL-32B-Instruct | 3.0 | โ€” | 3.9 | | Qwen 2.5-VL-72B-Instruct | 4.4 | โ€” | 5.0 | | Kimi-VL-A3B | 9.7 | โ€” | 10.3 | | UI-TARS-72B-DPO | 24.0 | 25.8 | 27.1 | | UI-TARS-1.5-7B | 24.5 | 27.3 | 27.4 | | OpenCUA-7B *(Ours)* | 24.3 | 27.9 | 26.6 | | OpenCUA-32B *(Ours)* | 29.7 | 34.1 | 34.8 | | **OpenCUA-72B *(Ours)*** | **39.0** | **44.9** | **45.0** |
*OpenCUA scores are the mean of 3 independent runs.* ### GUI Grounding Performance
| **Model** | **OSWorld-G** | **ScreenSpot-V2** | **ScreenSpot-Pro** | **UI-Vision** | |-------|-----------|---------------|----------------|----------| | Qwen2.5-VL-7B | 31.4 | 88.8 | 27.6 | 0.85 | | Qwen2.5-VL-32B | 46.5 | 87.0 | 39.4 | - | | UI-TARS-72B | 57.1 | 90.3 | 38.1 | 25.5 | | **OpenCUA-7B** | 55.3 | 92.3 | 50.0 | 29.7 | | **OpenCUA-32B** | 59.6 | 93.4 | 55.3 | 33.3 | | **OpenCUA-72B** | **59.2** | **92.9** | **60.8** | **37.3** |
### AgentNetBench (Offline Evaluation)
| **Model** | **Coordinate Actions** | **Content Actions** | **Function Actions** | **Average** | |-------|-------------------|-----------------|------------------|---------| | Qwen2.5-VL-7B | 50.7 | 40.8 | 3.1 | 48.0 | | Qwen2.5-VL-32B | 66.6 | 47.2 | 41.5 | 64.8 | | Qwen2.5-VL-72B | 67.2 | 52.6 | 50.5 | 67.0 | | OpenAI CUA | 71.7 | 57.3 | **80.0** | 73.1 | | **OpenCUA-7B** | 79.0 | 62.0 | 44.3 | 75.2 | | **OpenCUA-32B** | **81.9** | 66.1 | 55.7 | **79.1** |
# ๐Ÿš€ Quick Start
โš ๏ธ Important for Qwen-based Models (OpenCUA-7B, OpenCUA-32B, OpenCUA-72B): To align with our training infrastructure, we have modified the model in two places:
## Installation & Download First, install the required dependencies: ```bash conda create -n opencua python=3.12 conda activate opencua pip install openai>=1.0.0 ``` Download the model weight from huggingface (optional, vLLM can download automatically): ```python from huggingface_hub import snapshot_download snapshot_download( repo_id="xlangai/OpenCUA-32B", local_dir="OpenCUA-32B", local_dir_use_symlinks=False ) ``` ## ๐ŸŽฏ GUI Grounding First, start the vLLM server: ```bash vllm serve xlangai/OpenCUA-32B \ --trust-remote-code \ --tensor-parallel-size 4 \ --served-model-name opencua-32b \ --host 0.0.0.0 \ --port 8000 ``` Then run the following code to test GUI grounding: ```python import base64 from openai import OpenAI # vLLM server configuration VLLM_BASE_URL = "http://localhost:8000/v1" MODEL_NAME = "opencua-32b" # Should match --served-model-name in vllm serve def encode_image(image_path: str) -> str: """Encode image to base64 string.""" with open(image_path, "rb") as f: return base64.b64encode(f.read()).decode() def run_grounding(image_path: str, instruction: str) -> str: """Run GUI grounding inference via vLLM.""" client = OpenAI(base_url=VLLM_BASE_URL, api_key="EMPTY") system_prompt = ( "You are a GUI agent. You are given a task and a screenshot of the screen. " "You need to perform a series of pyautogui actions to complete the task." ) messages = [ {"role": "system", "content": system_prompt}, { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{encode_image(image_path)}"} }, {"type": "text", "text": instruction}, ], }, ] response = client.chat.completions.create( model=MODEL_NAME, messages=messages, max_tokens=512, temperature=0, ) return response.choices[0].message.content # Example usage image_path = "screenshot.png" instruction = "Click on the submit button" result = run_grounding(image_path, instruction) print("Model output:", result) ```
Expected result: ```python\npyautogui.click(x=1432, y=344)\n```
You can also run the grounding examples in [OpenCUA/model/inference/](https://github.com/xlang-ai/OpenCUA/blob/main/model/inference/): ```bash cd ./model/inference/ # vLLM (requires running vLLM server first) python vllm_inference.py # HuggingFace Transformers python huggingface_inference.py ``` ## ๐Ÿ–ฅ๏ธ Computer Use Agent **[OpenCUAAgent](https://github.com/xlang-ai/OSWorld/blob/main/mm_agents/opencua_agent.py)** is developed in the [OSWorld](https://github.com/xlang-ai/OSWorld) environment based on OpenCUA models. It iteratively perceives the environment via screenshots, produces reflective long CoT as inner monologue, and predicts the next action to be executed. OpenCUAAgent uses 3 images in total and L2 CoT format in default. Command for running OpenCUA-32B in OSWorld: ``` python run_multienv_opencua.py \ --headless \ --observation_type screenshot \ --model OpenCUA-32B \ --result_dir ./results --test_all_meta_path evaluation_examples/test_all_no_gdrive.json \ --max_steps 100 \ --num_envs 30 \ --coordinate_type qwen25 ``` ## Important Notes on Coordinate Systems
**OpenCUA models output absolute coordinates after smart resize:** ```python # Example output: pyautogui.click(x=960, y=324) # These are coordinates on the smart-resized image, not the original image # Convert to original image coordinates: # Please refer to the smart_resize function in: https://github.com/huggingface/transformers/blob/67ddc82fbc7e52c6f42a395b4a6d278c55b77a39/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py#L55 def qwen25_smart_resize_to_absolute(model_x, model_y, original_width, original_height): # First, calculate the smart-resized dimensions resized_height, resized_width = smart_resize(original_height, original_width, factor = 28, min_pixels = 3136, max_pixels = 12845056) # Convert model output to relative coordinates on original image rel_x = model_x / resized_width rel_y = model_y / resized_height # Then convert to absolute coordinates on original image abs_x = int(rel_x * original_width) abs_y = int(rel_y * original_height) return abs_x, abs_y ```
Understanding Smart Resize for Qwen2.5-based Models:

The Qwen2.5-VL models use a "smart resize" preprocessing that maintains aspect ratio while fitting within pixel constraints. For coordinate conversion, you need the smart resize function from the official Qwen2.5-VL implementation.

## Research and Commercial Use OpenCUA (including the model, dataset, tools, and code) may be used for **research, educational, and commercial purposes** under the **MIT License** (see `LICENSE`). ## Citation If you use OpenCUA models in your research, please cite our work: ```bibtex @misc{wang2025opencuaopenfoundationscomputeruse, title={OpenCUA: Open Foundations for Computer-Use Agents}, author={Xinyuan Wang and Bowen Wang and Dunjie Lu and Junlin Yang and Tianbao Xie and Junli Wang and Jiaqi Deng and Xiaole Guo and Yiheng Xu and Chen Henry Wu and Zhennan Shen and Zhuokai Li and Ryan Li and Xiaochuan Li and Junda Chen and Boyuan Zheng and Peihang Li and Fangyu Lei and Ruisheng Cao and Yeqiao Fu and Dongchan Shin and Martin Shin and Jiarui Hu and Yuyan Wang and Jixuan Chen and Yuxiao Ye and Danyang Zhang and Dikang Du and Hao Hu and Huarong Chen and Zaida Zhou and Haotian Yao and Ziwei Chen and Qizheng Gu and Yipu Wang and Heng Wang and Diyi Yang and Victor Zhong and Flood Sung and Y. Charles and Zhilin Yang and Tao Yu}, year={2025}, eprint={2508.09123}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2508.09123}, } ```