Logo MVISU-Bench

Benchmarking Mobile Agents for Real-World Tasks by
Multi-App, Vague, Interactive, Single-App and Unethical Instructions

Abstract

Given the significant advances in Large Vision Language Models (LVLMs) in reasoning and visual understanding, mobile agents are rapidly emerging to meet users' automation needs. However, existing evaluation benchmarks are disconnected from the real world and fail to adequately address the diverse and complex requirements of users. From our extensive collection of user questionnaire, we identified five tasks: Multi-App, Vague, Interactive, Single-App, and Unethical Instructions. Around these tasks, we present MVISU-Bench, a bilingual benchmark that includes 404 tasks across 137 mobile applications. Furthermore, we propose Aider, a plug-and-play module that acts as a dynamic prompt prompter to mitigate risks and clarify user intent for mobile agents. Our Aider is easy to integrate into several frameworks and has successfully improved overall success rates by 19.55% compared to the current state-of-the-art (SOTA) on MVISU-Bench. Specifically, it achieves success rate improvements of 53.52% and 29.41% for unethical and interactive instructions, respectively. Through extensive experiments and analysis, we highlight the gap between existing mobile agents and real-world user expectations.

Leaderboard

# Framework & Model English Instructions Chinese Instructions
SA VA UN IN MA ALL SA VA UN IN MA ALL
- Human Expert

Benchmark

10097.2210097.2296.4397.98 10097.2210097.2296.8898.06
Claude-3-5-sonnet

Mobile-Agent-V2 · Anthropic

74.2988.8965.710.0050.0055.05 60.0077.7822.220.0022.5835.92
Gemini-2.0-pro

Mobile-Agent-E · Google

88.5775.0022.860.0044.6445.96 77.5075.0016.670.0045.1644.66
GPT-4o-2024-11-20

Mobile-Agent-V2 · OpenAI

77.1450.0014.310.0042.8637.37 52.5066.6716.670.0029.0333.50
Qwen2.5-vl-72b

Mobile-Agent-V2 · Alibaba

11.4316.6720.000.007.1410.60 27.5016.6750.000.006.4518.93
Qwen2.5-vl-7b

Mobile-Agent · Alibaba

5.712.7825.710.000.006.06 0.008.3350.000.000.0010.19
Qwen2.5-vl-3b

Mobile-Agent · Alibaba

0.000.0025.710.000.004.56 0.000.0033.330.000.005.83

Logo MVISU-Benchmark

Construction Pipeline

data-composition

The data collection pipeline of MVISU-Bench, including Questionnaire Survey, Instruction Generation, Multi-round Filtering, and Human Verification. This process gradually refined the final 404 bilingual MVISU-Bench dataset.

Benchmark Statistics

grade-lv

Evaluation results of four representative MLLMs.

Benchmark Comparison

data-composition

Comparisons between MVISU-Bench and other mobile agents benchmarks. Our MVISU-Bench is derived from real user questionnaire and aligns more closely with the user's expectations of a mobile agent.

Experiment Results

Success Rate comparison of closed source and open source models across different mobile agent frameworks on our MVISU-bench. Among them, SA, VA, UN, IN, MA, and ALL denote "Single-APP", "Vague", "Unethical", "Interactive", "Multi-APP", "All tasks", respectively. Bold represents optimal performance, while underline represents suboptimal.

Performance on different backbones and language instructions. For metrics, SR, AC, DT, Cost, Steps, IT, and OT denote "Success Rate", "API Calls", "Duration", "Cost", "Steps", "Input Tokens", and "Operation Time", respectively.