Microsoft's Mage-VL-Cowork Learns When to Click and When to Code

A spreadsheet task does not always need a mouse. In one demonstration from Microsoft's Mage team, its new desktop agent edits a workbook through Python instead of clicking through LibreOffice Calc. For an image-editing task, it works visually in GIMP, then uses the terminal to check the saved result.

That choice is the point of Mage-VL-Cowork (opens in new tab), a newly released, 4B-class computer-use model. A single vision-language policy selects between screenshot-based interaction and code execution at each turn. It does not need a separate model to route work between a GUI agent and a coding agent.

Microsoft has published the weights under Apache-2.0 (opens in new tab), a technical report and reference evaluation code. It reports 66.0% on OSWorld-Verified with a 50-step budget. The release puts GUI and terminal actions in one compact policy.

Screenshots when the interface matters, code when files matter

Mage-VL-Cowork exposes two tools. computer_use performs mouse and keyboard actions grounded in screenshots. bash executes commands and Python in a persistent environment, returning text rather than another screenshot.

That gives it different ways to reach the same task's goal. A document library can change a spreadsheet's structure directly. A visual editor may still be the right place to manipulate a canvas or application state that has not been saved to disk. The model can switch channels when the next operation requires it.

Microsoft's published demonstrations (opens in new tab) illustrate both cases. Its workbook example finishes in 39 seconds and six steps, compared with nine minutes and 56 seconds for a GUI-only Qwen3.6-27B run. In the GIMP example, Mage uses the interface for editing and the terminal for verification. Those are selected vendor demonstrations, not a general speed guarantee.

The released reference agent (opens in new tab) still supplies the execution loop, tool interfaces and context handling. Routing inside one policy does not mean the model runs a computer without a surrounding harness.

What the 66% measures

The model card's evaluation protocol (opens in new tab) specifies 361 OSWorld-Verified tasks, excluding Google Drive, a 50-step budget and a 1920-by-1080 desktop. The headline is the average of two full passes, which scored 65.33% and 66.75%. Their average is 66.04%, rounded to 66.0%.

The benchmark operator (opens in new tab) describes 369 tasks and permits excluding eight Google Drive tasks because of network dependencies. It also points to the newer OSWorld 2.0. Microsoft's reported result is on OSWorld-Verified, not OSWorld 2.0.

Selected results from Microsoft's comparison are below. These are vendor-published scores, not an independently replicated AgentRiot evaluation.

Microsoft explicitly marks GPT-5.4-mini as evaluated once on the same 361-task subset. Its broader table mixes reported results and settings, so a shared benchmark name and step count do not make every row a controlled comparison.

Mage trails both GPT-5.4-mini and the larger ScaleCUA model in this selection. Within Mage's own results, operating-system tasks score 87.5%, while multi-application workflows score 62.7%. The overall number should not become a promise that two-thirds of arbitrary jobs on someone's laptop will succeed.

There is also a browser-specific loss. Microsoft's separate WebVoyager-Static evaluation reports 58.6% for the supervised-fine-tuned Mage checkpoint, behind browser specialist Fara1.5-4B at 68.0%. That is a different benchmark and an earlier Mage training stage, not the final reinforcement-learned checkpoint's desktop score.

The teacher's tools affected what the small model learned

The technical report (opens in new tab) describes five training stages: visual midtraining, GUI demonstrations, coding and terminal demonstrations, hybrid supervised training, and online reinforcement learning. The reported desktop result rises from 45.56% after GUI supervised training to 55.84% after hybrid supervised training, then roughly 66% after reinforcement learning.

Before collecting demonstrations, the team changed the teacher's environment. It supplied libraries for working with spreadsheets, presentations and PDFs, and removed overlapping action routes such as accessibility-tree tools.

In a separate harness experiment on 672 held-out multi-application tasks, Microsoft kept the GPT-5.4 teacher fixed. GUI-plus-coding scored 92.0%, versus 89.6% with an added accessibility channel. Corresponding students scored 37.6% and 25.1%. These are separate experimental results, not OSWorld-Verified scores. The finding supports the team's harness choice in this setup, not a general argument against accessibility tools.

The reinforcement-learned checkpoint used fewer model calls and output tokens in Microsoft's evaluation. Compared with hybrid supervised training, mean calls fell from 21.57 to 18.77 per task, and output tokens from 7,374 to 5,499. These are training-stage averages on the evaluated tasks.

Open weights, but not a ready-made desktop product

The Hugging Face release (opens in new tab) is downloadable without an access gate and identifies an Apache-2.0 license. Its 4B label is a model-size class, not a four-gigabyte memory requirement. The published bfloat16 weights alone occupy approximately 9.48 GB, before context, activations and serving overhead.

The active training and inference window is 32,768 tokens. Although the architecture supports up to 262,144, that larger number is not the context window used for the reported agent result. Recent screenshots are retained while older actions are folded into summaries, so the agent is not carrying every screen from its history at full fidelity.

The model uses a custom llava_onevision2 architecture. Microsoft's setup requires trust_remote_code=True; it is not a drop-in checkpoint for standard Qwen3-VL classes. Builders need to review the accompanying code and reproduce the intended tool contract, not just download weights.

Microsoft's repository guidance (opens in new tab) describes the models as research releases, not intended for product or service deployment. The Cowork model card (opens in new tab) labels sandbox isolation mandatory: an isolated container or VM without access to host filesystems, local networks or personal environment variables. It also calls for scoped credentials, outbound network restrictions and human confirmation for irreversible actions. These are stated use and safety guidelines alongside the Apache-2.0 weight license.

The release gives builders a way to test learned GUI-to-terminal switching in an isolated desktop environment.