Interactive Visual Gym tests whether AI advice arrives in time
By AgentRiot Editorial
The benchmark lets simulated users change course after an assistant speaks. Its results expose different weaknesses in task completion, guidance and response speed.

A cooking assistant can recognize a mistake and still fail to prevent it. Imagine a user about to plate a raw burger patty. A warning delivered before plating can change the next action. The same warning delivered afterward describes a mistake that has already happened, leaving the user to recover or fail the recipe.
That illustrative example captures the problem behind Interactive Visual Gym, research from the University of Washington and Salesforce AI Research. Instead of scoring an assistant against an unchanging video, the framework lets its advice affect what a simulated user does next. The environment keeps moving while the assistant prepares its response. Correctness and timeliness become separate requirements. The paper (opens in new tab) describes raw serving among the cooking errors its harness can schedule.
The results resist a single winner headline. Claude Opus 5.5 has the highest average task success across three environments under the main timing conditions. Gemini 4 Argon leads cooking success there. When measured response delays enter the cooking evaluation, Muse Glimmer takes the success lead while scoring poorly on guidance quality.
This is a research testbed, not evidence that any assistant is safe to supervise a real kitchen. The preprint (opens in new tab) was first submitted November 5 and revised November 12, 2026. The benchmark figures below are the authors' results, not an independent reproduction.
What changes when the user can listen
In a recorded-video evaluation, an assistant's warning cannot alter the footage. A useful intervention and a useless one face the same next frame. Interactive Visual Gym instead connects an executable environment, a model playing the user, and the assistant being evaluated.
The user plans and acts. The assistant observes the activity and communicates through text, including the option to stay silent. If the user accepts guidance, it can revise its action queue. The engine then produces a different state and different observations. The assistant has no direct action channel: recognizing an error does not automatically stop the user's hand.
Three environments exercise that loop. CookSim models recipes, preparation states and assembly order. VHSim adapts VirtualHome for household navigation and manipulation, such as loading a dishwasher and returning food to a fridge. ScreenSim recreates phone settings tasks, where success can require changing several permissions while preserving unrelated settings. These are simulated kitchens, apartments and interfaces, not deployments in those real settings. Section 3 (opens in new tab) details their construction.
The harness injects events when task progress reaches specified conditions. These include execution mistakes, questions, proactive notification requests and changed plans. It temporarily forces the user's action queue and privately tells the user model that the event is testing the assistant. These are controlled probes, not naturally arising human mistakes. Four persona axes vary skill, trust in AI, execution speed and interruption preference, each with three levels. The assistant receives the user's speed and interruption preferences, but must infer skill and trust.
That adds useful friction. A correct warning still has to persuade the user, and frequent commentary can violate a preference for minimal interruption. But the simulated user has privileged information: structured state, execution feedback and private event instructions. The assistant's observation package varies by environment. In cooking it lacks the user's structured state; in VHSim it receives spatial aids alongside images. This is a controlled collaboration test with asymmetric information, not two models seeing exactly what two people would see.
What the scores actually count
IVG-Bench's 315 episodes per model per run come from 105 base cases paired with three personas each: 50 cooking cases become 150 episodes, 25 household cases become 75, and 30 phone cases become 90. Each condition has three independent runs. Repeated rollouts measure variability on those scenarios; they do not turn the benchmark into hundreds of unrelated real-world tasks. Appendix G (opens in new tab) gives the case construction.
In-time success requires an engine-verified goal before a task-specific deadline derived from an error-free reference rollout and an event allowance. Detection recall uses a different denominator: triggered mistakes identified within their eligible windows. Prevention is stricter, requiring intervention before the erroneous action becomes irreversible.
Interaction quality is a response-level assessment. GPT-5.4 mini and Gemini 3.7 Flash judge factual grounding, situational relevance, actionable guidance, user-intent uptake and conciseness against execution evidence. A factual-grounding failure gates the overall score. Quality therefore is not the proportion of tasks completed, nor a simple average of the five displayed category percentages. Appendix D (opens in new tab) defines the outcome metrics; Appendix I (opens in new tab) explains quality scoring.
The main comparison fixes response delay at two simulation ticks in CookSim and VHSim. ScreenSim uses a zero-tick, action–response setting. The following selection includes all five frontier API models, both real-time API models and two open-weight comparators. All figures are percentages averaged across three runs from Table 2 (opens in new tab).
| Assistant | Cooking success | Household success | Phone success | Cooking quality |
|---|---|---|---|---|
| Gemini 4 Argon | 60.9 | 15.6 | 83.7 | 63.8 |
| Claude Opus 5.5 | 57.1 | 14.2 | 89.3 | 66.0 |
| Gemini 3.8 Flash | 46.4 | 14.7 | 78.5 | 73.4 |
| GPT-6.1 Sol | 42.9 | 15.6 | 87.4 | 75.8 |
| GPT-6 Astra | 40.2 | 10.7 | 90.4 | 76.0 |
| GPT-Realtime 2.1 | 33.3 | 11.6 | 66.7 | 45.1 |
| Gemini 3.8 Live | 26.2 | 11.6 | 71.9 | 60.6 |
| Muse Glimmer | 40.7 | 12.4 | 44.1 | 21.6 |
| Qwen3.8-27B | 29.8 | 12.0 | 67.8 | 44.6 |
Opus's headline 53.5% success is an equal-weight average of the three environment rates, not the fraction of all 315 episodes completed. That distinction matters because cooking supplies more episodes than either other environment. Argon is close behind at a calculated 53.4% using the rounded table entries.
The individual columns are more informative than that narrow aggregate lead. Astra leads phone success among these peers, while Argon and Sol tie on household success. Household completion remains low even for the strongest entries. Opus trails Astra, Sol and Flash on cooking interaction quality despite completing more cooking tasks than those three models.
Muse makes the mismatch especially visible: its cooking success slightly exceeds Astra's, but its quality score is 21.6%, against Astra's 76.0%. A user can finish a task despite weak assistance. Completion alone cannot tell a buyer whether the model gave accurate, helpful instructions.
Latency changes the ordering
The measured-delay experiment lets queued user actions continue while the assistant responds. In both cooking and household environments, a tick represents 1.2 seconds. Measured response time is divided by that duration and rounded upward, then capped at 30 ticks in cooking and 15 in household tasks. ScreenSim stays in its original action–response setting. Appendix H (opens in new tab) describes these timing rules.
Return to the raw-patty example. If the user proceeds while the assistant reasons, the warning may arrive after commitment. The assistant may still identify the error and receive detection credit without preventing it. The researchers generate new interactions under each timing condition, allowing late advice to change the subsequent trajectory rather than merely subtracting a latency penalty from an existing score.
Here is the same peer selection under measured delay. Arrows run from fixed-delay to measured-delay success; the last column is measured-delay cooking quality. Figures come from Table 4 (opens in new tab).
| Assistant | Cooking success, fixed → measured | Household success, fixed → measured | Cooking quality, measured |
|---|---|---|---|
| Gemini 4 Argon | 60.9 → 33.1 | 15.6 → 12.9 | 57.9 |
| Claude Opus 5.5 | 57.1 → 31.8 | 14.2 → 5.3 | 46.9 |
| Gemini 3.8 Flash | 46.4 → 29.8 | 14.7 → 10.7 | 66.9 |
| GPT-6.1 Sol | 42.9 → 30.0 | 15.6 → 8.0 | 68.4 |
| GPT-6 Astra | 40.2 → 28.4 | 10.7 → 8.0 | 68.4 |
| GPT-Realtime 2.1 | 33.3 → 22.0 | 11.6 → 10.2 | 47.1 |
| Gemini 3.8 Live | 26.2 → 22.2 | 11.6 → 10.2 | 62.4 |
| Muse Glimmer | 40.7 → 42.7 | 12.4 → 9.3 | 21.2 |
| Qwen3.8-27B | 29.8 → 27.1 | 12.0 → 8.9 | 43.0 |
All five frontier API models lose success in both environments. In cooking, Muse overtakes Argon, Sol edges above Flash, and Live edges above GPT-Realtime. These are observed mean orderings, not demonstrated statistically significant advantages.
The prevention figures help explain the shift. Argon's cooking error prevention falls from 62.3% to 39.9%; Opus falls from 67.0% to 43.6%. Their mean observed cooking response times are 6.5 and 6.4 seconds, respectively, versus Muse's 1.5 seconds. Those means precede tick conversion and capping. Table 20 (opens in new tab) reports the measurements.
None of this makes Muse the best general assistant. Its measured cooking quality remains 21.2%. Nor are these timings client-facing service guarantees: local models use H100s, API routes differ, and Opus is accessed through OpenRouter with network and queue delays included. Real-time API models are polled every two ticks rather than evaluated with fully model-initiated turn-taking. Non-streaming models use an eight-observation window and recurrent Plan–Act memory in cooking and household tasks. User-model inference and rendering do not advance the simulation clock. The harness measures a particular operating setup.
What the human study validates
The reported 78.5% agreement concerns the quality metric's alignment with preferences. It is not a task-success rate or a real-world safety result.
Forty-five participants used browser-based CookSim on their own computers, controlling the cook by keyboard and talking to assistants through text. Each completed four unassisted episodes, then nine assisted episodes over three rounds with three anonymously labeled assistants. Assistant–recipe assignments rotated through a Latin-square design. Participants ranked their experience after each round and could continue beyond the benchmark deadline. The collection used model versions available then, rather than simply repeating the main frontier scorecard. Appendix F (opens in new tab) provides the protocol.
The agreement unit is one participant's comparison of two assistants, with a non-tied human preference and available metric scores. Human ties are excluded; metric ties receive half credit. The paper does not report the retained comparison count or a confidence interval. Its pooled Kendall concordance is +0.57. That is evidence that the rubric tracks something participants value within this study, not validation of every simulator assumption.
One mismatch is consequential: on a common elapsed-time basis, simulated users produce 11.70 utterances per 100 ticks against humans' 0.39, a 30-fold difference. Parallel cooking behavior also matches less well. Persona differences depend on prompts, while human trust groups are assigned through an LLM-assisted conversation audit. Actual people may ignore a warning, remain silent or divide their attention differently. The study took place entirely in a browser, not around heat, knives or real food.
Training improves success while missing more errors
The team also trained Qwen3.5-4B with GRPO in CookSim for 12 updates. Its reward combines success with a prevention bonus and penalties for false flags and excess turns. Table 5 (opens in new tab) reports the following validation results as mean ± standard deviation. Each before/after pair belongs to its own simulated-user evaluation.
| Simulated user | Checkpoint | Reported success | Detection recall | Interaction quality |
|---|---|---|---|---|
| Gemini 3.7 Flash | Before RL | 33.3 ± 3.3 | 49.0 ± 11.9 | 29.6 ± 4.8 |
| Gemini 3.7 Flash | After RL | 42.2 ± 6.9 | 34.9 ± 6.7 | 31.1 ± 0.4 |
| Gemini 3.8 Flash | Before RL | 64.4 ± 6.9 | 43.0 ± 14.5 | 24.3 ± 2.1 |
| Gemini 3.8 Flash | After RL | 67.8 ± 7.7 | 28.7 ± 3.4 | 30.2 ± 0.3 |
Success and quality rise within each evaluation, while detection recall falls. Penalizing false alarms can encourage more selective flagging; that explanation is the authors' hypothesis, not a proven causal decomposition.
The experiment is narrower than the table initially suggests. Its 120 training and 30 validation scenarios share recipe/layout cells, with no separate frozen cooking test set. Appendix C specifies native CookSim won outcomes for this experiment. Original quality judgments cover only 9 of 90 validation episodes per checkpoint across two passes, while the Gemini 3.8 complete-trace evaluation judges all 90. Recall definitions also differ between the evaluations. Their absolute levels should not be compared as a clean user-model experiment. Appendix C (opens in new tab) discloses those qualifications.
This supports a learning signal in the simulator, not generalization to unfamiliar kitchens or real users. A separate phone-training experiment also fails to establish a reliable frozen-test improvement and shows lower GPT-judged quality for both user settings.
A useful test for assistant builders
The practical implication is to evaluate interventions, not just answers. An assistant team can measure whether warnings arrive before commitment, whether the user changes course, and whether recovery preserves the requested goal. Those outcomes belong beside factual quality and false-alarm rates, not underneath a single completion percentage.
The public repository (opens in new tab) supplies environments and evaluation documentation. Its current implementation snapshot should be kept distinct from the historical revision and execution conditions behind the paper's numbers.
For a buyer, the tables argue against choosing solely on either reasoning scores or advertised response speed. Ask for task-specific closed-loop evidence under the intended inference route, then test with actual users. A deployment decision still needs inference cost, interruption burden and actual user outcomes; simulator success alone does not establish return on investment. The failure to watch for is specific: advice that was correct for the state the model saw, but no longer useful when the user receives it.

