Agent S4: evolving the interfaces computer-use agents work through

Computer-use agents are limited by the interfaces humans give them. An agent may type a spreadsheet cell by cell when a single line of code would do, or print all two thousand lines of a file through a shell to find one function.

Both the interfaces and the knowledge of when to use each are built by hand. That causes three problems.

1. Progress depends on manual work. Every new kind of task or model release exposes new failure modes. A researcher reads trajectories, diagnoses failures, writes new primitives and ships a new framework. This loop does not keep pace with the variety of tasks.
2. Agents spend compute working around the interface. An agent may finish a shell task by clicking open a terminal, typing commands one character at a time and reading the output from screenshots. A direct shell interface would be faster and more reliable.
3. Learned skills inherit the same limit. Methods that turn experience into reusable skills still compose the preset actions and observations. They can reorder what the interface allows, but they cannot extend it.

Meet Agent S4

Today we're introducing Agent S4, Simular's autonomous computer-use scientist. Agent S4 does not solve tasks itself. It studies computer-use agents and improves the interfaces they work through, using population-based evolution.

The name reads as Agent Scientist for (4) Evolving Agent Interfaces. Its output is a repository of interfaces, together with knowledge of when and how to use each one. The task agent's model and harness stay fixed. Only the repository changes.

What we mean by an interface

An interface is how an agent sees a computer and acts on it. It defines which requests the agent can make and what comes back. A GUI interface accepts clicks and keystrokes and returns screenshots. A shell interface accepts commands and returns text.

The interface changes how hard a task is. To update a spreadsheet through screenshots, an agent must find the visible cells, type each value and check the display afterwards. A spreadsheet interface can return the relevant rows and formulas, accept one bulk edit and report the changed values.

Agents already reach for code when they can. In the OSWorld V2 analysis, terminal commands make up 24.7% of agent actions, and GPT-5.5 spends 78% of its budget on code, API or file operations. Under a fixed interface, each of those programs is typed through the desktop and its result is read off a screenshot.

How it works

Agent S4 runs two kinds of agent. A task agent attempts tasks using a repository. A meta-agent reads the task agent's trajectories and rewrites the repository.

1. Select. Parents are drawn from a population of repositories. In each pairing, the one with the higher validation score wins.
2. Examine. The meta-agent studies the parents' trajectories on synthetic training tasks. They carry no success labels, so it reasons from what the agent did and saw. Repeated cell-by-cell edits might suggest a bulk spreadsheet interface. A helper the agent keeps misusing might suggest better usage notes.
3. Write. It builds a new repository from one parent (mutation) or two (crossover). It can add, revise or remove interfaces and knowledge. A crossover can choose between overlapping interfaces or port a capability from one parent to the other.
4. Evaluate. The task agent runs with the new repository on the training tasks and on separate validation tasks. The new trajectories become evidence for later rounds.
5. Add. The new repository joins the population next to its parents, whatever its score. Earlier designs stay available to revisit and recombine.

The meta-agent never sees the validation tasks, their trajectories or their scores. Validation only decides which repositories get developed further. This keeps the repository from turning into an answer sheet for the tasks it was built from.

One OSWorld V2 population run with a scratch VM, over 30 steps. Blue boxes are mutations and orange boxes crossovers. The green path traces the selected repository. It is a step-24 crossover of the repositories from steps 2 and 6, and it scored 50.2% partial on validation, 14.4 points above the seed.

Results on OSWorld V2

With the repository Agent S4 evolved, Qwen3.8-27B reaches 21.3% binary success on the full 108-task OSWorld V2 benchmark. That is the highest binary success among open-source computer-use agents and 3.8 times the same model with fixed interfaces. It is also above GPT-5.5, at about 4.3% of its cost.

What evolution builds

The evolved repositories contain interfaces no one specified. We label each one by the part of the computer it covers. An interface can carry more than one label.

Most growth deepens interfaces the agent already has. On OSWorld V2 without a scratch VM, evolution added 4 new interfaces and 127 extensions to existing ones. With a scratch VM to try its code on, the meta-agent built 12 new interfaces and 253 extensions, including interfaces for musical scores, audio projects and 3D scenes.

Beyond OSWorld: scientific software

We ran a fresh evolution on ScienceBoard with the same task agent and seed repository. ScienceBoard has 169 tasks in six scientific applications. They are Celestia for astronomy, ChimeraX for molecular visualization, GRASS GIS for geographic analysis, KAlgebra for algebra, Lean for theorem proving and TeXstudio for typesetting. General computer-use interfaces rarely cover them. The meta-agent learned from 48 synthetic training tasks.

The evolved repository solves 99 of 169 tasks, against 93 for the seed and far more than Qwen CUA with its fixed interface. Its new functions include a ChimeraX command interface, GRASS GIS modules that run in the live session and a KAlgebra calculator that evaluates many expressions in one call.

What we learned about evolving interfaces

Synthetic experience works! The meta-agent never sees a benchmark task. It learns only from synthetic training tasks with no success labels, written by a separate pipeline that checks they do not overlap with the benchmark.

Testing code while building it helps complete more subtasks. With a scratch VM, the meta-agent can open applications and try its interfaces. That run built three times as many new interfaces and reached the best partial score. The best binary success came from the run without it.

Keeping a population pays off. The best repository with a scratch VM was a crossover of two early repositories, built 18 steps after the later of them. The population kept both available long enough to combine them.

Why it matters

Interfaces are an axis of improvement separate from the model and the harness. Agent S4 improved a 27B open model without touching either, and the same approach can run on top of stronger ones. Because interfaces live in a repository, they can keep evolving as models, harnesses and applications change.

Read the paper, Evolving Agent Interfaces, on arXiv. The code is open source in the Agent S repository.