Over the last year, we’ve run trillions of tokens on some of the hardest computer-use tasks in insurance.

Pulling policy documents means logging into a carrier portal, finding the right customer, searching through years of files, and downloading requested reports. Recovering a deductible means reading an inbound request, finding the hidden claim in one system, checking it against the source policy in another, then updating the case.

These workflows are easy to describe and hard to automate. They run across systems with no APIs, mix repetitive data entry with real judgment, and can stretch from a few minutes to days of work.

In this post, we’ll share what we learned building computer use agents that can complete these operations reliably at scale.

Visual Agents

Our first computer use agents were visual. Just like humans, the model took a screenshot, chose a mouse or keyboard action, and looped until completion. These agents worked in demos but failed in production.

Take a document pull. Only one part of the task really needs intelligence: deciding which reports to retrieve. The rest is clicking through a carrier portal. Our visual agents treat these steps equally, spending their reasoning on navigation and burning through their context windows before reaching the decision point.

We knew models and context windows would improve, but our harness was fundamentally flawed: even successful runs were expensive and slow because agents reasoned through the same mechanics every time.

Playwright MCP

Next, we tried Playwright MCP. Instead of screenshots and pixels, the model inspects and acts on a text-based page snapshot.

- navigation:
  - link "Dashboard"
  - link "Policies"
  - link "Documents"
  - link "Claims"
- main:
  - heading "Policies" [level=1]
  - textbox "Search policies": Riverside Builders
  - table:
    - row "WC-882401 2024–25 · Active"
    - row "WC-882400 2023–24 · Expired"
    - row "GL-771203 2024–25 · Active"

For carrier pulls, our Playwright agent could now identify the right policy and retrieve documents with 5 times fewer tokens: each page snapshot cost half as much as a screenshot, and the agent needed far fewer of them. Since context was no longer bloated, this opened room for the reasoning work that actually mattered, improving precision and accuracy on long-horizon work.

But the model was still driving the browser one action at a time. A routine form could require 20+ tool calls: snapshot the page, click a field, type a value, inspect again, and repeat. The model clearly understood the page, but it wasn’t acting on this information.

Code Mode

In February, we got early access to a new frontier model, and gave it a few computer use tasks inside a coding agent. To our surprise, it solved complicated flows that our purpose-built harness still struggled with.

Digging into the traces, we noticed how often the agent wrote ad hoc scripts to batch sets of actions. We realized the model had become more limited by our harness than by its capabilities.

Our harness became one tool call: run_code. We gave the agent skills for Playwright and Windows Accessibility APIs, and then let it write and execute its own code.

Once the agent solved a repeated piece of work, we could turn that code into a reusable script. Agents still handled real-time judgment like identifying the right policy or resolving an ambiguous result, but repeated mechanics like logging in, filling forms, and downloading files became fast, cheap, and deterministic.

const page = await login()
for (const doc of newDocs) {
  await download(doc)
  ledger.append(doc)
}

skills/
  amtrust-policies/
    login.js · pull.js
  travelers-direct-bill/
    login.js · pull.js
  ⋯ 12 more carriers

Continual Learning

Code Mode lets us perform work with fast and efficient scripts. But what happens when our scripts run into edge cases?

During one run for a carrier pull, a script failed because it found the requested documents but could not pass them to a downstream sub-agent. The documents existed in the browser sandbox but weren’t in the agent’s own sandbox.

To solve failures like this, we allow our agent to edit the code of the harness itself. Instead of stopping, the agent inspected the environment, wrote code to sweep the browser sandbox, synchronized the files into its own workspace, and continued the task.

Now, all future runs inherit the fix.

Results

As more of the workflow is codified in code, we can use smaller models to orchestrate the known path. We keep frontier models as advisors for edge cases and judgement calls.

Compared to visual agents, optimized code mode agents run 30x cheaper while maintaining higher rates of completion. We benchmark the three harnesses across frontier and budget models on popular insurance tasks below:

Conclusion

Our work on computer use represents the type of challenges we tackle at Pace: embedding model intelligence into the durable systems surrounding them.

By simplifying the harness, representing the action space as code, and carrying successful repairs forward, our system improves with every run instead of starting from scratch.

The ideas described in this post are just the beginning of what we’re exploring, including:

• A shared skill layer, so that solving a portal once solves it for every customer, while their data and approvals stay their own

• Synthetic environments at scale that can replay workflows, generate edge cases, and grade outcomes automatically

If any of this interests you, please reach out. We’re hiring.