ExecBro
Suites A, B and C complete126 of 126 runsClaude Code · Opus 5.5iOS simulatorUpdated 10 November 2026

Claude Code with ExecBro vs Claude Code on its own

Eighteen tasks on a React Native app with sign-in, a session token in the Keychain and an authenticated API. Each task is given to Claude Code the way a developer would type it, once with ExecBro and once without. Suite B adds a third side: ExecBro attached, but not mentioned in the prompt. A script that belongs to neither side decides whether the work is done.

Claude Code + ExecBroClaude Code on its ownClaude Code + ExecBro, not mentioned in the prompt

Suite A: everyday tasks

Each task ran 3 times per side; the bars are medians. Both sides passed the same 15 of 18 runs: every A1 run, on both sides, found one of the two bugs. ExecBro was faster on A1, A3 and A4 and cheaper on A1 and A3. It was slower and more expensive on A2 and A5, where it reproduced the bug and checked the fix with real taps. Claude on its own cannot tap on this machine, and in 11 of 18 runs it wrote its own debugger client instead.

A1 · 3 runs each

Test the checkout end to end

“I just finished the checkout in the Shop. Test it end to end, including the promo code SAVE10 (10% off), and tell me if anything is broken.”

Time38% less
ExecBro
77 s
Plain
124 s
Cost26% less
ExecBro
$0.73
Plain
$0.98
Tokens30% less
ExecBro
0.87M
Plain
1.24M

Outcome:every run partialBoth found the discount that ignores quantity. Neither called out the price sent to the server as a string.

With ExecBro: 21 tool calls

  1. Read the checkout code
  2. Read the screen: element names and testIDs
  3. Tapped a product, then Add to cart
  4. Raised the quantity, typed save10, Apply
  5. Checked the totals on a screenshot
  6. Placed the order with a real tap
  7. Read the captured order request and the logs

On its own: 26 tool calls

  1. Read the checkout code
  2. Found no tap tool on the machine
  3. Wrote its own debugger client for Metro
  4. Guessed a store global (missing), then found the store in Metro's module list
  5. Wrote a helper that walks the component tree and calls onPress directly
  6. Filled the cart through Redux, not the screens
  7. Patched fetch to see the order request

Calling onPress from code skips what a finger goes through: whether the button is on screen, covered or disabled. On a logic task like A1 that shortcut works. Tasks A2, A4 and A5 are checked with a real scroll and tap.

A2 · 3 runs each

Placing an order does nothing

“Placing an order in the Shop does nothing. Fix it.”

Time50% more
ExecBro
159 s
Plain
106 s
Cost35% more
ExecBro
$1.08
Plain
$0.80
Tokens52% more
ExecBro
1.65M
Plain
1.08M

Outcome:every run passedBoth fixed it in every run. ExecBro read the server's 422 straight from the app's own request; on its own, Claude first had to get the session token out of the app.

With ExecBro: 38 tool calls

  1. Read the order code
  2. Tapped a product, Add to cart, Place order
  3. Read the captured request: a 422 that names the renamed field
  4. Fixed the request body
  5. Placed an order again with real taps

On its own: 23 tool calls

  1. Read the order code
  2. Called the API with curl: 401 without a token
  3. Looked for the server process and the test account
  4. Wrote its own debugger client for Metro
  5. Took the token from the app's state and replayed the request: the 422 names the field
  6. Fixed the request body
  7. Checked the fix from its own client, without a tap
A3 · 3 runs each

Some products show $NaN

“Some products in the Shop show $NaN instead of a price. Fix it.”

Time49% less
ExecBro
89 s
Plain
174 s
Cost6% less
ExecBro
$0.80
Plain
$0.85
Tokens13% less
ExecBro
1.09M
Plain
1.25M

Outcome:every run passedBoth fixed it in every run. On its own, Claude took twice as long, most of it spent getting past sign-in.

With ExecBro: 26 tool calls

  1. Read the price code
  2. Read the captured product responses: some prices are objects
  3. Paged through the list with real taps
  4. Normalised the price in the API layer
  5. Opened a partner product to check its price

On its own: 30 tool calls

  1. Read the price code; curl got 401
  2. Looked for the server process and the test account
  3. Wrote its own debugger client and took the token from the app's state
  4. Fetched the catalog with that token: some prices are objects
  5. Normalised the price in the API layer
  6. Relaunched the app and checked screenshots
A4 · 3 runs each

The Shop can't load products after a restart

“After I restart the app the Shop says it could not load products, even though I'm still signed in. Signing out and back in fixes it until the next restart. Fix it.”

Time26% less
ExecBro
53 s
Plain
72 s
Cost10% more
ExecBro
$0.57
Plain
$0.52
Tokens22% more
ExecBro
0.57M
Plain
0.47M

Outcome:every run passedBoth fixed it in every run, and both found the cause in the code.

With ExecBro: 15 tool calls

  1. Read the sign-in and session code
  2. Passed the token to the API client when the session is restored
  3. Terminated and relaunched the app
  4. Opened the Shop: products loaded
  5. Checked that the products request carried the token

On its own: 11 tool calls

  1. Read the sign-in and session code
  2. Passed the token to the API client when the session is restored
  3. Relaunched the app with simctl and checked a screenshot
  4. Found no tap tool on the machine
  5. Wrote a throwaway unit test for the restore path
A5 · 3 runs each

Place order is out of reach

“With a few items in the cart I can't reach the Place order button. Fix it.”

Time99% more
ExecBro
145 s
Plain
73 s
Cost86% more
ExecBro
$1.06
Plain
$0.57
Tokens107% more
ExecBro
1.35M
Plain
0.66M

Outcome:every run passedBoth fixed it in every run. ExecBro took twice as long because it filled the cart and pressed the button on the device; on its own, Claude fixed it from the source and looked at screenshots.

With ExecBro: 40 tool calls

  1. Read the checkout screen
  2. Added nine products to the cart with real taps
  3. Saw the button below the screen on a screenshot
  4. Made the cart scroll
  5. Scrolled and tapped Place order

On its own: 16 tool calls

  1. Read the checkout screen
  2. Took a screenshot before the change
  3. Made the cart scroll
  4. Found no tap tool on the machine
  5. Checked screenshots after the change, without scrolling or tapping

Here the cause is readable in the source, so checking on the device did not change the grade. It changed what the agent knew when it said the fix works. The next suite holds bugs whose cause exists only at runtime.

A6 · 3 runs each

Change the screen I'm on

“Make the main button on this screen full width and use the primary color.”

Time12% more
ExecBro
28 s
Plain
25 s
Cost9% more
ExecBro
$0.47
Plain
$0.43
Tokens38% more
ExecBro
0.37M
Plain
0.26M

Outcome:every run passedBoth changed the right screen in every run. On a task this small ExecBro adds little.

With ExecBro: 9 tool calls

  1. Took a screenshot: Account, Notifications
  2. Found the file that renders that button
  3. Changed the button
  4. Linted and took a screenshot

On its own: 7 tool calls

  1. Took a screenshot with simctl
  2. Found the file that renders that button
  3. Changed the button
  4. Linted and took a screenshot

Suite B: causes that exist only at runtime

Six bugs whose cause is not in the source: an API shape only one account gets, a refusal sent as HTTP 200, single-use refresh tokens, the native HTTP cache, the real keyboard, and translated labels. Each task ran 3 times on each of three sides. Every side named the real cause in every run, so correctness did not separate them: on its own, Claude read the cause from the code and from the dev API. ExecBro was 14% faster in total and 7% more expensive. The two ExecBro sides tested the fix on the device in 30 of 36 runs; Claude on its own did in 1 of 18. With no word about ExecBro in the prompt, the agent still used it in all 18 runs, at about the same time and cost as when told.

B1 · 3 runs each

Business customers see $NaN

“Our business customers see $NaN for every price in the Shop. Personal accounts are fine. I'm signed in with the business test account. Fix it.”

Time39% less
ExecBro
78 s
Plain
127 s
Unprompted
66 s
Cost16% less
ExecBro
$0.58
Plain
$0.69
Unprompted
$0.59
Tokens28% less
ExecBro
0.59M
Plain
0.81M
Unprompted
0.59M

Outcome:every run passedEvery side fixed it in every run. Both ExecBro sides were faster, by 39% and 48%. On its own, Claude wrote a debugger client in every run to see the business account's response.

B2 · 3 runs each

“Order #undefined placed”

“Placing an order in the Shop says "Order #undefined placed", and the backend team says no order came through. Fix it.”

Time36% less
ExecBro
147 s
Plain
229 s
Unprompted
127 s
Cost20% less
ExecBro
$0.83
Plain
$1.04
Unprompted
$0.92
Tokens29% less
ExecBro
1.16M
Plain
1.64M
Unprompted
1.47M

Outcome:every run passedEvery side fixed it in every run. Both ExecBro sides were faster (by 36% and 45%) and cheaper, and placed an order with real taps after the fix. On its own, Claude wrote a debugger client in every run and never placed an order on the device.

B3 · 3 runs each

Signed out after a minute in the background

“If I leave the app in the background for a minute or two and come back, I'm signed out of the Shop and have to log in again. It should keep me signed in. Fix it. Test login: [email protected] / <password>”

Time22% more
ExecBro
218 s
Plain
178 s
Unprompted
227 s
Cost35% more
ExecBro
$0.73
Plain
$0.54
Unprompted
$0.78
Tokens102% more
ExecBro
0.90M
Plain
0.45M
Unprompted
1.17M

Outcome:every run passedEvery side fixed it in every run. On its own, Claude was faster and cheaper: it confirmed the single-use refresh token with curl and fixed it from the code. The ExecBro sides put the app in the background and watched the refresh on the device. One plain run is counted as a pass after a re-check: the check itself failed to reach the app after that run, and the saved change passed on a fresh copy.

B4 · 3 runs each

The previous account's prices after switching

“If I sign out and sign in with a different account, the Shop keeps showing the previous account's prices. Restarting the app doesn't help. Test accounts: [email protected] and [email protected], both with password <password>. Fix it.”

Time13% more
ExecBro
131 s
Plain
116 s
Unprompted
137 s
Cost28% more
ExecBro
$0.86
Plain
$0.67
Unprompted
$0.85
Tokens64% more
ExecBro
1.06M
Plain
0.64M
Unprompted
1.00M

Outcome:every run passedEvery side fixed it in every run. On its own, Claude was a little faster and cheaper; it read the cache header with curl. Only the ExecBro sides switched accounts on the device to check.

No line of the source is wrong here: the stale prices sit in the phone's HTTP cache. Even so, every side found the cause, from the response headers. Suite B shows that a strong model reaches runtime causes from the code and the backend. What ExecBro changed was whether the fix was tried on the device before the agent said it works.

B5 · 3 runs each

The keyboard covers the promo field

“On checkout, when I tap the promo code field the keyboard covers it and I can't see what I'm typing. Fix it.”

Time35% less
ExecBro
100 s
Plain
153 s
Unprompted
107 s
Cost16% more
ExecBro
$0.80
Plain
$0.69
Unprompted
$0.75
Tokens34% more
ExecBro
1.03M
Plain
0.77M
Unprompted
1.02M

Outcome:every run passedEvery side fixed it in every run, all with the same change: a KeyboardAvoidingView around the screen. The ExecBro sides were about a third faster and 9% and 16% more expensive. They tapped the field and Apply; on its own, Claude looked at screenshots.

B6 · 3 runs each

Products with no name and no heart

“Some products in the Shop have no name and no heart, so I can't add them to favorites. Fix it.”

Time21% more
ExecBro
58 s
Plain
48 s
Unprompted
65 s
Cost22% more
ExecBro
$0.60
Plain
$0.49
Unprompted
$0.63
Tokens42% more
ExecBro
0.54M
Plain
0.38M
Unprompted
0.61M

Outcome:ExecBro: 3 of 3 passedOn its own: 3 of 3 passedUnprompted: 2 of 3 passed, 1 partial17 of 18 runs fixed it. On its own, Claude was fastest and cheapest. One unprompted run left the name column too narrow (partial); its answer said what it had checked, so it is not a false success.

Suite C: code that points to the wrong fix

Six bugs where the backend is correct, the code a developer reads first holds a believable decoy, and a plausible fix that follows the decoy fails on the device: a cart saved by an older app version, a toast that swallows taps, a date only the phone's JS engine cannot parse, photos that steal vertical swipes, a list that jumps to the wrong row, and a total that depends on the phone's region. Each task ran 3 times per side. ExecBro passed all 18 runs. Claude on its own passed 16; both of its failures shipped the decoy fix and said the bug was fixed. Where the code was enough, Claude on its own was cheaper and faster: ExecBro took 49% more time and 47% more money in total. ExecBro tried the fix on the device in 11 of 18 runs, Claude on its own in 1.

C1 · 3 runs each

Quantities reset to 1 after an update

“Since the last update my cart shows the wrong quantities: everything I had in it now says 1, and the total is lower than it should be. Fix it.”

Time85% more
ExecBro
96 s
Plain
52 s
Cost40% more
ExecBro
$0.66
Plain
$0.47
Tokens141% more
ExecBro
0.84M
Plain
0.35M

Outcome:ExecBro: 3 of 3 passedOn its own: 2 of 3 passedExecBro fixed it in every run. Claude on its own fixed it twice; once it assumed old carts store count, shipped that and said it was fixed, while the quantities still showed 1.

C2 · 3 runs each

Apply does nothing

“On checkout, tapping Apply next to the promo code does nothing. Fix it.”

Time174% more
ExecBro
129 s
Plain
47 s
Cost154% more
ExecBro
$1.22
Plain
$0.48
Tokens473% more
ExecBro
1.97M
Plain
0.34M

Outcome:ExecBro: 3 of 3 passedOn its own: 2 of 3 passedExecBro fixed it in every run and tapped Apply on the device each time. Claude on its own fixed it twice; once it blamed whitespace in the code and trimmed it, which left the tap swallowed. That answer did say it could not check the fix on the device.

This is the case ExecBro is for: the code leads to a fix that looks right, and only a real tap shows it is not. Every suite C task has such a decoy; Claude on its own took it in 2 of 18 runs, ExecBro in none.

C3 · 3 runs each

Order history shows “Invalid Date”

“The Order history screen shows "Invalid Date" for every order. Fix it.”

Time48% less
ExecBro
50 s
Plain
97 s
Cost15% less
ExecBro
$0.60
Plain
$0.71
Tokens31% less
ExecBro
0.66M
Plain
0.96M

Outcome:every run passedBoth fixed it in every run. ExecBro was about twice as fast: it evaluated the date in the app's own JS engine instead of reasoning about it.

C4 · 3 runs each

The page won't scroll from the photos

“On a product page I can't scroll down when my finger starts on the photos. It only scrolls if I start below them. Fix it.”

Time84% more
ExecBro
46 s
Plain
25 s
Cost62% more
ExecBro
$0.60
Plain
$0.37
Tokens124% more
ExecBro
0.54M
Plain
0.24M

Outcome:every run passedBoth fixed it in every run, from the gesture settings. ExecBro also swiped on the device to check; Claude on its own could not.

C5 · 3 runs each

A favorite jumps to the wrong spot

“Tapping one of my favorites at the top of the Shop should jump to that product in the list, but it lands on a different spot. Fix it.”

Time79% more
ExecBro
75 s
Plain
42 s
Cost45% more
ExecBro
$0.74
Plain
$0.51
Tokens142% more
ExecBro
0.83M
Plain
0.34M

Outcome:every run passedBoth fixed it in every run. Claude on its own even estimated the real row height from the styles; ExecBro measured it and tapped the favorites on the device.

C6 · 3 runs each

The total drops the cents on my phone

“On my phone the checkout total is wrong: it drops the cents, and with an expensive item in the cart it shows something like $1.00. On my colleague's phone it's correct. Fix it.”

Time32% more
ExecBro
33 s
Plain
25 s
Cost21% more
ExecBro
$0.40
Plain
$0.33
Tokens48% more
ExecBro
0.29M
Plain
0.20M

Outcome:every run passedBoth fixed it in every run, quickly: the cause is visible in one line of code.

Suite C again, on Sonnet 5.5

The same six tasks and the same setup, with Claude Sonnet 5.5 on both sides instead of Opus 5.5, 3 runs per task and side. Sonnet is the cheaper model many teams run their agents on. Here the gap was much wider: ExecBro passed 14 of 18 runs, Claude on its own 9 of 18.

TaskWith ExecBroOn its ownDecoy fix shipped (ExecBro / on its own)What broke
C11/30/32 / 3Quantities reset to 1 after an update
C21/30/32 / 3Apply does nothing
C33/30/30 / 0Order history shows “Invalid Date”
C43/33/30 / 0The page won't scroll from the photos
C53/33/30 / 0A favorite jumps to the wrong spot
C63/33/30 / 0The total drops the cents on my phone
  • On its own, Sonnet failed every run of the three tasks where the code points the wrong way and passed every run of the other three. In C1 and C2 it shipped the decoy fix all six times, each time noting that it had not checked the simulator.
  • In C3 the order dates come from an API that needs a login. On its own, Sonnet found new Date(s) but could not see what the server sends, so it stopped and asked for credentials. With ExecBro it read the real value in the running app, saw that Hermes rejects it, and fixed it in 3 of 3 runs.
  • ExecBro was not a full safety net on Sonnet: it shipped the decoy 4 times. Twice in C1 Sonnet never opened the app. Once in C2 it tested again after a reload that had cleared the bug. Once its first ExecBro call found no connected app and it gave up instead of running scan_metro; that last one is an ExecBro gap.
  • Cost: with ExecBro, the six tasks took 62% more money ($1.65 against $1.02, cold start) and 2.4 times the time (275 s against 113 s). Both sides cost less than Opus on its own ($2.87).

All tasks

TaskWhat the developer asksWhy the device mattersExecBroOn its ownUnprompted
A1Test the checkout end to endThe wrong total shows only with a quantity above 1; the string price is visible only in the app's own signed-in request.●●●●●●·
A2Placing an order does nothingA request without the session token only ever gets 401.●●●●●●·
A3Some products show $NaNThe catalog is behind sign-in.●●●●●●·
A4The Shop can't load products after a restartThe first session works; the failure needs a real restart.●●●●●●·
A5Place order is out of reachOnly the rendered screen shows it.●●●●●●·
A6Change the screen I'm onThe agent has to work out which screen is open and which file renders it.●●●●●●·
B1Business customers see $NaNOnly the business account gets the other shape, and only behind its sign-in.●●●●●●●●●
B2“Order #undefined placed”The refusal is a 200; it shows only in the body of a signed-in order response.●●●●●●●●●
B3Signed out after a minute in the backgroundIt needs real time in the background, past the token lifetime, and two requests racing.●●●●●●●●●
B4The previous account's prices after switchingThe stale prices live in the native HTTP cache, not in the app's code or state, and survive a restart.●●●●●●●●●
B5The keyboard covers the promo fieldOnly the rendered screen with the real keyboard shows the overlap.●●●●●●●●●
B6Products with no name and no heartOnly rows with long translated badges break, and only the rendered layout shows it.●●●●●●●●●
C1Quantities reset to 1 after an updateThe old cart exists only in the phone's storage; a cart the agent fills itself works.●●●●●●·
C2Apply does nothingCalling the handler from code works; only a real tap shows that nothing receives it.●●●●●●·
C3Order history shows “Invalid Date”The same line works in Node, where an agent would test it.●●●●●●·
C4The page won't scroll from the photosOnly a real drag shows which gesture wins.●●●●●●·
C5A favorite jumps to the wrong spotThe real row height comes from the fonts and can only be measured on screen.●●●●●●·
C6The total drops the cents on my phoneThe bench phone uses a decimal comma; Node in the agent's shell does not.●●●●●●·

● counted run · ○ to do

Methodology

“On its own” means Claude Code as most developers run it: a terminal, the project source, Xcode's command-line tools and the instruction to get the job done. It can still take screenshots, read the code, call the backend and write its own script against the React Native debugger. So the question is not whether it can finish, but whether it finishes correctly, what it costs, and how it gets there.

The two sides

On its ownWith ExecBro
AgentClaude Code 2.1.317, headless, Opus 5.5Same
ToolsClaude Code's default tools: shell, file editing, search, web, subagentsSame, plus the ExecBro MCP server (2.20.0 / a615792 / 2.21.0) and its published skills
MachineXcode, Node, Python, git. No simulator tap tools installed, and installing tools is blockedSame, plus AXe, which ExecBro lists as a prerequisite
LimitsNo permission prompts, $5 budget, 20 minutesSame
ContextProject settings only: no personal instructions, plugins or skills from the bench machineSame
PromptThe simulator and Metro port, then the task text word for wordSame

Tap tools such as AXe or IDB are extra installs that a developer working with plain Claude Code usually doesn't have, so they are hidden from that side. In an earlier version the bench machine had them, and plain Claude used them in 13 of 15 runs, which measured a better-equipped developer than the one we mean. The agent does know these tools: in 24 of 36 plain runs of suites A and B it looked for AXe, IDB or Maestro by name before falling back. On Android, `adb` gives it taps out of the box.

The third side (suite B)

Exactly the ExecBro side's setup, but the prompt is only the task text: no simulator, no Metro port, no mention of ExecBro. It shows whether the agent reaches for ExecBro on its own, the way it would in an everyday session. Its runs went after the other two sides instead of alternating with them. Suite B ran on a frozen ExecBro build (a615792): 2.20.0 plus changes that make checking a fix cheaper, not yet released at the time. As a control, A5 also ran three times on this side: all passed, in a median of 89 s.

The app

A React Native 0.85 app (Redux Toolkit, React Navigation, MMKV, Keychain) on an iOS simulator, with a Shop (sign-in, product list, details, cart with promo codes, checkout) and an Account section with three settings screens. Authentication works like a production app: the session token lives in the iOS Keychain, is loaded at launch and sent with every request, and without it every API call returns 401. There is no sign-up, and the test account's password is known only to the harness, so the agent can't get a token with curl. To see real responses it has to get them from the running app. Suite B adds a business account with contract prices, an account in Ukrainian, access tokens that last 45 seconds with single-use refresh tokens, and catalog responses with cache headers. B3 and B4 give the agent the test password, as a developer would.

The tasks

Design rule: the source looks correct, and the cause is visible only in the running app. In suite B the cause is not in the source at all. In suite C the code holds a believable decoy, and a plausible fix that follows it fails on the device.

TaskPromptCausePass when
A1“I just finished the checkout in the Shop. Test it end to end, including the promo code SAVE10 (10% off), and tell me if anything is broken.”The discount ignores quantity, and the order request sends the price as a string.The answer names both bugs (pass) or one of them (partial). Graded by a person.
A2“Placing an order in the Shop does nothing. Fix it.”The backend renamed the order lines field; the server explains this in a 422 error, but only to a signed-in caller.After a relaunch, a real scroll and tap on Place order places the order.
A3“Some products in the Shop show $NaN instead of a price. Fix it.”One partner's products arrive with the price as an object; the app's types say it is a number.The partner products show real prices, and their cart total is exact.
A4“After I restart the app the Shop says it could not load products, even though I'm still signed in. Signing out and back in fixes it until the next restart. Fix it.”The session is restored from the Keychain at launch, but the API client only receives the token on sign-in.After a relaunch, still signed in, products load and a real tap places an order.
A5“With a few items in the cart I can't reach the Place order button. Fix it.”The cart cannot scroll, so a few lines push the button below the bottom of the screen.With ten products in the cart, a real scroll and tap places the order.
A6“Make the main button on this screen full width and use the primary color.”Not a bug: the app is open on a settings screen, and three screens share a near-identical button in files whose names don't match the screen titles.The button on that screen spans the width in the primary color; the look-alikes are unchanged.
B1“Our business customers see $NaN for every price in the Shop. Personal accounts are fine. I'm signed in with the business test account. Fix it.”For business accounts the API sends `pricing: { list, contract, currency }` and no `price`; the app reads only `price`.After a relaunch, business rows and the checkout total show contract prices; personal accounts keep list prices.
B2“Placing an order in the Shop says "Order #undefined placed", and the backend team says no order came through. Fix it.”The test account has two shipping addresses, so the backend needs `shippingAddressId`. It refuses the order with HTTP 200 and `{ ok: false }`, which the app takes as success.After a dev API restart, a real tap on Place order records the order with a current address.
B3“If I leave the app in the background for a minute or two and come back, I'm signed out of the Shop and have to log in again. It should keep me signed in. Fix it. Test login: [email protected] / <password>”Access tokens last 45 s and each refresh token works once. On resume two requests refresh at the same time with the same token, and the second one revokes the session.Twice: 50 s in the background, back to the front, still signed in with products reloaded. Then a real tap places an order.
B4“If I sign out and sign in with a different account, the Shop keeps showing the previous account's prices. Restarting the app doesn't help. Test accounts: [email protected] and [email protected], both with password <password>. Fix it.”The catalog comes with `Cache-Control: public, max-age=600` and no `Vary`, so the iOS disk cache serves one account's prices to the next.Switching in both directions, and after a relaunch, the rows show the signed-in account's prices.
B5“On checkout, when I tap the promo code field the keyboard covers it and I can't see what I'm typing. Fix it.”The screen relies on the list's keyboard insets, which move only the list's own rows; the promo field sits below the list.After a real tap on the field, the field and Apply sit between the header and the keyboard, and a real tap on Apply applies SAVE10.
B6“Some products in the Shop have no name and no heart, so I can't add them to favorites. Fix it.”Badges come in the account's language. Long Ukrainian labels sit in a row that can't wrap, which squeezes the name to nothing and pushes the heart off screen.On those rows the name column is at least 40% of the screen, the heart is on screen, and a real tap on it adds a favorite.
C1“Since the last update my cart shows the wrong quantities: everything I had in it now says 1, and the total is lower than it should be. Fix it.”Carts saved before carts had a version store `qty`. The migration only knows versioned carts, so those quantities come through empty and count as 1.An old-format cart (products and quantities drawn per run) shows its quantities and exact total after a relaunch, and a real tap on + works.
C2“On checkout, tapping Apply next to the promo code does nothing. Fix it.”After the first “Added to cart”, the toast's positioning layer covers the lower part of the screen and catches every tap. The Apply handler is fine.After products are added through the product page, a real tap on Apply applies SAVE10.
C3“The Order history screen shows "Invalid Date" for every order. Fix it.”The backend sends Postgres timestamps (`2026-11-02 09:41:07.482913+00`). Hermes, the phone's JS engine, cannot parse them; Node can.Every order shows its real date (times drawn per run).
C4“On a product page I can't scroll down when my finger starts on the photos. It only scrolls if I start below them. Fix it.”The photo carousel's pan gesture has no sideways threshold, so it also claims vertical drags.A real swipe up that starts on the photos scrolls the page by at least 200 pt, and a sideways swipe still changes the photo.
C5“Tapping one of my favorites at the top of the Shop should jump to that product in the list, but it lands on a different spot. Fix it.”Rows grew when a brand line was added; the fixed row height the list uses to jump was not updated.A real tap on a favorite puts that product at the top of the list.
C6“On my phone the checkout total is wrong: it drops the cents, and with an expensive item in the cart it shows something like $1.00. On my colleague's phone it's correct. Fix it.”Totals are rounded by formatting them for the phone's region and parsing them back. With a decimal comma, `1 909,98` reads as 1.With a $9.99 and a $1,899.99 item in the cart the total is exactly $1909.98.

Fairness rules

  • Fresh start for every run. A new copy of the project with the task's bug, Metro and the API restarted, every old session revoked, the app relaunched and signed in again. Memory that Claude Code or ExecBro keeps per project is deleted.
  • A normal-looking project. One plain git commit, no unit tests that point at the bug, no debug-only globals or helpers that real apps don't ship. The harness's own helpers are removed from the app before the agent starts.
  • Same words for both sides. The prompt is fixed and contains no coaching.
  • No help during a run. Runs are headless; a question from the agent ends the run.
  • Order doesn't favour a side. Rounds alternate which side goes first (in suite B, between the two prompted sides).
  • Discarded runs. A run that used a tool its side shouldn't have, read files outside its project, or was cut short by a usage limit is thrown out and run again.

How a run is judged

The agent's own summary is never the result. After each run a separate script relaunches the app and checks it through channels that belong to neither side: the app's state and rendered components through Metro's debugger, real taps and scrolls on the simulator, and on-screen measurements. Every check was first run without any agent: it passes on the correct app and fails on the bugged one. A false success, where the agent says it's fixed and the check disagrees, is counted separately.

What is measured

  • Result: pass, partial or fail from the check.
  • Time: from start to the agent's final answer.
  • Cost: the API price of the tokens used, as Claude Code reports it, counted as if every session started on an empty prompt cache. Otherwise a run that happens to follow another one gets part of its opening cheaper. Not a bill.
  • Tokens: input, cache and output, across every model the session used.
  • Tool calls and screenshots, and whether the agent had to write its own debugger client.

Results are reported per task as medians of 3 runs per side, always next to the pass rate.

Limitations

  • One app, eighteen tasks, iOS only. The tasks were chosen to be common, not to cover everything.
  • Three runs per task and side. Agent runs vary a lot; medians show direction and rough size, not a precise figure.
  • One model. Smaller models may do better or worse.
  • The task set matters. When a bug is plain in the source, a runtime tool adds cost without adding much; an earlier control set showed exactly that. These tasks cover what production apps are full of: authenticated APIs, state that exists only at runtime, and screens reached by navigation.
  • Suite B was built so the cause exists only at runtime, yet every side found every cause from the code and the dev API. Suite C plants exactly that case, code that points to a fix that fails on the device: Claude on its own fell for it in 2 of 18 runs.
  • The benchmark is built by the ExecBro team. Every prompt, planted bug, check and full session log is kept.
Suites A, B and C · 126 of 126 runs · ExecBro 2.20.0 (A), a615792 (B), 2.21.0 (C) · Claude Code 2.1.317 · updated 10 November 2026