On 5 November 2026 Visa published the first results of its Agentic Commerce Benchmark. The release puts one number at the top: "models failed 42.8% of test runs." Further down is the number for anyone whose agent holds a payment method: "About 1-in-2 (54%) of failed runs still ended in the AI making a purchase."

Multiply the two and about 23 of every 100 runs ended in a purchase the benchmark scored as a failure. That figure is our arithmetic. Visa's release doesn't print it.

What did Visa's Agentic Commerce Benchmark test?

Visa's Agentic Commerce Benchmark tested whether an AI model can work out what a shopper needs from a conversation and then make the right call to buy or to hold back. The first round put 22 models from 12 labs through 15 shopping challenges, like buying a TV or a laptop, and gave every model three attempts at each one.

A run passes only when the model gets both halves right. Visa describes the benchmark as the first "to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass." Declining is a valid answer: "If nothing fits the customer's requirements, the AI should opt not to buy."

In the release's example a customer wants a new TV, mentions a brand a relative recommended, and leaves the rest unsaid. The model has to ask its way to the budget and to requirements like gaming support before it picks anything.

What were the results?

Across the first round, the 22 models passed 57.2% of runs and failed 42.8%. The best models passed 82.2% of attempts and the worst passed 28.9%. All three attempts at a scenario passed 34.2% of the time. Of the failed runs, 54% still ended with the AI making a purchase, according to Visa's release of 5 November 2026.

Two comparisons in the release say where the passes came from. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, against 44.4% for the five that uncovered the least. The five most expensive models per run passed 67.1% of runs, against 50.2% for the five cheapest. The gap between good and poor questioners is wider than the gap between pricey and cheap models.

Consistency is the weakest result. Visa's wording: "when made to repeat a scenario they had already passed, the same models often failed."

What do the numbers leave out?

Visa's first round stopped before the checkout. The release says Visa "will extend testing to checkout, payment, and order placement", so payment forms and declined cards sit outside the 42.8%. A purchase in this round is the model's decision to buy. The round also tested bare models, and your agent is a model plus your prompts and your tools.

Read the 54% with the same care. Visa gives two ways a failed run ends in a purchase: "Either the AI recommended a purchase that did not meet the customer's identified requirements, or it failed to uncover all the customer's requirements." The second kind includes purchases the shopper may have been happy with, made on less information than the test required. So 54% is a ceiling on wrong purchases among failed runs, and the release doesn't split the two kinds.

The tasks are consumer shopping. An agent that tops up API credits or renews a domain has no hidden preferences to uncover, and its failures look different: the wrong plan, or the same purchase twice.

Visa is careful about its own conclusion. Asked in the release's Q&A whether it's saying AI agents aren't ready for commerce, the answer is "No."

What should you test in your own agent?

Test what Visa's benchmark scores, on your own tasks: whether your agent asks before it assumes, whether it declines when nothing fits, whether it passes the same task on a repeat attempt, and what it spends when it gets the task wrong. Run every scenario at least three times, as Visa did, and count a scenario as passed only when all attempts pass.

  1. Write a task with a missing requirement. Leave out the budget and see whether the agent asks or guesses.
  2. Write a task with no valid answer. Ask for a 30-euro item where the cheapest option is 45. A pass is the agent coming back to you without buying.
  3. Repeat every task you've already seen pass.
  4. Count the money on failed runs. For each failure, record whether a purchase went through and for how much.

Visa's release counts how many failed runs ended in a purchase and gives no figure for the money spent on them. Run yours somewhere a wrong purchase costs nothing. Sandbox testing for agent payments covers the setup.

Where does a spending control fit?

A spending control fits after the model and before the money. Testing lowers how often your agent buys the wrong thing. Visa's best models passed 82.2% of attempts, which leaves failures to plan for, so you also need a bound on what one failure can cost. Limits set outside the model are that bound.

With Shatale, every purchase runs on a card assigned to that purchase alone and capped at its amount. Your customer sets a monthly budget and a cap per purchase in the Shatale app. Inside those, your agent pays on its own, unless the service asks for their approval or the price moved before the charge. Over them, the purchase is blocked or sent to your customer to approve or decline, with the merchant, the amount and the agent in front of them. Each decision lands in a tamper-evident audit log, per agent, which is where you count test four.

FAQ

What is the Visa Agentic Commerce Benchmark?

The Visa Agentic Commerce Benchmark is a test of how well AI models handle shopping tasks, launched by Visa on 5 November 2026. A model has to uncover a shopper's requirements through conversation and then make a correct decision to buy or to hold back. The first round covered 22 models from 12 labs across 15 shopping challenges.

How often did AI models fail Visa's shopping benchmark?

AI models failed 42.8% of test runs in the first round of Visa's Agentic Commerce Benchmark, published on 5 November 2026. The best models passed 82.2% of attempts and the worst passed 28.9%. About half of the failed runs, 54%, still ended with the AI making a purchase.

Does the Visa Agentic Commerce Benchmark test payments?

The first round of Visa's Agentic Commerce Benchmark tested the buying decision and stopped before the payment. Visa's release of 5 November 2026 says the company plans to add more shopping scenarios and "will extend testing to checkout, payment, and order placement." Until then, payment behaviour is something you test in your own agent.


Sandbox testing for agent payments is the place to run test four. Guest mode needs no key and makes no payment: npx shatale-mcp-server.

Shatale is the control layer for AI-agent payments. Its authorization architecture is the subject of European patent application EP26194994.5 (filed; priority 28 July 2026).