When better models get worse at your tools

Simon Willison flags a finding from Armin Ronacher that’s worth sitting with if you build software with AI in the loop: the newest, most capable Claude models are getting worse at one specific thing — calling a custom tool whose schema they weren’t trained on. Hacking on his own coding harness, Ronacher watched Opus 4.8 and Sonnet 5 pad his edit-tool calls with invented fields that broke the schema, while older models in the same family handled it cleanly.

The likely cause is the interesting part. These models have been reinforcement-trained to be superb at the edit tools baked into Claude Code, and that specialization seems to leak into how they treat everyone else’s tools. Sharper along the trained-for path; shakier in the general case.

We keep coming back to this, because it cuts against an assumption baked into how most people adopt AI tooling: newer equals better, full stop. It doesn’t, not uniformly. A capability that looks general can turn out to be a trained behavior that doesn’t transfer to your harness, your schema, your app. For a small shop wiring AI into real tools, the lesson is an old and reassuring one: test against your own use, not the leaderboard. The model at the top of the benchmark isn’t automatically the one that behaves for you.