This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.
I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.
I’m not sure if the underlying data is counting subscription use for Fable, which is where a lot of people are using it because token pricing is very expensive. I wouldn’t be surprised if this was counting enterprise token usage only. As rich as enterprise customers are, they’re not exactly willing to double the cost of SWE salaries on tokens.
Either way, a model used to solve the top 10% of problems that people use AI to solve for, being used 10% of the time … seems like it’s in a decent place.
I find Fable indispensable, and measurably better than alternative models, for complex feature development in an existing codebase. It’s the closest thing I’ve seen to nearly one-shotting features. Even still, I only use it for the hardest features and Opus 5 does a good enough job on the rest.
Very much agree on Fable. Over the past month or so it has shown to be the only Anthropic model that can understand a largish dbt model codebase. Opus 5 gets almost everything wrong. (High reasoning on both)
More importantly other way cheaper models can do what Opus 5 does. So you can pay for Claude to use Fable 5 exclusively for harder stuff and planning, then get the same value you'd otherwise get from switching back to Opus by using other cheap LLMs for day-to-day coding tasks.
Yeah as soon as CFOs realized AI was racing to become one of the most expensive line items along with salaries and AWS bills, they started cracking down on the most expensive ones.
I’ve never had an agent fail to write the actual implementation. Has it done so badly, yes, but not nothing but tests. This sounds to me like a rare case that doesn’t generalize.
If the general idea is that these agents write too many tests, sure I guess? ‘Too many tests’ doesn’t sound like a failure case of engineering to me; typically software has had too few tests. Also, a lot of the power of these agents is their ability to self-verify and correct, which the test loop is a part of.
Nobody is making you pay this supposed tax. Just tell it not to write tests.
Y'all need to try PR review tools that split PRs into chapters. Stage was the best product in this space, but Linear Review has it, Graphite has it, and some other tools too. You get the full contexts while each piece is still reviewable individually.
For me absolutely not. Fable 5 has been a step function change in the ability to hand off stuff to Claude. Opus 4.5 was itself a step function but I was still steering that significantly. Fable is one-shotting stuff that took multiple redirections in 4.5.
From my view Fable has been pure marketing bullshit, my workflows peaked at 4.6, Fable is neither smarter, its language is more annoying and it breaks stuff more easily.
I'm with you 4.6 is still King for me although all models require careful attention to ensure they maintain taste. If you don't know enough about what you're doing to keep the code clean yourself they will all add complexity and drift with time until you get to a point that you must rely on the model to fix it because you no longer understand it. That's a situation I hope to never find myself in.
Exactly, the most important part is to still engineer the schematics 100%, diagram whats going on and steer it towards these set-in-stone standards and patterns, otherwise you'll have no idea whats going on fast.
I good exercise for me is to constantly look at the folder structure and skim the code, i don't have to approve every line, but the primitives, the datastructures and other skeleton should be human readable, hand writable in an easy maintainable way following existing standards / libs. etc - so you can continue if suddenly all AI disappeared.
This is phenomenal. I know everyone is asking for their pet feature in this thread, but I would really like to see this in split keyboard format. They are much more comfortable to use since they put your wrist in a more natural position. And there are hardly any good split mechanical wireless keyboards – and certainly none this thin.
Great work on this and congratulations on the launch!
And to be specific, the METR study was using the Cursor harness with Claude Sonnet 3.5/3.7, along with other models of that era of the participant’s choosing.
Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good.
The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.
And the comment you’re replying to, which is entirely a reasonable opinion, was flagged, proving their point. I’ve been here for 18 years and similarly alienated by the cynicism here.
I don’t think this is entirely wrong, in that there is a ‘class thing’ about riding the bus, but it’s more practicality than a class marker for a lot of people.
- In SF you can either walk 1-10 minutes to the bus, wait 0-15 minutes for the bus, tap on (while watching most other passengers evade the fare), get dropped off, and then walk 1-10 minutes to your destination… or spend an additional $5-10 to get Ubered door to door at a third of the time. First and last mile are real costs.
- In SF I Uber, unless Muni/BART is a straight shot. In NYC I take the subway. It’s not really a class thing. In NYC it takes longer to Uber much of the time and it costs several more times than the subway. You still have a 1-5 minute first and last mile problem, but headways on trains is decent and above ground taxis are incredibly inconsistent with traffic.
That about matches up with the experience with social groups in similar classes in these areas too. Most of my SF friends Uber. Most of my NYC friends take the subway.
This comment is legit! So many of these comments here are wildly biased to people's own personal experiences (usually the male/female Karens are the most noisy). You make many good points here.
My question(s): Why do you think Uber works so well in SF? Why don't they get trapped on Market Street with crawling speeds?
I lived in NYC (Manhattan) many years ago and I always felt that when I needed a taxi (cold/snow/rain), they were hard to get. As a result, I almost never took a classic yellow cab in NYC/Manhattan.
I’m all for standardization but you could just use this argument to keep any suboptimal status quo in place. XML is good enough and a standard. SOAP is good enough and a standard. etc.
The claim is that Conventional Commits are good enough and standardized enough that having another structure isn’t really worth it. But “worth it” is subjective. I’d say that if you are making commits and reading PRs every work day, and the conventional commits format causes a little bit of friction, that friction can add up. Having another option other than seeing conventional commits as a law of nature gives options for teams who prefer it. (Most teams aren’t generating changelogs anyway.)
The new structure needs to be "better enough" that it overcomes the built-in deficits of the older structure, and it can't introduce so many new problems that make it a net negative.
JSON was definitely a huge improvement in simplicity and readability compared to XML for many contexts. Similarly REST a much better option than SOAP (and all of these are examples of the general over-engineered, design-by-committee architectures that came out of the late 90s/early 00s - see also the original EJB spec - before a larger trend towards simplicity and ease of use won out).
But it this case, a lot of the differences just feel like potayto/potahto, i.e. minor stylistic preferences. And I have been in jobs where more than 50% of my time was doing code reviews, and while often there were e.g. some linter rules or whatever that I found suboptimal, it was a lot easier to just go with it than waste the energy to have the battle over why I think for loops are actually OK.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!