I think benchmarks like this are too subjective and narrow to be useful. For example, whether a patch "bloats" the codebase really depends on the situation: If it's building a feature that will grow in the future, or refactoring code that has a long history of bugs, then a larger patch might in fact be good. It's not clear from the blog just how much context the LLM judge receives about the long term project goals and history. Benchmarks should be focused on evaluating the final result only. Maybe ask the coder to build a full app, or implement many new large features for an existing app in sequence, with a larger set of requirements, or have another LLM roleplay as the human to make the instructions a little more underspecified. When done, ask a reviewer harness to test the product for 5 hours, not the code. Count the number of bugs and weigh them by severity. "Taste" would then become an automatic consequence of correctness.
Then maybe you should abstain, because your comment is a complete load of nonsense.
Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not.
Bloat is bad regardless, because it increases the overall complexity of the whole software development lifecycle, for the whole team, forever (or until refactored out): It's harder to keep track of the code and how it works to write new requirements, it's harder to write, it's harder to read and review, it's harder to debug, etc.
You can write extremely poor code that has no bugs, it doesn't make it tasteful. This is simply a ridiculous statement.
>Maintainability is important because you can never know if a feature will be built upon in the future or not.
Of course maintainability is important. It's almost like saying good code is important (duh). The issue is that what is or isn't maintainable depends on the problem at hand. Sometimes you need to build heavier abstractions or refactor existing code when implementing a feature because it will pay off later. Other times, that exact same approach is horrible over-engineering because a simple, direct fix was all that was needed, so in fact you introduced a maintenance burden. You cannot reliably decide whether a patch is "bloated" or "tasteful" when looking at a diff without knowing where the project is headed.
>You can write extremely poor code that has no bugs, it doesn't make it tasteful.
You can, but it becomes increasingly hard to do so as you try to add features and maintain it. Taste, whatever that is, should ultimately lead to a measurable increase in the quality of the final product; if it doesn't, then your definition of "taste" is irrelevant. What I'm proposing is to skip trying to measure this ill-defined concept and only assess the quality of the final product, after the agent spent a significant amount of time working on it, and a reviewer spent a significant amount of time testing it. Agents should be assessed on their ability to build entire projects (e.g., many large features or even an entire app), not just a single feature. If an agent has no taste, then its bad decisions will compound and result in it stalling, or its output having more bugs and performing worse, given a sufficiently large scope.
You buy a wooden dinner table, it is fully functional and looks perfect. It’s sturdy. You have dinner on it and it survives a few spills.
A few months later you find out it is made of PU foam and printed waxed paper. A misplaced knee could bring it down. It’s likely to completely fall apart in a year. Is that irrelevant?
Yes it is relevant and testable. It's exactly what I meant by "a measurable increase in quality of the final product". In fact a proper test harness would reveal that problem. You are forgetting that with LLMs, testing software does not have to end at the usual unit/integration/e2e level.
But how is that testable? If your test is validating the rigidity, water resistance, etc, they will all pass even if the underlying material is a bad choice. Or the glue will degrade in six months.
You can't test if a codebase will be extensible or maintainable as requirements change in the future, if the abstraction level or architecture is sound - that's down to code quality measures like the ones used here. LLMs are very good at slightly cheating to pass tests even when the implementation is wrong. Introducing subjectivity - the kind of input a human will provide - leads to improved output.
That's why we should simulate changing requirements, for example with an LLM roleplaying as a human who's co-developing with an agent. Simply asking the LLM to add one big feature is not enough. I don't see why we shouldn't be able to build a more advanced benchmark. Attempting to benchmark "taste" is not the way.
Yes, please do leave. The thing is that this isn't even necessarily about software engineering as much as it is about benchmarking/epistemology in general.
The fact that claude and gpt 5.5 have nearly the same scores tells me your benchmark is not capturing a significant gap in capability between these two. What the linked page says about Claude is true in my experience: It frequently forgets important instructions and likes to take lazy shortcuts. Gpt by contrast is much more attentive and takes its time when needed to deliver a complete and robust solution. I have tested both models on two private repos (c#, go) on two long-horizon tasks with well-defined stop conditions and observed the same pattern in both cases. Both models still require a large harness to reduce shortcuts and architecturally unclean code, but gpt performs much better, to the point where I find claude unusable for any significant work.
GPT 5.5 does significantly outperform Opus 4.7 in the coding parts of our evals.
We also incorporate live decision making on social games (where GPT 5.5 has actually regressed from earlier models, which shouldn't be a huge surprise if you ever tried talking it out of some of its nits).
We are still looking for a way to integrate "logical" intelligence with social intelligence in a less arbitrary way, so I'd take a look at the use case that applies to you (probably coding): https://gertlabs.com/rankings?mode=agentic_coding
Worth noting that LLMs are very bad at writing cetz code, even if you try to feed them all the docs. I had to use TiKZ and import the resulting PDFs for some of the more complex illustrations in my thesis.
But as design criteria go, that is certainly a sensible one to include.
Just a random first idea, the key effectively auto updates, I.e it’s a time varying key chain. I can think of several ways to do that, so the time varying nature can’t be replicated by someone else without the same originating account. But couldn’t say if any were good or not. It is something to design carefully, as all cryptographic systems need to be.
Other criteria would be easy revocation by the original key holder. Keys that are created from any multiple number of independent accounts, blind to each other, that the key recipient chooses.
I don't get his "modern" proof. Specifically the step where he says "it's easy to see geometrically that these matrices differ by a rotation" seems to be doing a lot of heavy lifting. The first matrix transforms e1 to (a,-b), the second scales e1 to (c,0). If you can see that you obtain one of these vectors by rotating the other, then you've shown that their lengths are equal (i.e. a²+b²=c²), which is what we want to show in the first place.
Set B as the origin, and let BC (the 'a' side), be on the the positive side of the x-axis. Let AC be on the positive side of the y-axis.
The left matrix is a clockwise rotation and scaling. This is clearly seen if you draw the transformation applied to the two axis basis vectors. (The scaling factor isn't obvious yet.)
Then the left matrix varies (1,0) to the side AB, which has magnitude c.
Z and carries (0,1) to an perpendicular line of the (importantly) same magnitude,
So it's a rotation and a scaling by c.
The right matrix obviously is a scaling by c.
> If you can see that you obtain one of these vectors by rotating the other, then you've shown that their lengths are equal (i.e. a²+b²=c²), which is what we want to show in the first place.
You're assuming that we know that the length of vector (a, -b) is a²+b². We don't know that.
We start by assuming that the position vector (a, -b) has length c. This implies that we can rotate that vector until it becomes the position vector (c, 0).
As you note, we can create the two vectors above from (1, 0) using linear transformation matrices [(a, b), (-b, a)] and [(c, 0), (0, c)]
So we could create the position vector (c, 0) by starting at (1, 0), applying the linear transformation [(a, b), (-b, a)], then applying a rotation to bring it back to the e1 axis.
Thus for some rotation matrix R,
R × [(a, b), (-b, a)] = [(c, 0), (0, c)]
The determinant of a rotation matrix is 1, so the determinant of the left side is 1×(a²+b²), while the determinant of the right side is c², which is how we end up with a²+b²=c².
Now the only thing which I'm not sure of is whether there's a way to show that the determinant of a rotation matrix is 1 without assuming the Pythagorean identity already.
> Now the only thing which I'm not sure of is whether there's a way to show that the determinant of a rotation matrix is 1 without assuming the Pythagorean identity already
You can define the determinant that way.
Now the question is why the cross multiplication formula for determinant accurately computes the area.
Yep - I'm just not sure if any of those proofs implicitly assume Pythagoras, and haven't thought through them properly.
I was initially going to say we know that det R = 1 by using the trigonometric identity cos²x+sin²x=1, but then found out that all the proofs of it seem to assume Pythagoras, and in fact, the identity is called the Pythagorean trigonometric identity.
I have laptop with a good-ish CPU that is only a few years old, and on page 3 tinymist is already starting to struggle. There is a noticeable input delay between me pressing a key on the keyboard, and the key getting typed & the preview updating. I think it's more of a tinymist issue though, as it has no debouncing and apparently also runs the preview updates on the same thread as vscode's input handling.
Interesting. I have not experienced that, except when trying out the pre-release version of tinymist, and did some messy multiple view+cropping into a big pdf (testing out the new pdf-image stuff.) I chalked it up to it being new and beta.
Admittedly, I have still not created large documents in Typst.
Yeah the only way I could really prove anything would be to fully own and operate the site, a task I would never take on. It might be interesting and educational for a day but without government immunity it would get risky fast.
>Causal models require machinery which is symbolic, which is able to generate hypotheses and test and prove statements about a world. LLMs are not yet capable of this and the fundamental architecture of the llm machine is not built for it.
Prove that the human brain does symbolic computation.
We dont know what the human brain does, but we know it can produce symbolic theories or models of abstract worlds (in the case of math) or real worlds (in the case of science). It can also produce the "symbolic" turing machine which serves as an abstraction for all computation we use (cpu/gpu/etc)
I'm actually confused about why banks are so aggressive in denying users the ability to use their apps while rooted. Unlike Google and Apple I can't think of any financial incentives for this, and the security argument is quite obviously nonsense, as I don't think there has been a single person in history who managed to fall for a scam that made them follow the complicated procedure of rooting a smartphone. Nevertheless there is a clear continuous effort in developing new root detection methods to keep me from using their apps.
I believe the root detection is a form of security-by-obscurity. Bank applications are required to be obfuscated, so you can't simply statically decompile them. The other way to do that is to run the app and set runtime breakpoints, which you can't do on production firmware.
Once the application is decompiled the attacker then can proceed to pentest the bank backend, or find any frontend-only security measures to bypass. One attack I heard in local news is not even a hack at all - they simply make script that use the mobile application API to automatically move money between sock puppet bank accounts. Once a victim get scammed, the money move around quickly. For privacy banks do not provide information about unrelated cross-bank transfers so even cops can't easily trace the multiple hops. That specific bank got in the news for that "weak security"
Security of banking shouldn't depend on the client software, it should be enforced at the interface the clients use to talk to the bank. It shouldn't matter whether the banking app can be disassembled or not. As much as I detest browser-based authentication in general online banking websites got it right: you just use a browser (and it's in your best interest to use a trusted browser -- one trusted by you) but all the bank cares about is that the user has the necessary pieces for authentication, be it numerical codes, passwords, and 2FA tokens. The browser doesn't have to be a bank-signed edition of MS Edge, it can be Firefox or even a browser you wrote yourself. But a banking app is basically a black box that you would have to allow to run in your system in order for the bank to talk with the software the bank itself trusts.
> to fall for a scam that made them follow the complicated procedure of rooting
If you are unable to imagine how a 3rd party might root a device without the principal being aware of it, then maybe it is a shortcoming of your risk survey, not theirs.
Rooting an Android device generally requires completely wiping it and reinstalling the OS. It's quite impractical to do secretly!
I think in any scenario where the principal can do that without you noticing (which means things like reinstalling & logging you back into all your apps, logging the device into your google account successfully, restoring all your device settings, re-adding your fingerprint or device pin to unlock the device, etc) then it's game over regardless. If they can do that, they could get into your bank app anyway, or they could easily just replace your phone with another one entirely, and now you're just logging into your bank on a stranger's phone.
Barring a _very_ major Android zero-day (which probably would evade attestation anyway) unexpected rooting of your device is really not a plausible attack scenario.
(Full disclosure, I'm not a software engineer.)