I don't see context size improving.

It was mostly 256k, then it went to 1M and now it has stalled there.

> You should absolutely be setting criteria that can be objectively measured and rejecting code that doesn’t meet those criteria or perform as specified

This is the classic "make no mistakes".

On a serious note, I might set as criteria "avoid code duplication". Does that mean that the model/agent will actually follow it?

> What does that accomplish, other than to slow your dev process down enormously?

I am an OSS developer and I often see PRs (i.e. from the general public) that look correct, pass all CI checks, are heavily documented and they are still wrong.

Most of the times either they duplicate code that already exists somewhere else, or they implement a "feature" by opening a can of worms for subsequent "features" in the same area.

> You can still read the code, ask the AI questions about it, and ask it to fix things

This only works in small projects. For large projects, it is close to impossible. Everybody talks about how new models appear all the time and nobody comments on the fact that context size has almost stalled.

Yes, it's certainly possible to generate code faster than you can clean it up. That's a choice. (Often called technical debt.)

Can't everything be broken down into smaller pieces? And isn't that actually a good principle in general?

Sure. But in order to do central refactoring and actually improve stuff you need to keep in context all the small pieces.

You can split a small program into piece A, B, C All of them look correct on their own. But they duplicate something in 3 different ways and person/agent who can "see" all of them can see the duplication and refactor.

Current model context is simply not enough for large projects.

Same problem for letting AI review code. A PR might look correct on its own and be small enough to fit into context. But somebody who has access to the whole code of the project again sees the duplication.

I am an OSS developer and when reviewing PRs I actually look at how the same problem was solved in other popular OSS projects. No AI can check this today because there is simply not enough context.

Basically if we had unlimited context what you said might be true. But context size is limited today.

I suppose it's not enough to break everything down into smaller pieces, your architecture also needs to be composable. That would address the issue of duplication. Think a plugin system.

Duplication was just an example. Code architecture is the general topic.

Yes a plugin system is great, but it only works if that plugin API/interface it designed correctly and gives plugins what they need while still enforcing good practices.

But somebody needs to design a plugin system that does this first. And designing a plugin system (for large projects) brings us back to square 1 :-) (that you need a large enough context to see what the code does in order to anticipate plugin needs).

12 factor is good but it is a very low bar.

It is perfectly possible to vibe-code a badly designed app that still passes those 12, 15 or whatever points you define.

Could you share some higher bars that may make up a better "rubric" for this issue? I am actively trying to do so. Shy of just condensing core Manning publications that cover domains of interest, I am struggling to find a good bar to have my clankers validate against outside of minimizing cyclomatic complexity.

Cyclomatic (and cognitive complexity) are a good start.

Some other ideas

1) Enforce architecture decisions (see archunit). But somebody needs to write them down first.

2) Check that tests actually break if the code that accompanies them is removed (several LLMs/agents today create tests that don't actually test the code they "guard against)

3) Automated performance testing. An LLM/agent might create a change that is "correct" but increases latency for 3x (best case) and 20x (worst case)

The hardest part that I see no solution for today is to understand when a change breaks backwards compatibility. LLMs/agents are trigger-happy and will happily refactor/remove stuff without any care about who is using that.I don't have a proposal for that, but the problem is there and is not covered by 12-factor config.

Excellent set of criteria, thank you. Something like am archlint, or performance lint, and sanity check on tests "Actually testing a seam or function" of the actual code base sound like good research avenues.

It is a warning to developers who don't understand the trade-offs involved.

Also a confirmation to people who have the same inner thoughts and are ashamed to admit in public that they think the exact same thing.

I think we need such kind of posts to combat the influx of AI news.

People who want a VR system that runs Linux

You can buy a pocket sized steam deck and do it yourself https://armadaos.dev/

Or use gamenative on your phone

You could already do that with gamenative. There are many videos.

Also you can run steam os on arm portables. https://armadaos.dev/

So all this was possible before the Steam frame was launched

Sure, but I would like to point out that gamenative largely exists thanks to FEX-Emu, which was started at Valve's initiative, and which Valve has stealthily funded since its inception.

Xreal or Viture

Either testing for something bigger OR demonstrating their power to a 3rd party with minimal real disruption