But they have "learned to implement any of the algorithms we know that we have used to make strong chess engines". Ask Claude Code to write you a chess engine. Your objection is that they don't implement MCTS in the neurons themselves? Neither does a human, we use a C compiler when we want to play chess using MCTS.
That's a separate question from whether an LLM (unaided by a C complier) can learn to play chess as well as human (also unaided by a C compiler). Certainly humans can't become grandmasters only by reading chess transcripts on the web, and certainly humans require many "thinking tokens" during a game to play effectively. Do you know for sure that a transformer can't reach grandmaster level if it is allowed to learn by playing games (as humans do) and is given a sufficient number of thinking tokens during the game? It seems near certain that they could, if someone wanted to spend the money (and I don't see why anyone would.)
Sutskever's claim is that in order to predict the next token a system must learn something about the "underying reality" that produced the token. In the context of chess that means that the LLM must learn something about playing chess (since tokens are the moves in a game of chess). My argument is that contrary to what should be expected if we take what Sutskever says to be true, they don't seem to have.
Yes, I do mean that the LLM's weights are set so that it will execute minimax or MCTS when it needs to. That has nothing to do with whether humans can do the same or not.
I don't disagree that a Transformer could learn to play chess if it was explicitly trained to do that. My argument is that LLMs, trained to predict the next token, have not learned to play chess. That's LLMs, not Transformers.
Just to make sure this is not taken as splitting hairs, the point is that there's all sorts of claims made about what LLMs learn when they train on text. For example, there was a claim by Sundar Pichai that one of their models had learned to translate Bengali without explicitly being trained to do so. It later emerged that Bengali was indeed included in the model's training set [1]. It's not clear whether that included parallel texts, e.g. between Begnali and English or another intermediary language, in any case Sundar Pichai's claim was that the ability to translate Bengali was "emergent".
So I'm interested in understanding the extent to which these "emergent" abilities are real or not. With chess, given the amount of textual data tracing games that floats about on the open internet, I would totally except some ability to play chess to "emerge". Maybe the reported 700-800 ELO level is even that sort of ability. Maybe we should only expect LLMs to learn to play at the level of an untrained, casual player. Maybe not. I have no idea.
On the other hand, the fact they keep making elementary mistakes like illegal moves must be taken to mean that, so far, LLMs haven't learned to play chess.
That wasn’t my takeaway from it at all. I think he’s quite precise in his definitions and the point he’s making about Rust, even if I found the prose a bit fatiguing due to the LLMisms.
Personally I think that the world needs more of your optimism. But what does your crystal ball say about energy costs and political stability?
It’s one thing to say that AI will help everyone create immersive games, but skyscrapers won’t be free unless energy is free. Do you also assume that AI will solve fusion?
What happens to the individuals who are not “strong” and how do they hold onto their remote work jobs? Why will the biomedical research technology you’re positing not be used to create biological weapons, or do you assume that AI also creates universal peace and harmony? If so, how does it do that while also preserving our ability to have our own ideas and disagree with each other?
If we want the great future you’re imagining, I think history teaches that we need to give at least as much attention to these questions as we do to making the technology work.
I think their work earns "theory" because it makes specific predictions both about how to make more effective prompt injection attacks and what activations you'd observe in the LLM during those attacks, and can also be plausibly extrapolated to suggest useful future research directions.
They were adopted by professionals long ago, and those highly tuned and validated proprietary models are going to kick the butt of the models that you have access to every day of the week.
I'm unpersuaded by the assertion that closing the source is an effective security bulwark.
From that page:
> Today, AI can be pointed at an open source codebase and systematically scan it for vulnerabilities.
Yeah, and AI can also be pointed at closed source as soon as that source leaks. The threat has increased for both open and closed source in roughly the same amount.
In fact, open source benefits from white hat scanning for vulnerabilities, while closed source does not. So when there's a vuln in open source, there will likely be a shorter window between when it is known by attackers and when authors are alerted.
The HN discussion on the announcement is just 90% posts of the theme "if a student can brute force your FOSS for $100, they can do you proprietary code for $200" and "if it's that cheap to find exploits, why don't you just do it yourself before pushing the code to prod?"
I believe that the reason the chose to close the source is just security theater to demonstrate to investors and clients. "Look at all these FOSS projects getting pwned, that's why you can trust us, because we're not FOSS". There is, of course, probably a negative correlation between closing source and security. I'd argue that the most secure operating systems, used in fintech, health, government, etc, got to be so secure specifically by allowing tens or hundreds of thousands of people to poke at their code and then allowing thousands or tens of thousands of people to fix said vulns pro bono.
I'd be interested to see an estimation of the financial value of the volunteer work on say the linux or various bsd kernels. Imagine the cost of PAYING to produce the modern linux kernel. Millions and possibly billions of dollars just assuming average SWE compensation rates, I'd wager.
Too bad cal.com is too short sighted to appreciate volunteers.
I think it's more prosaic, OSS is great for building a userbase but not great at generating revenue. So just wave the OSS flag while you build a userbase, then pull out whichever flimsy excuse seems workable at the time when you want to start step two of your enshittification plan.
Not only are they good at reading and writing machine code now, they are actively being used to turn video game cartridge dumps back into open source code the community can then compile for modern platforms.
If you believe they really did it for security, I have a very nice bridge to sell you for an extremely low price ...
Look, tech companies lie all the time to make their bad decisions sound less bad. Simple example: almost every "AI made us more efficient" announcement is really just a company making (unpopular) layoffs, but trying to brand them as being part of an "efficiency effort".
I'd bet $100 this company just wants to go closed source for business reasons, and (just like with the layoffs masquerading as "AI efficiency") AI is being used as the scapegoat.
Claude is already shockingly good at reverse engineering. Try it – it's really a step change. It has infinite patience which was always the limited resource in decompiling/deobfuscating most software.
It's SaaS though. You don't have access to the binary to decompile. There's only so much you can reverse-engineer through public URLs and APIs, especially if the SaaS uses any form of automatic detection of bot traffic.
Thanks you. This is what the parent post was trying to say. Don't know why it is down-voted. AI or not, if the API end points are well secured, for example use uuid-v7, then their is little that the ai can gain from just these points.
According to that site, Robert Kennedy's speech on the night Martin Luther King was killed[1] was almost entirely the product of GenAI, as were both of Obama's inaugural addresses[1][2].
By this logic, I'd venture a guess that "AI" was also responsible for some of Shakespeare's most famous lines.
That's a separate question from whether an LLM (unaided by a C complier) can learn to play chess as well as human (also unaided by a C compiler). Certainly humans can't become grandmasters only by reading chess transcripts on the web, and certainly humans require many "thinking tokens" during a game to play effectively. Do you know for sure that a transformer can't reach grandmaster level if it is allowed to learn by playing games (as humans do) and is given a sufficient number of thinking tokens during the game? It seems near certain that they could, if someone wanted to spend the money (and I don't see why anyone would.)
reply