> A.I. doesn't steal code, and just reproduces the ideas in the code
Then they don't need to train on github, no? Why not release a new model trained from Knuth's Art of Programming, Cormen's Introduction to Algorithms and the C specification.
Feel free to throw in any other published literature related to STEM, but stick to the code samples from the books.
I'm certain it'll be able to change the color of a CSS button, right?
What you said doesn't disagree with what the parent said.
LLM can be trained on a code and at the same time reproduce the core ideas. That's what LLMs do after all - they convert the training data into their own internal models and representations, and then reproduce the ideas.
Sure, some things/patterns, that were repeated multiple times, LLMs will tend to repeat verbatim as well, but that's not that big of a problem.
As a person who invented a few algorithms on my own I absolutely love LLMs and I don't mind them being trained on my work, but yeah - I've been way less likely to publish open source over the last year. In the past, if some of my stuff got traction, the credit was close to automatic (early adopters credited or at least knew where they got it from). Nowadays, LLMs will train on these ideas, rewrite them, and give no credit.
Still, I prefer this to having no LLMs at all.
> but stick to the code samples from the books.
> I'm certain it'll be able to change the color of a CSS button, right?
A good enough LLM will just decompile a browser, figure out CSS spec from it, and yes - figure out how to change the color of a CSS button from first principles. There is no point to do this with CSS, but with other things it's now easier to just dig through sorces or direct bytecode than to bother checking docs.
> Then they don't need to train on github, no? Why not release a new model trained from Knuth's Art of Programming, Cormen's Introduction to Algorithms and the C specification.
Because they're really really stupid and only make up for this by being really really stupid really really fast.
This has been ruled, by actual courts, to not be "stealing" (not even in the "you wouldn't steal a car, piracy is theft" sense that film and music studios campaigned on).
The last I heard was the "Chinchilla" scaling law was ~20 training tokens per parameter. Humans are, if you'll excuse a very hand-waving Fermi estimate, 100,000 times more data-efficient at learning stuff (it's really hard to tell given we're visual creatures that happen to speak, while LLMs are text-based things that happen to see).
To be fair, you also don't restrict yourself to those texts either. You read news, you use other programs, you look at websites and so on. And while the norms vary per field, things aren't really reinvented from scratch. The standard FPS controls aren't reinvented for every shooter game. The standard website layouts aren't reinvented for every website. The standard command line behaviour isn't reinvented for every CLI program and so on.
But software isn't like traditional academia. It might have grown from it but most advances aren't really published in the traditional sense, you've got blogposts, presentations and source code instead.
This would be like teaching cooking without looking at any recipes, just from physics and first principles. Or learning music without looking at the sheet music / listening to any existing songs, just generic musical theory and chords. I don't think humans can do it "zero-shot" either...
I don't see how this disproves my point. There's a vast divide between "theoretical" and "applied" software engineering. You don't usually see Google engineers publish their search optimisations in journals. And similarly, you don't see university professors making commercial libraries from their articles' ideas. This isn't the case in every field but in software, it very much is.
And this implies that training on the "theoretical" side of things doesn't give you much insight on the "practical" side. Stuff like cyclomatic complexity, UML diagrams and all that stuff might be well-represented in literature but way less so in real software, so training on the literature will produce completely different software than training on production software code.
> "Why not release a new model trained from Knuth's Art of Programming, Cormen's Introduction to Algorithms and the C specification."
What makes you think that wouldn't work? I think a lot of the hype around AI is vastly overblown but that seems to be well within the scope of what they can be expanded to do in the not too distant future. AlphaGo was trained through self-play reinforcement learning IIRC and I don't really see a reason that some sort of equivalent couldn't be done for generating code starting with textbooks and access to a Linux CLI as a reference. It would be an interesting experiment at least.
While Stallman may have been originally upset about his printer, the primary motivation behind open source software is:
Nothing up my sleeve.
This of course requires audit and that requires that the code is written for human consumption, otherwise nobody will bother. Sure you can vibe a printer driver but how sure are you that your LLM didn't include a backdoor in the millions line of slop?
How sure are you that any printer driver didn’t include a backdoor? What’s the difference between the LLM code with the backdoor and the human code with the backdoor?
If not that, the ability to trust the reputation of the author which creates an incentive not to willingly insert a backdoor.
Yes, `npm install` was always bullshit because most people didn't bother to check, which is exactly why it was exploited multiple times, which created a conversation about "supply chain security".
If you want to argue that "npm changed the world" then you are correct. It did not change the world for the better though.
Trusting software produced by others is not a new problem; I consider AI to be a tool, and so my techniques for establishing trust are the same as they have always been. I look at the person wielding the tool.
I look at the whole chain of influence behind them, and it's quite a lot more upsetting than the smiling face I interact with.
The whole "AI is a Lovecraftian tentacle monster wearing a smiley face" thing applies to simple bureaucracies (replace Lovecraft with Kafka), to corporations (replace Lovecraft with IDK, most anarchists?), and governments (Orwell?)
Tools made by tools made by tools, along more steps than most people know even when their job is one of them. Somewhere there's a kid working a dangerous mine without the right safety equipment, elsewhere there's a sweatshop, another place a "reeducation camp".
But who do I see? A cashier, mostly. Someone whose job involves smiling to customers even when we're idiots.
I'm not sure what point you're trying to make. I put trust in individual people, not organizations. You seem to be making a point about supply chains, but I'm not sure how it relates to the point about trust. You seem to be making a different point about some kind of exploitation.
Almost everything I interact with was made by an organisation (or a disorganisation), not by an individual.
Say I download an app. Who made it? The programmer? Their PM? Apple's store requirements? The US government, for whom there was a special tickbox I had to agree to last time I uploaded an app?
Trust is for the mechanic, driver, builder; but for 90% of my interactions I have to trust my government set good rules and other people followed them.
I can't do this with AI, neither good rules nor them being followed, but I also can't do it when the OS company and app devs are foreign, as they generally are to me now.
That's a motivation for source available. You can have commercial copyrighted software that's NUMS. You can deliver the customer source code (slightly customized for traitor-telling of course) and tell them to do what they want but never share it. grsecurity even managed this with GPL software!
While I agree with you regarding shady business practices, you're very conveniently skipping over the fact that open source licenses _REQUIRE_ attribution.
> While I agree with you regarding shady business practices, you're very conveniently skipping over the fact that open source licenses _REQUIRE_ attribution.
When you use the code as is or create a derivative work. The knowledge embodied by the code and encapsulated in an LLM doesn't strike me as needing to give attribution because the code the LLM would product doesn't match any particular open source code base.
At least that's my thinking. I'd be curious to see an example where you think attribution is necessary and how you would actually do it given an output from an LLM.
I hear you, but I think you might miss my point, which is while LLMs are clearly trained on copyrighted material, what they produce (their output) is NOT a copy of a specific code snippet they were trained on in a way that you would say "that's a copy from this code base".
Thanks for that link to the definition and requirements for something to be considered a derivative work.
I think my interpretation, based on your link, holds: unless the LLM output (transformation) substantially bears the original source code author's creation and personality, there is nothing to give attribution to.
Treating code as a low-entropy amorphous blob is a quick way to see bridges collapse.
It works for concrete because its tensile and compressive strengths are well known for a given composition, but for high-entropy products like software you pretty much need to also generate a formal proof to have any assurances.
Which, ignoring the production costs and assuming that the proof is in fact correct, is only ever as good as the specification.
If you were looking for a physical analogy for software development, it's closer to growing crystals than pouring concrete.
So do hackers. A big chunk of the open standards revolve around sandboxing malicious code, and given the amount of tracking happening anyway, fail at it.
Many do indeed hold the position that all LLM output is uncopyrightable plagiarism. They're probably right, but there's an even stronger argument here:
Science papers of a phd level must contain:
1. one or more novel insights
2. a long list of citations to contextualize them and
3. some work to prove that the insights are in fact meaningful
---
In this context, consider a prompt based diffusion model which, when asked, will happily produce a few pictures of a horse in orbit. You then tell it "silly robot, horses can't breathe in space" to which it adds the necessary space suit in a follow up image.
That image is twice plagiarized:
1. the model did not come up with the original idea of putting a horse in space, nor with insight that horses need a space suit
2. the model failed to cite where it pulled the "horse" and "space" concepts from.
It merely did the work (3) to combine the concepts using the user provided insight.
---
The implied accusation here is that OpenAI used the insights from an existing prompt to train a new model that was able to one shot "a horse race in space" picture, and they were all wearing space suits.
This is still academic plagiarism, even if you disagree that all LLM outputs are.
I neither agree nor disagree that all LLM outputs are plagiarism. I merely objected that the line of argument engaged in was specious given the context.
As to your stronger argument. You only cite prior novel insights that you're actively building off of and that (approximately speaking) fall outside of the status quo. You don't for example cite leibniz or newton despite your paper making heavy use of calculus.
So is there any actual evidence that openai trained on the data in question? And further, did the openai proof directly build on someone else's novel insights as opposed to deriving everything from scratch? (I don't pretend to know but the vast majority of what I've seen so far in the comments here is what I'd characterize as brain-dead screeching. Certainly not the level of discussion I come to HN for.)
Separately, consider the implications of what you're arguing for there. Suppose your horse in a space suit picture were somehow valuable to society. Suppose that due to shortcomings of your tool you lacked the ability to readily and accurately identify the originators of the relevant concepts. Should you refrain from publishing this useful work due to the lack of citations? How are you supposed to handle this situation?
Remember that in this analogy everyone throughout society is on the same page that your tool consistently recycles other people's ideas while being technically incapable of producing reliable citations. The question is a simple trolley-esque problem - do you publish without proper citations for everyone's benefit and if so what are you supposed to say?
evidence that openai trained on the data: they would have denied it if they didn't train on it.
did the proof build on the insights:
the influence of an individual text in the training data is deeply weighted by quality, relevance, etc. a high quality proof in advanced mathematics written by a codex user is going to get boosted to the max.
the model is post-trained on prompt material. that is again going to boost it.
the prompt will boost this material specifically. perhaps they even rammed dense maths in particular into the model in post training.
anecdotally i have been able to get near-verbatim copies of original material out of models at inference. the type of work that buckmaster and alpoge fed into openai feels like the exact type of concept that would cause an "aha!" or "but what if?" in chain of thought. in fact i would bet that their work is in the logs.
the likes of astra and fable are thought to be up to 10T parameters in size. i consider it highly plausible that a semantic representation of the euler proof could be pulled out of the model weights in good shape.
The chats the professor had are not generic knowledge. And yes of course you still need to cite Newton and Leibniz depending on what result you want to mention. What’s allowed to be not cited are not status quo, the term you’re looking for is “folklore” results aka results that have been around so long that 1) nobody knows who came up with them or 2) everyone knows who came up with them.
The second point: if you say you can’t prove that OpenAI actually used it, it doesn’t mean that OpenAI did not use it. It’s hacker news not lawyers news here lol. And OpenAI can’t prove that they didn’t use it either. The whole point is that Levent felt he had reasonable suspicion to believe the AI did use the result, because he felt like without his input on an unpublished paper it was unlikely for AI to reach the same result. I haven’t read the paper so I don’t know where I stand on that.
On the last point, about your “for the greater good” argument. It’s higher maths lol. I don’t know about this field but I doubt it’ll be very useful for society. Maybe it’ll make one part 2x faster which makes some rocket cheaper to launch. Does the average person care? Debatable. I think it’s reasonable to hold published papers in proof based fields to a higher standard. Otherwise the current & future problems of ML engineer fields just expand to other fields. No thanks.
Finally, if you anonpost to the autistic Internet forum that everyone else is “brain dead screeching”, it really just says something about yourself lol.
Even if AI used the result, AI pushed it to the finish line while Levent and Tristan did not. But I understand the approach was different, the information leak was only that it was "doable".
https://www.youtube.com/watch?v=nFZP8zQ5kzk
reply