Yann Lecun is "famous" for (besides being one of the fathers of "modern" AI) his opinion that transformers are inadequate and other directions should be explored.
Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age.
It is viable as a toy project, but there are vanishingly few career opportunities.
They'll learn so much more that won't transfer to as many job opportunities. For ex, say more about C++ and less about cutting edge CSS (because modern browser tech is an ocean). I suppose they might luck into other adjacent or unrelated roles with the same skills.
I have a hard time imagining anything where writing a browser wouldn't be an excellent transferable foundation. Even your example: writing css will never teach you as much as writing a css engine.
Sorry if I was unclear. I mean they'll never have enough time to learn a significant amount of modern web tech because they'll be working up to it from lower layers and basics.
Browsers from scratch are multi-year projects for multiple people. Even just skinning and minor tweaks to modern browsers is a deep well for one person.
Building a browser in the early web was actually a very achievable goal for exactly the reasons it isn’t now. There was not JS. No CSS. No SVG. In fact very few widely supported image formats (and graphical browsers weren’t around in the earliest days of the web anyway). TLS didn’t exist. HTML only had a subset of methods. And even POST was usually just managed by CGI/BIN calling an external process, often written in C++ or Perl.
They will also know the fundamental underpinnings of the web, you know - the thing that those html devs are actually using.
That’s like saying someone who fabricates cars doesn’t have the skills to drive them. Perhaps not, but they’re very well placed to pick it up quickly. They’ve also shown they can do something far more challenging, which is actually better than hiring for narrow immediate skills.
In other words: I’d hire that candidate in a heartbeat.
You'd hire an auto designer to be a limo driver? I think it's just two different skill sets. The auto designer is clearly smart but a limo driver also needs to have customer service skills, etc.
I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists. I used to build display drivers for graphics cards. Wonderful learning opportunity but other employers not in the graphics card manufacturing business didn't give a shit--those three years were essentially treated as an employment gap. "Well that's nice, but we really wish you had general experience writing CRUD apps..."
I would argue that in a post-LLM world, becoming as specialized as possible is the only way to survive. If you have the serious systems programming skills required to write graphics drivers, your talent would be wasted writing CRUD apps anyway. I hope you eventually found/will find something more appropriate for your skillset!
I'd argue the exact opposite. If I want a graphics driver, I'll ask the clanker for one. But if I don't know enough to even know what I want, there's no path forward.
People keep talking about a browser in the modern context but the GP specifically said “early days of the internet” (which, in fairness, would mean pre-web. But I think it’s safe to assume they meant “web” not Internet).
In those days, it was actually a much simpler exercise to write a browser than it is today. I even wrote one! And writing a browser absolutely teaches you how HTTP and HTML worked. Plus a lot of backed development was forms data sent to CGI and thus written in languages we wouldn’t even dream of using for web development nowadays, including C++.
So in the early web, writing a browser absolutely was a transferable skill. It might not be now, but in the context defined by the GP, it was.
> I think he's saying that building a browser is not a transferrable skill, like making a generic web page is. Employers don't like specialists.
I recently switched roles, and among the seven places I interviewed, none of them seemed to see my then-current browser job as a problem, even though they were not related to browsers. (The closest one was a company implementing a HTTP reverse proxy, and I did not work on the browser's HTTP stack.)
This is like saying writing a compiler doesn't give you transferable skills. The surrounding competencies required to do this grant a pretty large amount of broad domain awareness
[Blake Ross] worked as an intern at Netscape at the age of 16 ... Ross became disenchanted with the browser he was working on and the direction given to it by America Online, which had recently purchased Netscape. Ross and Hyatt envisioned a smaller, easy-to-use browser that could have mass appeal, and Firefox was born from that idea ... in 2003 all of Mozilla's resources were devoted to the Firefox and Thunderbird projects. Released in November 2004, when Ross was 19, Firefox quickly grabbed market share ... with 100 million downloads in less than a year
In the meantime Mozilla was mocked on slashdot.org and elsewhere relentlessly after the first two years of the project when nobody believed there would be any value in the effort. Hats off to the team that took around five years to get to Firefox 1.0 (and released Mozilla browser in the interim). It took a lot of conviction to see it through.
For this example, if you built a browser in 2000 it would put you in a great position to launch android in 2008. I think this is pg’s point. You spend all this time learning the interesting bits and when an inflection comes along that makes a new product or service possible you could be the one to likely launch it
FWIW I built browsers from 1999 for a long time. (But I was never a wunderkind, just somewhat tenaciously curious).
And I guess I am doing fine, but not amazingly rich or so.
Browsers were always a project closer to research/charity. I think Marc Andreessen said something similar - that he would never do that again. B2B is where you can make money.
That’s like saying if you can dunk in 8th grade you could be the next LeBron James. A LOT more things have to fall perfectly in place at the perfect time for that end result to materialize.
I am talking from the perspective of the individual developer.
You don't need your impressive product to succeed to land a good career.
If Alice is doing LLM-from-scratch work today, and Bob is doing agent harness work today, Bob's project is far more likely than Alice's to become useful/popular/profitable.
But if neither project survives, in 5 years, Alice will be more employable/at a higher market rate than Bob.
Neither one has even gone to college yet. I’d expect Alice’s project to be more likely to interest professors, get some undergrad research experience, maybe that can turn into something.
There is a confounding variable, however -- the person is most likely a good dev. It stands to reason that they've had a decent career at least since then.
Yup like you go far enought you will pick lots of transferable skills like data cleaning in the case of LLM, DOM parsing case of web browser or welding case of rocket.
That may have worked at the time, but no companies are interested in learning projects today. If you didn’t do the reqs list for the last 5-10 years with the same title, forget about it. Because there are dozens of folks who have, lined up. No one is indulging career changers (and most fresh grads) for the time being.
Early internet let me create the best personal web page in my city that I knew of with 2 weeks experience as a 12 year old. I imagine that same 12 year old could be more knowledgeable about LLMs than 99% of people in the same period.
I doubt it. Creating a website can be done by copy pasting a few snippets together and checking if it visually looks like expected.
Good luck with that approach when trying to toy around with models and their training/inference.
There is also a lot of math basics missing that a 12 year old may be able to grasp, but I would bet they are at least 13 by the time the knowledge is deep enough to understand what operations are happening.
Really? The latter was immediately useful to lots of people which is motivating, and it had a nice smooth learning curve (html -> js -> php -> databases -> apps -> backend). Learning HTML is the first step to learning how to make full blown apps. Making a browser at 17 is like trying to climb Everest as your first hike. The expected outcome is burnout and demotivating failure. At best you'll learn some C++ or Rust.
17 is an interesting age. There are way too many comments here saying things like, 17 year olds should just do whatever seems interesting or bum around the world or focus on getting into university. But historically most kids were expected to be productive adults at 16 or 18. 17 is about the right time to be thinking seriously about what kind of work you'll do, how you'll make a living. University won't help and will just delay this decision.
I contributed to a browser engine around that age (KHTML, which later became WebKit and Blink), and while I don't work in browsers right now, much of that knowledge, mindset and of course the professional network have done much to shape my life. And a fairly successful career, for that matter.
Contributing to an open source project is fine, but the original analogy was "it's like telling teenagers to build browsers".
If teenagers could make small contributions to LLMs via open source then sure, go for it. Optimizing llama.cpp or similar would be a good learning project that might later get you good work via social networks. Contributing to open source is how I got started too.
Unfortunately, training LLMs isn't something that fits well to open source open collaboration. Inferencing codebases are better.
Excellent point, and I agree. Much of the benefit I saw was from working within a like-minded, smart team, not going it alone. And also specifically working on software with a real user audience to learn what providing value to them actually constitutes.
In that sense it's more a "seek out the open source community and real projects when young" rather than "do web browsers", with a bit of "look for ambitious types of projects few get to work on".
Early web didn’t have JS. Nor PHP. In fact a lot of early web pages were written using static HTML with C++ invoked via CGI/bin for processing form data. So writing a browser would teach you the HTML plus C++ too.
The early Internet (which the GP mentioned) didn’t even have the web. But that’s nitpicking.
17 is a weird age but ymmv. I left home at 16 alone to study abroad. I had tons of free time due to dorm curfews and such. Unfortunately, I was too poor to have a computer and the computers we had access to were completely locked up. (Naturally we waltzed past the locks to play some games but it was also under surveillance)
Paradoxically I coded way more between ages 12-14, I regret my wasted late teens.
I mean... that's actually amazing advice. Not because they would grow up to create browser startups. But because they would grow up to create web startups that succeed because of very fundamental of how the web is rendered.
This site exists because paulg created a webshop in a niche language (Lisp) and got a prototype bought out (and discarded) in the gold rush. No browser internals needed to make a fortune on the platform. Just rapid development of an application with natural monetization, and being in a place to do so way at the beginning.
Jeff Bezos did similar, but he did his own fulfillment and hired out the coding.
That seems like a perfectly good use of time for a kid in the 70’s! You’d learn a lot of engineering skills and demonstrate a tenaciousness that most people don’t have. I don’t think the goal is to predict what will be the important technology in 10 years. The goal is to challenge oneself with hard tasks and learn interesting stuff. A lot of the “AI” people today were compilers people yesterday
There are more career opportunities building rockets than designing rockets. Lots of welders, machinists, electrical engineers etc build rockets, and those skills transfer.
The same is not so different for AI. A few people design novel AI, but there are a lot of people training AI (especially if you include fine tuning) and implementing AI, even as a hobby.
But he’s not saying that it’s a career opportunity.
It seems to me like he’s saying that doing this thing would be 1) fun and 2) a great way to become employable in the future. I don’t believe he’s saying that this project would be some kind of job training exercise.
I "guess" you could put all your effort into moving up to the next level in computer science, or you could put the same progressive effort into reaching new levels in video games on the same hardware.
Alternatively you could even put all your effort into social activities and leave the technology to other people entirely.
It might even be possible to find a balance between things that are widely "understood" "socially", and those that are not ;)
When I was 17 and still witnessing Apollo Moon launches, I wanted to build an AI that would handily outperform LLMs as we know them today.
But that was way back in the early 1970's and all I had to work with was a mainframe.
Well the mainframe itself wasn't bad, the real show-stopper was that I didn't own the computer outright, no strings attached, no debt, etc.
>I'd probably try to make an LLM that I could use on some specific problem.
I thought so too back then, still do so I guess this is one of those things that could stand the test of time. I always wanted to start with something a lot simpler than a Moon mission myself. At 17 I already had a significant breakthrough in the chem labs and it was from alternatives to a single processing step plus everything that descended from that, rather than trying to tackle a much more complex detailed multi-step synthesis. I was only 17 but I was not trying to be a slouch, I don't think pg was either at that age but his advice is not for just anybody. I couldn't have done it if I hadn't made major progress since being 16, and it really emphasized at the time how much maturity can make a difference. My imagination ran wild as I extrapolated :)
In a reply from LeCun to pg:
>>I'll figure out a set of methods and architectures beyond LLMs that can quickly learn to perform physical tasks as efficiently as humans and animals.
That last item is also what I would if I were 30, 40, 50, or 66 years old
I see no reason to stop at 66 either ;)
But I figured that people owning more computer power than I could ever afford were going to be doing something like this as soon as they could, without having to wait for something like an LLM to arrive before getting peoples' attention.
It did seem like things were going to take longer than you expect, so it's pretty good to have a lifetime of concentrating on the specialized natural science domain expertise, focused now for 50 full years on how it would combine if AI ever got good enough.
Both the natural science and the AI need to be a major cut above, I still see dramatic room for improvement in my own work. If I'm going to have to rely on "other peoples' AI" then that natural science component is going to have to pull a lot of weight to keep up with the kind of computers that only rich-as-hell high-rollers have access to.
For games in general, one measure of complexity is branching factor. It means average number of possible actions or states at each turn. It is knowable.
"Solvability" would mean number of turns to solve the game. It is known for some puzzles and can be found by brute force, otherwise you need to figure out a proof.
I think there are two ways one could look at this. One is to make each move be a move of the player's location. If you do that, then the branching factor is obviously < 8. But there's a second way you could define a "move" for the purposes of a solver. And that would be to only consider pushes. In that case, the branching factor would be < 8*num_stones.
In either case, I think when trying to assess complexity it might also be useful to consider the "narrowness" of the winning move sequence. Positions where the number of moves that win/make progress towards the goal is a small fraction of the number of available moves would arguably be harder or more complex than positions where a larger percentage of the moves win/make progress. In other words, finding a smaller needle and/or in a larger haystack makes the problem harder / more complex.
Hmm. The push representation makes sense because solve progress is entirely dependent on it. And the movement state tree can be reduced to the push tree, which would only prune useless paths. The push tree can probably also be pruned for moves that leave to softlock, but I wonder whether it can be reduced to a different representation still. Push tree already requires us to maintain a mask of where we can move to, so it's not computationally free. I can imagine representing box pushes as every position we can push it to in the current setup, but that would also make it more computationally expensive.
I feel like there's an interesting tradeoff of storing/computing cheap representations vs exploring a smaller tree.
Stuff like this exists for bicycles. For mopeds and motorcycles there are electrification kits.
It should be possible to just replace the flywheel altogether with a generator/motor that both brakes and powers during each cycle, cancelling vibrations. Then more generation during deceleration, and more power during acceleration.
This (classic) argument is symmetric with respect to the value of money and quantity of goods. As in "if you know money will buy more in the future, it increases your incentive to sell now rather than wait for higher prices. And if you know prices will increase, you will hoard products." The argument doesn't favour either side.
One mechanism of inflation is that it effectively lowers wages (and other contracts) without negotiation. Asset prices are valued by markets and increase with inflation. It effectively transfers wealth from wage earners to capital owners.
Deflation would effectively increase wages instead, and require occasional renegotiations if productivity isn't keeping pace.
The problem is you can't really hoard products. Most products depreciate - it's a force of nature called entropy.
I think the argument from symmetry still holds, but it leads into a different conclusion. Since products (goods, physical assets) depreciate in value over time, money must too decrease in value. Hence you get inflation.
I believe that "natural rate of inflation" is driven by natural depreciation of goods and the free market mechanism that exchanges money and products as you describe.
Capital has multiple forms, I agree that physical goods generally depreciate over time. But there is also land, equity, and bonds and they all have their own market forces to deal with.
I'm hand-waving a lot of arguments and considerations with this statement, but from my perspective one advantage to 2-3% inflation is to incentivize owning capital that will outpace inflation. Land, equity, and bonds all have that potential.
Deflation may incentivize renegotiation of labor, but it also incentivizes hoarding of cash, which itself is not otherwise valuable. The value comes from it being passed around through the economy buying more assets. The more purchases -> the more money to be passed around -> the more opportunity to grow the economy. In a deflationary environment (at least in theory) this slows all of that down and decreases economic opportunity, which we generally don't want.
The average price of electricity in Europe is 29 cents. The average price in the US is 17 cents. Both have a very wide variance. I'm not sure if there's anywhere in Europe that pay more than PG&E customers in the US...
I deleted the numerical checks a while back after confirming the backward pass is correct to keep the code base lean - running https://github.com/markusheimerl/gpt/blob/main/transformer/a... is also somewhat of a confirmation that the backward pass is correct, since an analytically incorrect backward pass cant fit perfectly to synthetic data.
No, basic algebra and calculus has not changed.