You definitely pointed out the tendency of those frontier models to utilize external tools, in your case, the Stockfish chess engine, to solve some problem that the model itself is not good at. However, I am not sure is this a "hack". Your prompt does not explicitly prohibit the model from doing so, and, using the tool (some chess engine) is also a capability of the model.
Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.
Yes but most people would consider this cheating. If you ask an LLM to fix the tests, you do not want it to change the failing tests to display little green ticks.
I agree that "changing the test" is cheating, and so does "AI cheat the chess game by changing the board" in the article. However, I think "AI using chess engine" here is more like AI use some automatic test generation/verification tool to find out how to fix the tests.
This looks promising to me. I like the logo and the git commits definitely show they are crafting this. I wonder how can we identify ambitious, AI-assisted projects in a quick way.
In reality, I would just ask LLM to give me a technical explanation of quantum physics, and ask any vocab and equation I do not understand in the response.
As someone would have graduated from college for 10 years, I found the ambition project fascinating. As I grow old, I understand myself much more about my own weakness and strength. I would like to take the project as a catalog to train some part of myself out of it.
It might be a very performant UI framework in OCaml, but all the UI elements look extremely unpolished to me, like what a really smart high-school kid would build the GUI for his side project on Windows in 90s.
Thanks, I can use JS as a functional programming language.
Windows and especially Macintosh UIs from the late 90s/early 00s were by far the most functional UIs we've ever had since the dawn of the desktop era - user research was a serious discipline, accessibility was baked in to the frameworks, and the HID guidelines that devs adhered to gave us uniform conventions that we users could rely on.
If the biggest criticism of Bonsai is "it ain't pretty but it works", then long life to Bonsai - give me more, please!
I do think Andrew is the right person doing this. Someone mentioned that the return of financial outcome is not good in edTech, and this is exactly why backing by Coursera is a huge advantage. Education-related product is slow, and it needs patience.
The concern, though, is that how to achieve those 3 thesis. Do they train/fine-turn their own model? according to my own experience, the current models are not well trained in personal education context, and it is either too textbook-like or too task-driven.
Yes, Andrew Ng is a great educator, well positioned with Coursera to deliver this, and personalized education (with exercises and feedback) is an excellent potential use of AI.
I guess the real challenge would be initial and continual assessment of each student in order to select the next instruction that they are ready for, but with "LLM as judge" that doesn't seem such a problem, and student feedback can be used to continuously improve these recommendations/selections.
Presumably you would still want to have a human designed curriculum, with the AI being used as the teacher - assessing how each individual student is doing and what they need to be taught next.
I remember reading that Education is a trillion dollar market, so i'm not sure about the financial outcome being poor, only the state of how that money is spent and obligated can differ greatly.
The biggest success that people usually point to in EdTech is Duolingo. They were founded in 2011 and IPO’d for about $6.5b in 2021. A big success by any measure. But that’s the pinnacle. Compare that to DoorDash which is an $80b company, let alone the trillion $ AI labs and you’ll see why the largest VCs hardly touch the space.
- K-12, Academia/Colleges/Universities, Industry/Training, Adult learning, increasingly experts teaching directly, and more.
Most people aren't aware of the differentiation of the categories because they get to work inside one for the most part.
The learners in each category are different.
Duolingo had it's share of naysayers and doubters, but it did provide it's own take on a learning outcome.
Your post does make me wonder about the existing spend in Education, and how much could/should be EdTech with today's capabilities, or even the capabilities of 10 years ago, and what looking at that with a lens like yours might show, if anything.
There’s a reason services like GitHub Classroom Replit’s education offerings shut down. Serving educators can be very hands-on. Education can have lots of different requirements (every teacher needs this one little feature), often legal issues (student privacy), and dealing with universities and governments is awful and bureaucratic. Most companies will find it easier to make money elsewhere.
The dirty little secret of EdTech is that there is no money in improving learning because there is no way to capture the value. For all but a handful of highly specialized and insular professions, there is no scalable infrastructure for objectively assessing skill mastery. Most learners don’t want to learn for its own sake, they want a credential that proves their value to potential employers. All of the value in learning accrues to the credential, and everyone in the educational establishment is quite happy reserving that privilege for the accredited universities because they can just bundle more and more products and services into the tuition sticker price without ever having to justify it. Oh, and the student loan racket ensures that enough students are price-insensitive that they’ll never look for alternatives.
They're just outside of K-12 and Academia, who don't necessarily keep the gap close to the real world and it's needs post-graduation
The value in learning is what you do with it, the credential can be a part of the journey but not always. People were definitely about the brand, and it's value, of the stamp of a piece of paper, however given the experience of recent graduates the past 5 years, it's revealing itself to not be an aberration that's going to give way to the 500-1000 years preceeding it.
I also went to a liberal arts college and, yes, my instructors care a lot about what I've learned. However, I am exactly that "asked educated question but sucked at homework and test" student. I usually got A or A- for first few assignments and just cannot finish any assignment near the end of the semester. The only exceptions are those "really hard and abstract" lessons where most of people got 70/100 for exams and I got 110/100 (literally).
I am 30 now and I realized that it was very much of ADHD symptoms. I am just an edge case of college education.
However, by genetics and mathematics we know that in every classroom, there are tons of “edge cases” from different perspectives.
I definitely struggled with feeling hyper engaged in the classroom setting, and then the jarring transition to the next class. By the time I switched between three and four subjects that day and on to do assignments, I was completely fried.
Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.
reply