It might not be the perfect analogy, but it does show some important parallels and differences. Compilers reason to some extent - for example type inference is definitely some deductive reasoning that I would have to do if the compiler didn't do it for me.
The hallucinations point is a difference and one that we haven't quite learnt to handle yet. Generally, compilers are deterministic (even it it might not feel like it sometimes) and there's a lot of useful properties that come from that.
I have recently put Claude into a CI/CD pipeline and it's made me realise how much I rely on pipelines being deterministic.
>>
Compilers reason to some extent - for example type inference is definitely some deductive reasoning that I would have to do if the compiler didn't do it for me.
<<
But this "reasoning" is done within a strict ruleset and includes "just find the optimum in X", and there is no space for "language interpretation", so I wouldnt call this reasoning compared to that type what LLMs are doing.
LLMs operate with fewer constraints and consequently a larger search space, but there is an apparent continuum seen in programming where constraints are relaxed and the search space grows. It's not just standard compilers and LLMs. There is a whole world of computer science found in-between.
In order to consider LLMs to be something different then there needs to be a clear dividing line and I'm not sure that there is. It seems more likely that it is the same type of reasoning, but that LLMs take it to a further degree.
Not really. Maybe more so than they used to be thanks to deterministic algorithms becoming faster, but most compilers still aren't deterministic by default. There are still advantages to non-determinism in compilers, like not having to worry about thread execution order. To be fair, most compilers these days allow you to optionally enable determinism. Then again, LLMs also allow you to optionally enable determinism.
If you consider hidden inputs to be part of the input space then the statement is obviously true, but then all computation is deterministic. It is fundamentally impossible for computers to be anything but deterministic. Maybe that is what the author meant, but it seems unlikely as it doesn't add anything. Mentioning determinism usually implies that there is an non-deterministic alternative, which in computing can only mean where there are hidden inputs.
What's in need of explanation? Computers are designed to be deterministic, and LLMs run on computers, so LLMs must also be deterministic.
There caveat to that is where external inputs give the illusion of non-determinism. Thread execution order, like we already discussed, is one such example. Technically still deterministic if you understand the external inputs, but for the sake of discussion we can consider external inputs to be non-deterministic. Which is why compilers usually end up being non-deterministic by default. However, computers are designed to be deterministic so there are ways to avoid introducing those external inputs, albeit often at the cost of performance. LLMs and compilers alike can be run deterministically.
LLMs have one additional property not typically found in a traditional compiler — a call to rand() — but rand() is also deterministic when configured with a constant seed and can also be turned off completely by setting temperature to 0.
Ok, technically you’re right. You can make LLMs deterministic if you make literally every other aspect of execution deterministic as well - self-hosting and open-weights model with identical, reproducible logits, batching, FP math, etc.
Yup. Just as you can technically make compilers deterministic if you control all aspects of execution, but it isn't typical to do so. In fairness, reproducible builds have started to become considered increasingly important recently over concerns like supply chain attacks so attitudes are starting to change, which is why most compilers nowadays give you determinism as an option, but the historical view was that the user didn't really care what the compiler spit out, so there was rarely any effort to make them deterministic.
What's old is new again. LLMs are back to believing that the user doesn't really care what gets spit out, preferring performance over determinism, but it is likely that eventually we'll find increasing importance in reproducible builds again — especially when, like we saw with compilers, the hardware/algorithms catch up and the performance cost of determinism isn't there anymore. The industry is highly cyclical. These aren't hard technical limitations, just human choices mixed with a humorous dose of "this time is different". But in hindsight the times never end up being different, do they?
I have been feeling that something - a few things - has to change about Agile rituals given a team is using Claude extensively. I haven't yet found any thoughtful writing on the topic.
The most obvious impacts have been shorter synchronous conversations for planning and refinement because so much more specification is written. I've also found myself doing more spiking.
The thing is, all the changes I see to Agile are matters of degree, not fundamental breaks with the Agile manifesto or tradition. I'm not sure whether that's because a) all that's needed is tweaks or b) because the existing way of doing Agile is so in-grained that it's hard to see that it doesn't fit today's reality.
yeah sometimes you end up making something very cool almost by accident. i tried to make shell pipelines properly concatenative like forth and friends and that had a lot of unexpected neat digressions in metaprogramming
It's not so valuable to assess the current state - what the impact of using AI is today. From personal experience it feels like overall impact on productivity was not positive a couple of years ago, might be positive now and will be positive in a couple of years. That means by assessing the current state of impact on product where just finding where we are on that change curve. If we accept that trend is happening then we know at some point it will (or has) pass the threshold where our companies will fall behind if they're not using it. We also know it takes a while to get up to speed and make sure we're making the most of it so the earlier we start the better. That's the counter arguement that we could wait for a later wave to jump on but that's risky and the only potential reward is a small percentage short-term productivity gain.
So you're saying instead of assessing the current capabilities of the technology, we should imagine its future capabilities, "accept" that they will surely be achieved and then assess those?
I would assess the directionality and rate of the trend. If it's getting better fast and we don't see a limit to that trend then it will eventually pass whatever threshold we set for adoption.
Spending is paid for out of tax and all those people will have paid tax. Paying less tax than someone else doesn't make a free rider. Deliberately opting out of an obligation while taking from the group makes a free rider.
I would def use this if there was a return “select …” option. There are heaps of scenarios where sql is modified based on parameters. If no doc string just use the return value maybe?
Our queries are typically large, not 3-5 liners.
(Filter view queries where you might add additional CTA’s to provide the necessary filter conditions, but aren’t desirable if particular filter parameter is nill, etc.)
Just keep in mind best practice is to use the built-in parameter interpolation that comes with your db library, since it handles escaping SQL injection for you.
Be very careful if you ever use bare string formatting to construct your queries.
This is spot on. I'd love it if it was possible to get the integration with the QGIS ecosystem. It could open source integrations or even a commercial offering that just joins things up in a cohesive way just something that enables a more smooth collaboration model.
That's an insightful nuance. I've seen you just create divisions in organisations because while it is a really fully featured desktop application, it implies a way of working that doesn't play well with the cloud, which creates barriers between experimenting and production.
As other comments have said, I'd prefer other solutions to get by all the tests to run faster. It would be interesting to see if it could be used to prioritise tests - get the tests more likely to fail to run sooner.
The hallucinations point is a difference and one that we haven't quite learnt to handle yet. Generally, compilers are deterministic (even it it might not feel like it sometimes) and there's a lot of useful properties that come from that.
I have recently put Claude into a CI/CD pipeline and it's made me realise how much I rely on pipelines being deterministic.