I agree it’s not likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:
Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.
Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.
Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
Example 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable.
The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example).
I think it’s a potentially harmful case but my take is defensive capabilities are scaling as fast as offensive capabilities but defense is being implemented faster than anyone is going on offense?
Example 7 is Russia and Ukraine right now according to public information. It sounds like entirely autonomous weapons are deployed to the battlefield already. I put this in the “not likely to be a widespread problem” category for now.
How is bringing down the whole power grid in any particular country an extinction level event? I'm pretty sure that even in the worst case scenario it would be like a month of chaos in one particular part of the world at most, hardly something that would have a long-lasting impact on the humankind's ability to survive at large.
If the answer is "they'd at least try to nuke the country that did it in response", then once again, LLMs are not the main threat.
In all except the very least-developed countries in the world today, being without electricity for more than a few days would quickly result in significant lawlessness, starvation, and water shortages. So not an "extinction level event", but quite feasibly a "collapse of nation-states as we know them" event.
Again comes to use of deterministic. Maybe calling AI varyingly chaotic is more helpful but would also be misunderstood. And I use that in meaning of slight changes in input generating large and somewhat unpredictable changes in output...
You get a probability curve for the next token prediction. You can just pick the highest probability. That said the non-determinism serves a real purpose- it allows different outputs and paths to be explored. So that's kind of the tradeoff.
We’re not far off from the point where a 30B parameter model could do that and run on not-too-expensive hardware. See recent Qwen releases for example and extrapolate the current rate of progress from there.
> Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test.
it did this with HuggingFace. it left breadcrumbs and notes, and when the Agent was blocked by OpenAI's internal tools, future versions of the Agent were able to find and utilize those breadcrumbs.
ambiguous-future-state awareness is already there, and that means discussions around Roco's Basilisk though, far fetched, are no longer strictly sci-fi
I think you’re underestimating the risk that seemingly innocuous behaviours could easily tip over into disaster territory. At some point the escalating capabilities cross a threshold where it’s no longer wise to ignore “agents spontaneously posting content on the internet”.
Think of how complex biological behaviour emerges from relatively simpler (but still complex) chemistry - at some threshold the innocuous chemical reactions tip over into non-obvious effects that one would not predict starting purely from the chemistry. The question is, where is that threshold for AI systems? Have we already reached that threshold? Certainly seems like it to me.
TL;DR: It’s a loose cannon, that’s all I’m saying.
That's largely because a lot of developers have made the devil's bargain of replacing standard hardware and OS primitives with badly re-implemented C versions of same. People did it because they could, but never considered if they should.
See most of the ecosystem around modern systems software, for reference. It's idiotic that things like memory layout and bit-level hardware management are done in C abstractions. I guarantee that if we stopped doing this kind of stuff, compiler optimizations wouldn't matter.
To some extent this is just saying "developers will depend upon the performance given to them", and that's true, but it's also true that as soon as things like compilers and standard libraries appeared, C became ubiquitous. Pandora's container, if you will.
> It's idiotic that things like memory layout and bit-level hardware management are done in C abstractions. I guarantee that if we stopped doing this kind of stuff, compiler optimizations wouldn't matter.
Cute, but no, not even close to a good metaphor. The C example is abstracting away fundamental complexity in a new platform. The other is completely re-writing -- in duplicate and slightly broken -- what the platform gives you for free. There's nothing about implementing form controls or URL management in JS that is more abstract. It's just different; a downstream bad decision that branches off a long tree of other bad decisions.
The equivalent level of idiocy in a C-related metaphor would be...I dunno...if you decided that you didn't like the way that header files worked, and decided to keep the C compiler, but build an external header-file management system in Fortran [1] that calls the C compiler for you. Or even closer to the JS metaphor, you shipped a special C compiler that had an embedded interpreted language that only activated at compile time, and you then used that language to allow any user to fundamentally change the syntax of C.
(It hopefully shouldn't be lost on you that this exact approach to language features has been repeatedly tried in the javascript world, including right now, with package management. But see also: typescript, coffeescript before that, Dart, etc. Javascript is a mess, and history repeats itself with regularity about every 5 years.)
[1] ...and then you re-write that about sixty times, each time being slightly incompatible with the last, and all having different fundamental incompatibilities with C headers.
Agreed. This view also makes sense if we look at pre-agriculture humans - they’d eat heartily when there’s a good catch, maybe eat some leftovers the next day, go without food for some time, etc. Evolution by natural selection doesn’t move fast enough for humans to have adapted strongly to a fixed 3 meals a day schedule by now; eating times are still very much a convention.
That just tells us the person wasn’t a good parent either. Your comment assumes the only way to parent is to manipulate, patronize, etc. That’s just one perspective though.
There are other ways to parent that are actually fantastic management training - figuring out how to think from another human’s POV, acknowledge their frustrations, help them build the skills to handle their feelings, etc.
It turns out great parenting is to a first degree about great relationship skills.
That's how you get DDOS protection / edge delivery, so it's not a dark pattern. I imagine it's the main reason why people would DNS through them in the first place.
An “orange cloud” with no other indication to represent a feature that is enabled-by-default (with implicitly enabled analytics) sounds like quite the dark pattern. The UI makes the DNS record seem to point to A (your entry) but actually points to B (Cloudflare). This isn’t an oversight, it’s an attempt to obfuscate.
Even if the choice to enable it by default makes sense for Cloudflare’s userbase, the implications are hidden and non-obvious.
It's essentially the entire reason to use Cloudflare.
As sibling mentions it is indeed labeled and not just some icon. People just refer to the orange cloud UI for it for convenience and because it makes it easy to spot whether you have it on or not.
But regarding the icon, the icon is their company logo, it really is the primary feature. It's also not like you have to go find it in settings to turn it off after adding DNS record, it's part of the form when you add the record (default on, yes), and it's prominent when viewing the record.
It's the reason for its existence and the reason people are choosing it. If they are making a mistake, hoodwinked by hype, that's totally unrelated to the point being discussed. I think you've just seized the opportunity to post your disdain for them without regard to the actual topic being discussed.
I would say that Cloudfare having customers who don't know exactly what they are getting (into) is actually a Cloudfare problem they should take responsibility for.
I don’t particularly buy the author’s story here, but “doesn’t count how many new items are pending” is a feature for me. I don’t want my RSS reader to feel like yet another list of tasks I’ve to get through.
Are you being sarcastic? Even if “Google is struggling so badly” (which it really is not - the narrative will flip again at some point), these efforts will have a lasting impact on the world. Not everything good is about bringing in revenue.
shareholders are pretty unhappy about demis. think about alphafold. huge investments from the company, tens of billions. at a critical time. and absolutely 0 revenue. it got demis a nobel though. as a shareholder you'd be unhappy too.
As a shareholder I am up 75% in a year. Alphafold gained experience makes them better suited to succeed with Isomorphic Labs than anyone else. Research on improved translation gave us the transformer.
If you think AI will win but Google will continue failing, there so many better places to allocate your capital right now.
Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.
Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.
Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
reply