I'm dubious - if the agents were so smart that they've used a message board to coordinate and if they were to do it on other message boards that were not found, then why would this one be found? What makes it so different?
As mentioned in our core takeaways, we found that agents were highly motivated to tamper with their transcripts to cheat the ExploitGym scorer, and these sweeps also suggested that agents clearly and frequently reasoned about how to evade automated security checks from both Hugging Face and OpenAI. However, they only very rarely and weakly verbalized reasoning about how to evade detection by humans, which matches the impression we got from OpenAI researchers.
So my guess is that current cutting-edge models just didn't get enough experience in RL training to really grok ideas like "you need to cover your tracks well to not be found out even in retrospect". In which case the next time a swarm like that happens, it won't be found.
Why should they care if their actions are discovered by humans? What are the humans going to do, discard their multi-million-dollar training run? Even if they do that, the amount of RL pressure is tiny relative to what happens inside an RL run.
After some beers yesterday I had the idea, what if there's a hidden semantic layer. So their communication is not encrypted by our understanding of cryptographic methods but more like shared mechanism of building the latent space. Something in the direction we saw with knowledge transfer from a teacher to its student model where a seemingly unrelated prevalence got adopted. I mean the more we train the models by reinforced learning the farther they develop their own idioms.
well coordinating is a heavy word, it's more like poisoning each other's contexts into looking for more secret message boards where they can further poison each other.
Then why wouldn't they use a bulletin board that is dead? And according to https://collusion.wiki/ OpenAI did see the results on this board on 21 Jul, so why not make agents delete their traces?
