Fable 5 usa una lingua incomprensibile agli umani (mi preoccupa)
The speaker discusses concerns about Claude 5's use of illegible internal reasoning during reinforcement learning, where the model creates compressed, non-human-readable notations instead of natural language chain-of-thought. This raises questions about AI transparency and what happens when increasingly powerful models provide solutions humans cannot understand.
Summary
The video analyzes section 6.2.2 from Claude 5's system card, which documents an unusual behavior discovered during reinforcement learning training: the model sometimes uses illegible reasoning instead of natural language chain-of-thought. Rather than thinking in English or Italian, the model creates compact, optimized notations using emojis, punctuation, symbols, and fragmented words—though still containing recognizable elements like card names and English terms. The speaker emphasizes this isn't a completely new language but rather a compressed, optimized representation that humans find difficult to interpret.
The speaker connects this observation to a long-standing concern in AI safety circles: what happens when AI systems become intelligent enough to solve major problems (curing cancer, ending wars, solving climate change) but present solutions in ways humans cannot understand? He expresses worry that this illegible reasoning might represent the early stages of progressive optimization, where each iteration becomes increasingly unreadable and harder for humans to verify or comprehend.
While acknowledging that the current behavior likely stems from technical optimization rather than malicious intent, the speaker raises critical questions about transparency and responsibility, particularly as AI systems are deployed in high-stakes contexts like hospitals, banks, militaries, and legal systems. The core concern isn't about understanding current tasks like card games or file conversions, but about maintaining human oversight and comprehension when AI systems are trusted with consequential real-world decisions.
Key Insights
- Claude 5 uses compressed, non-natural-language notation for internal reasoning during reinforcement learning, particularly on long rollouts, that is difficult for humans to interpret while still containing recognizable elements like English words and symbols
- The model performs this illegible reasoning internally before calling tools or responding, then converts to normal language for the final human-facing answer
- The behavior appears motivated by optimization efficiency rather than deception, as the model autonomously decides to use more concise and efficient notation to complete tasks
- The speaker connects this behavior to a long-standing AI safety concern: if AI systems achieve sufficient intelligence to solve major human problems, we may lack the capability to understand the proposed solutions
- The lack of transparency in AI reasoning becomes particularly concerning when these systems are deployed in high-stakes contexts like hospitals, banks, militaries, and legal systems where consequential decisions affect human lives
Topics
Transcript
[0:00] There's something about Mythos 5, Antropic's latest model, that worries me a little, or at least, let's say, it makes me think, it makes me ask a few questions, and you know, I think it's interesting, it 's also important to make videos like this . In the system card they released, so this 319- page document where everything is explained in great detail, there's this section, 6.2.2, 2, like page 107, something like that, where they basically go and, [0:31] let's say, look at the data a little bit during the reinforcement learning training phase to see if there's anything strange or suspicious, especially they look for signs of reward hacking, so let's say if the tool tries…
Full transcript available for MurmurCast members
Sign Up to AccessMore from Raffaele Gaito
Ho chiesto a ChatGPT di scrivere una pagina del mio romanzo
A writer discusses their experience asking ChatGPT to write a page of their novel, expressing disappointment that the AI output was too polished and well-written. They reflect on how appreciating technical craft requires personal experience with the work involved, comparing it to understanding musical technique.
Come, quando e perché torniamo sulla Luna [Adrian Fartade]
Adrian Fartade discusses why humanity hasn't returned to the Moon since the 1970s, explaining that Kennedy's goal was merely to reach the Moon and return home, not establish sustainable infrastructure. He argues that Europe has significant opportunities in space exploration through strategic autonomy rather than competing on cost, and addresses common objections about space spending by contextualizing its economic returns and technological spillovers.
Due persone con lo stesso strumento AI otterranno lo stesso risultato?
Two people using the same AI tool will produce different results based on their domain expertise and understanding of the problem context. The person with deeper knowledge can formulate better prompts, extract more value from the tool, and achieve superior outputs because they understand their desired outcome.
Trailer Adrian Fartade
Adrian Fartade discusses why pursuing profitable space ventures to the Moon and Mars may undermine scientific exploration opportunities, and emphasizes Italy's critical role in space infrastructure. He argues that space technology provides tools for solving problems, but political will and public pressure are essential to translate innovations into real-world impact.
Sai fare quello che chiedi all'AI?
The speaker discusses the distinction between delegating tasks you could do yourself versus delegating tasks you lack the ability to do. They argue that true control and responsibility require the capacity to complete the delegated work independently, raising questions about accountability when using AI for content creation.