@tao@mathstodon.xyz I want to clarify, although I am less sure about how to formalize this part, I do mean actually destroyed (not merely that the AI is 'wrong' and that wrongness is localized). To explain what I mean, it is analogous to how historians gather information about the ancient past. One might think that the more manuscripts a historian finds that are verified to be from a certain period the more information the historian has about that period. But this isn't necessarily true. Lets say a historian finds a manuscript A and B that tells us something about that period. And lets say for the sake of argument that what A and B says are true (but the historian has no way of proving that). The historian takes as evidence for the truth of A and B that A and B were written independently, and yet agree with each-other. But later, a new manuscript C and D are discovered, and they claim something about the event that contradicts something in B, and agree with each-other. In reality, A and B are true, but C and D contain untrue rumors that spread about the event. In this instance, the historian has more information than before, but is less certain about the truth of A and B, since C and D undermined the validity of B. So, the discovery of C and D destroyed some information the historian previously had about the event, since it made the historian more uncertain. Similarly, when an AI adds a widespread amount of noise, it becomes less and less clear what is true/important/valuable etc, destroying previous repositories of information.