It is not obviously the case that all software systems we rely on need to be understood by humans. That was just a coincidental property of the last 80 years of software systems. Humans rely on many systems that we don’t understand. The systems just need to be observable and, ideally, predictable.