Explainability as a form of system closure

There’s a question that’s ringing out all over the machine learning world these days: what the hell counts as an “explanation” of a complex system? If you listen for a while to conversations in the labs of practitioners, the forums of mechanistic interpretability, the ivory halls of theory, or even the labs of AI-savvy neuroscientists or biologists, you’ll start hearing this question thrumming…