>For example we often see people thinking that because an LLM can explain how to do something that therefore it knows how to do it, like arithmetic. That's because if a human can explain how to do something, we know that they can. Yet for an LLM outputting a token sequence for an explanation of something, and outputting a token sequence for solving a problem statement for that problem domain are fundamentally different tasks.

Please explain to me the process currently happening in your visual cortex as you read this text.

The fact that neuroscience exists as a field (with so many remaining questions) shows that humans also do not understand how we can do all the things we do.