> my position is that we are completely ignorant as to how LLMs do what they do

That's simply wrong.

Whole teams of very smart people at places such as Anthropic are working on this exact problem (the field of "mechanistic interpretability"), and have made considerable headway, and have published their results.