The surprising thing here is that people think that LLMs are "powerful at solving math tasks".

That's not really surprising either. We have evolved to recognize ourselves in our environment. We recognize faces and emotions in power outlets and lawn chairs. Recognizing intelligence in the outputs of LLMs is less surprising than that. But the fact that we recognize intelligence in LLMs implies intelligence in them just about as much as your power outlet is happy or sad because it looks that way to you.