I think the most important part of this article is how people focus too much on evaluating interactions with the model, but not evaluating the whole system that enables a feature or workflow that uses an LLM.

This is totally true! And I've talked with people who have found that "the LLM is a problem" only to find that upstream calls to services that produce data to be fed into the LLM were actually the ones causing problems.