Does he? Then he's using the wrong word. Refactoring is changing the implementation while keeping the interface and semantics.

Which, I agree, is one of the shortcomings of unit testing: placing an additional burden on improving interfaces and semantics.