Your heavily crafted implementation docs will be gabage since implementation can be changed easily.

I've heard this before but I'm not convinced. I'm sure lots of product teams use Agile and still manage to have thorough, up to date docs. You just need to factor doc updates into your issues.