If you're agnostic regarding the metaphysical details then you're not a materialist. Materialism is a strong metaphysical position, not a form of agnosticism.