From a textualist standpoint you're right, but in practice the Supreme Court has expanded the First Amendment to prohibit government actions (besides just lawmaking) that would deter speech.