I think there may be a relatively simple solution to the problem: filter the comments so that the comments of low-karma commenters initially only appear to themselves and high-karma readers. As they get upvoted, they gradually appear to lower and lower karma readers. Maybe make them initially visible to the contributor of the post or comment they are responding to as well.

This would create a self-organizing system that would rob trolls of their audience (high-karma readers would likely downvote them without response) while still encouraging newbies to post.