The Foobar2000 ABX plug-in can do this for you. It automatically sets up a blind test, and calculates the probability that you can tell the difference between.

http://www.foobar2000.org/components