Simon Singh's book was my introduction to crypto and the wonderful Mathematics and mathematicians behind its vivid history. I had it with me all the time while doing the crypto-class.com online course a couple of months ago for a good historic perspective supporting Prof. Boneh's hardcore crypto topics. Highly recommended. Actually, his Fermat's Last Theorem book is also fascinating if you are interested in Math history.