When I looked into it a few years ago, Spotify used a custom protocol which was similar to BitTorrent but was not actually using BitTorrent. It had some clever features to optimize for the chunk of the song to be played and it had encryption for the DRM.