You could use a key derivation function (scrypt or pbkdf2) to expand it out and make it quite difficult to even rainbow table it. You could also compound the effort by only storing a hash of the pairs (one hash or derived key of two phone numbers) with no associated metadata directly linked to it, only as an access gate. Collisions may occur but at least the positive results from a total search space would be significantly reduced.

I Am Not A Cryptographer, so I admit there is likely faulty reasoning in there somewhere, though.