I'm thinking that this could be a fairly easy problem and the RDBMS may be making it harder. This is probably a tricky indexing situation, so a lot of queries might be effectively unindexed. I'd like to see what the performance would be using really simple methods, like dumping it to a TSV file and doing regex searches with grep or awk. It might be surprisingly fast.
Comments
What are some typical queries look like? Several minutes for searching 7M records doesn't sound right. Are the columns indexed properly?
I'm thinking that this could be a fairly easy problem and the RDBMS may be making it harder. This is probably a tricky indexing situation, so a lot of queries might be effectively unindexed. I'd like to see what the performance would be using really simple methods, like dumping it to a TSV file and doing regex searches with grep or awk. It might be surprisingly fast.