They assumed performance wouldn't be a problem

We did pretty extensive performance tests, but not for long enough. We load tested tested for hours at a time, and the problems started to show up after days of production load and really compounded after that. It's probably a topic for another post, but the performance of database tables with a lot of churn (rows being frequently inserted and deleted) really degrades over time.