Paying down my mortgage is just a bit more expensive than renting my apartment was.

Unless you already have a substantial nest egg or took out an expensive mortgage, your life savings were substantially reduced by the down payment. In addition to a more expensive monthly payment there are property taxes, maintenance costs ($ and time), HOA fees, and other burdens depending on your region and property type.