If your construction cost is significantly greater than your resale value, and you intend to borrow a large portion of this cost, then there is not enough collateral for the lender to recover their investment.

A house which you will treasure, but is considered undesirable by the masses will be a problem.