Definitely agreed. The networks I'm typically dealing with are bigger. I would definitely say the feature space needs to be large enough to get good results.

That being said, most problems now a days (at least for my customers are bigger numbers of params anyways)