For what Ng is doing, I doubt that flowers would work because there are far too many too different cases of flowers.

Only if you see it as a supervised learning problem. An alternative is to find the nearest matches, after which you can let human intelligence make the final visual match. Often, the webpage containing the matched image will have enough context to identify the object being searched for.