Search is a dirty business - in the sense that you have to index everything, even the content you don't like, in order to be truly useful (with he common sense exceptions for scummy, abhorrent and/or illegal content).

I could be wrong, but I think Apple might have an ideological problem with a lot of content. It's their call if they decide to filter that stuff out, but it's censorship, and I struggle to see how that would result in a "better" search engine.

EDIT: Applebot still has uses beyond a search engine though. I think Apple are being straightforward in it's explanation. It's for Siri and Spotlight.