Open CV is free and decent to start with. Lots of wrappers exist for it so you could pretty much use it with whatever environment you are using. If your idea is validated you could then look into some of the more expensive commercial libraries out there. They would give you more features flexibility scale and (maybe) better results.