I assume you already know about http://embed.ly but are concerned about security, like embedding unknown/untrusted html on your site?

I find your idea great and I have built something similar my self. I focused mostly on the interface and the fact that many times when watching a video you want to be able to switch between videos and make them fullscreen without interrupting their download. For example, when you have a long video on youtube that is split into many parts.

Playlists and ratings for videos is also quite useful in my opinion.

If you want some ideas check it out: http://lideo.me - you don't have to register/signup to start using it. You can do that later.