This is an area of research called natural image matting that has been going on for over 20 years.

Some influential papers include:

https://www.microsoft.com/en-us/research/wp-content/uploads/...

https://webee.technion.ac.il/people/anat.levin/papers/Mattin...