Because they also noticed that IPv4 has some design errors. So if you are anyway making something incompatible you can fix those and bring in new features.

IP packet having checksum that is also calculated on the hops remaining. Well, do you really really need that when most popular protocols TCP and UDP have also a checksum? So getting rid of it entirely is actually a smart move.

And then whole ARP, DHCP etc. Can we do something more sensible instead of that sort of thing. Different mindset, but reasonable attempt.