I wonder if someone has done analysis on what features make browsers so complicated.

There is a visualisation of the chrome binary here: http://neugierig.org/software/chromium/bloat/ I'm not sure how up to date it is now, but it gives a vague idea.

I guess a large chunk of this stuff should be in the OS, so that other apps could benefit from it. And another large part of it should be in plugins, so the browser can benefit from all the codecs on the system, for example.

That is the case for Safari for example (using PDFKit for pdfs etc and system codecs for video and audio). Mozilla like to bundle everything with Firefox because they view Firefox as an OS itself, rather than just another application for viewing html documents. Most of the huge size and complexity in modern browsers is due to people trying to turn the browser into an operating system: https://en.wikipedia.org/wiki/Inner-platform_effect