Streaming video is not like static files from a torrent. The term "popular content" is a bit of a misnomer. Nominally with streaming a single uploaded video will be encoded into a number of different versions at different frame sizes and bitrates packaged into individual stream segments. So while a particular video URL might be popular, the actual video segments served to viewers will depend heavily on the configuration/environment of all the viewing devices.

So a FHD stream for a video could be super popular while a 480p version might not have any viewers. When some 480p viewers load the stream they don't get any benefit from the P2P/cache popularity of the FHD stream.

This is compounded by the fact that for video time to first byte doesn't matter very much. A decoder can't do anything with a single byte or even a partial stream segment. It's the time to locally buffer a decodable stream segment and a steady reception of segments as the play head reaches their time stamp.

If a segment gets held up (as happens with P2P) it can't be decoded so you either pause playback or drop frames (if you've got later segments available). You want the media segments to cover at minimum a single GOP (Group of Pictures) which is a relatively large I-frame and the following smaller P and B frames. If you were to tune the video segmentation such that each segment was some tiny time slice it would balloon the bitrate since you need way more I-frames. Super long GOPs don't help either because segment drops end up ruining playback until a new I-frame is received.

Streaming video is hard enough with a reliable CDN, adding P2P makes much harder and orders of magnitude less reliable.