Skip to content

Comment on 4.5B Posts Scraped from TikTokparent

Comments

It's that mix of dense, impressive sounding jargon, but even scanning across it raises glaring problems. Like, if you have 4.5 billion videos on HF, and it's 289GB, it's about 60 bytes per video. Checking the column fields as well, there doesn't seem to be any video files*.

There's an `is_video` column, perhaps containing a lot of 0's

More directly, there simply aren't any video files uploaded. It's just parquet files, which contain no video columns (I'm not even sure if it supports it).

Seems the expectation is the purchaser uses the info/parameters in the 289gb file to decide which videos to download, then uses the /v1/video/info endpoint to download individual videos.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.