Skip to content

Comment on Similarity search and deduplication at scaleparent

Comments

Wouldn't that find articles that are semantically similar, rather than structurally similar (which I interpreted GP as wanting)?

In the structural-comparison case, I imagine you might have better luck with just doing cosine similarity across the term frequency vector or some such, possibly doing random projection first to reduce dimensionality.

Or really, an LSH would do the trick.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.