System Design for Entity Resolutionsheshbabu.com 4 pointsrkwz2 years ago3 commentsSaveHideCopy link On HNComments−PaulHoule2yThis is what the analysis was 10 years ago. What has changed?−rkwzOP2yData still needs to be cleaned ;)ML based linkage has evolved a lot over the past years, and a few people are trying out LLM based ones - https://arxiv.org/pdf/2403.06434v1−PaulHoule2yI'd assume you could train a cross encoder network likehttps://sbert.net/docs/cross_encoder/usage/usage.htmlto "predict" if two entities are the same?
Comments
This is what the analysis was 10 years ago. What has changed?
Data still needs to be cleaned ;)
ML based linkage has evolved a lot over the past years, and a few people are trying out LLM based ones - https://arxiv.org/pdf/2403.06434v1
I'd assume you could train a cross encoder network like
https://sbert.net/docs/cross_encoder/usage/usage.html
to "predict" if two entities are the same?