What approach would you be going for initially for deduplicating same urls? | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		gerenuk on March 18, 2019 \| parent \| context \| favorite \| on: Ask HN: Identifying duplicate data from a large da... What approach would you be going for initially for deduplicating same urls?

verdverm on March 19, 2019 [–]

You might look at a real data processing system, like something from the Apache projects

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact