A Progressive Approach for Duplicate Detection with Map Reduce

Duplicate detection is a crucial step for data quality and data integration. Cloud infrastructure is a popular paradigm that enables efficient parallel processing of data and computational intensive tasks such as duplicate detection on a very large datasets. Different cloud computing applications make use of a programming model called MapReduce that supports parallel execution of data-intensive computing tasks in cluster environments with up to thousands of nodes. Most of the duplicate pairs in the duplicate detection process can be identified as early as possible by using a method called Progressive Sorted Neighbourhood. To reduce the typically high execution times, this paper investigate how progressive sorted neighborhood for data intensive duplicate detection can be realized in a cloud infrastructure using MapReduce. This paper mainly focuses on the use of MapReduce programming model aiming at a highly efficient duplicate detection implementation for a very large datasets.