MARFE and ALSPH: a high-performance framework for scalable and efficient image deduplication

Abstract

Image deduplication has become increasingly important with the rapid growth of digital image repositories in cloud storage, multimedia systems, and large-scale visual databases. Existing approaches face a trade-off between detection accuracy and computational efficiency. Deep learning methods provide high robustness but require substantial computational resources, while traditional perceptual hashing and handcrafted features offer faster processing but have limited resilience to image transformations. To address this issue, this paper proposes an efficient image deduplication framework integrating multi-scale adaptive region-based feature extraction (MARFE) and adaptive locality-sensitive perceptual hashing (ALSPH). MARFE adaptively selects information-rich image regions to preserve discriminative visual characteristics while reducing redundant computations. ALSPH generates compact binary fingerprints through covariance-guided projection learning for efficient and robust similarity matching. The framework was evaluated on benchmark and web-scale image collections, with additional scalability analysis performed at dataset sizes of up to 10 million images. The framework achieved an F1-score of 0.979, processing speed of 187 images per second, and 49.3% storage savings, demonstrating a good balance between accuracy, efficiency, and scalability.