Mobile Diminished Reality for Preserving 3D Visual Privacy

Privacy is an essential human right. While the world has witnessed massive improvement in immersive technologies and devices in recent years, making them more ubiquitous and accessible in our daily lives, this has led to privacy implications where users' private and sensitive information can easily be exploited in inference attacks. State-of-the-art privacy-preserving approaches are either 2D, focused on protecting a single type of information like faces, or screens, or based on the least privilege paradigm. Preserving privacy in virtual environments, specifically 3D visual privacy in Mixed Reality (MR) applications, remains generally an unsolved problem. To mitigate the risk of private content inadvertently leaking from an MR user's environment, we propose a privacy-preserving Diminished Reality framework to obfuscate 3D objects while preserving the realism of a 3D environment during a live MR feed. We demonstrate the privacy preservation ability during 3D rotation around the sensitive object, real-time performance of our 3D privacy-preserving platform, as well as its implementation using local computations on mobile devices. The proposed framework enables users to select private objects to obfuscate by either enclosing them with 3D virtual objects or detecting their contours. Once identified, the objects are tracked using optical flow and obfuscated in 3D using three inpainting methods. The framework is tested on Android devices and evaluated using frames-per-second (FPS), Peak Signal-to-Noise Ratio (PSNR) and privacy metrics, achieving an average of 15 FPS for 15% mask sizes, PSNR between 25 and 27, and maintaining above 90% privacy.