Penetralium: Privacy-preserving and memory-efficient neural network inference at the edge

The proliferation of artificial intelligence and edge computing has led to an increase in the deployment of proprietary deep learning models on third-party edge servers or devices to power mission-critical applications. However, this trend raises concerns about model privacy, particularly on untrusted edge platforms. Protecting model privacy in such scenarios requires addressing challenges such as untrustworthy model deployment environments, resource-constrained Trusted Execution Environments (TEE), and vulnerability to privacy inference attacks. To address these challenges, this paper proposes Penetralium, a system-algorithm jointly optimized model inference system on edge computing platforms. Penetralium runs models in the TEE by building an underlying computational engine. We propose an adaptive decomposition algorithm that builds a computing pipeline for models, which adapts to the underlying trusted components. Additionally, Penetralium uses a lightweight confidence score perturbation policy to protect against advanced privacy inference attacks on deep learning models. Experimental results demonstrate that Penetralium provides strong security guarantees with reasonable performance. The system not only reduces inference latency and memory consumption overhead but also improves the overall robustness of the system against advanced attacks.