# Instance-optimality in differential privacy via approximate inverse sensitivity mechanisms

NIPS 2020, (2020): 14106-14117

EI

Keywords

Abstract

We study and provide instance-optimal algorithms in differential privacy by extending and approximating the inverse sensitivity mechanism. We provide two approximation frameworks, one which only requires knowledge of local sensitivities, and a gradient-based approximation for optimization problems, which are efficiently computable for a b...More

Code:

Data:

Introduction

- The authors study the estimation of a function of interest under differential privacy, where strong privacy protections usually decrease utility relative to non-private data analysis.
- Instance-specific lower bounds show that the risk the authors expect for an ε-differentially private algorithm on instance x is in general roughly ωf (x; 1/ε) for 1-dimensional functions and loss L(s, t) = |s − t| [4].
- Having described the difficulty of sampling from the inverse sensitivity mechanism in general, the authors develop two approximation frameworks that are applicable for a broader range of functions while maintaining some of the instance-optimality guarantees of the exact mechanism.

Highlights

- We study the estimation of a function of interest under differential privacy, where strong privacy protections usually decrease utility relative to non-private data analysis
- In an effort to improve the utility of private algorithms, it is of utmost importance to design mechanisms that adapt to the hardness of the underlying data
- Instance-specific lower bounds show that the risk we expect for an ε-differentially private algorithm on instance x is in general roughly ωf (x; 1/ε) for 1-dimensional functions and loss L(s, t) = |s − t| [4]
- Applications We study three problems that illustrate the methodological possibilities of the inverse sensitivity framework and its approximations in Section 4: mean estimation, PCA and linear regression
- Having described the difficulty of sampling from the inverse sensitivity mechanism in general, we develop two approximation frameworks that are applicable for a broader range of functions while maintaining some of the instance-optimality guarantees of the exact mechanism
- We complement our analysis with instance-specific lower bounds for vector-valued functions, which demonstrate that our mechanisms are instance-optimal under certain assumptions and that minimax lower bounds may not provide an accurate estimate of the hardness of a problem in general: our algorithms can significantly outperform minimax bounds for well behaved instances
- Our examples include (i) mean estimation with unbounded range, (ii) principal component analysis and (iii) linear regression, and show that our techniques yield private algorithms with better noise distributions resulting in improved utility, which in some cases can significantly outperform existing minimax-optimal algorithms

Results

- The authors provide utility guarantees for the exact and approximate inverse sensitivity mechanisms for vector-valued functions.
- The authors remark that using the smooth sensitivity framework to preserve pure differential privacy does not usually result in such high probability bounds due to using noise distributions with heavy tails such as
- Given a function f : X n → Rd, the authors prove instance-specific lower bounds on the loss that any private mechanism must incur.
- The authors' examples include (i) mean estimation with unbounded range, (ii) principal component analysis and (iii) linear regression, and show that the techniques yield private algorithms with better noise distributions resulting in improved utility, which in some cases can significantly outperform existing minimax-optimal algorithms.
- Using Lemma 4.1, the authors can efficiently sample (Algorithm 4 in Appendix D.1 which runs in O(n log n) time) from the inverse sensitivity mechanism.
- To appreciate the instance-specific upper bounds of Proposition 4.3, recall that existing private algorithms for empirical risk minimization of L-Lipschitz and λ-strongly convex functions achieve excess loss E[Ln(θ) − Ln(θn)]
- The inverse sensitivity mechanism outperforms smooth Laplace uniformly for every instance for natural families of sample-monotone functions.
- Proposition 2.2 shows that—for certain choices of approximations—the approximate inverse sensitivity mechanisms uniformly outperform the smooth Laplace mechanism for every instance.

Conclusion

- The smooth sensitivity framework requires adding noise with heavy-tailed distributions and unbounded moments to preserve ε-differential privacy, in contrast to the approximate inverse sensitivity mechanisms which has noise with exponentially decaying tails, resulting in better high-probability bounds and confidence intervals.
- The authors hope that this work—and instance-optimality in differential privacy in general [4]—can lead to a better understanding of the privacy-utility trade-off of private algorithms for the underlying data at hand.
- By exploiting the average-case nature of data in real life, the authors believe that the instance-optimal algorithms the authors develop can achieve satisfying utility with significantly stronger privacy protections for users.

Related work

- The most widely used frameworks for instance-dependent noise are smooth sensitivity [25] and propose-test-release [12]. The former adds noise that scaling with a smooth upper bound on the local sensitivity, and the latter adds noise scaling with a prespecified upper bound on the local sensitivity— whose validity the algorithm tests—in a neighborhood of the instance. Applications are numerous: Smith and Thakurta [30] develop an algorithm based on propose-test-release for high-dimensional regression problems, and Bun and Steinke [7] design noise distributions for smooth sensitivity and use them to estimate the mean of distributions with unbounded range. Other applications include principal component analysis [18], outlier analysis [26], and graph data [22, 32]. The inverse sensitivity framework is a distinct approach to instance-dependent noise that Asi and Duchi [4] investigate ([20, 29, 8] propose variants of the mechanism). Their results suggest that this framework, in contrast to smooth sensitivity and propose-test-release, is instance-optimal for a range of functions, and can have quadratically better sample complexity than smooth sensitivity mechanisms.

Funding

- Funding Transparency Statement Funding in direct support of this work: NSF CAREER CCF-1553086, ONR YIP N00014-19-2288, Sloan Foundation, NSF HDR 1934578 (Stanford Data Science Collaboratory), and Stanford DAWN Consortium.

Reference

- M. Abadi, U. Erlingsson, I. Goodfellow, H. B. McMahan, N. Papernot, I. Mironov, K. Talwar, and L. Zhang. On the protection of private information in machine learning systems: Two recent approaches. In 30th IEEE Computer Security Foundations Symposium (CSF), pages 1–6. IEEE, 2017.
- T. W. Anderson. The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. Proceedings of the American Mathematical Society, 6(2):170–176, 1955.
- Apple Differential Privacy Team. Learning with privacy at scale, 2017. Available at https://machinelearning.apple.com/2017/12/06/learning-with-privacy-at-scale.html.
- H. Asi and J. Duchi. Near instance-optimality in differential privacy. arXiv:2005.10630 [cs.CR], 2020.
- R. Bassily, A. Smith, and A. Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th Annual Symposium on Foundations of Computer Science, pages 464–473, 2014.
- M. Bun and T. Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography Conference (TCC), pages 635–658, 2016.
- M. Bun and T. Steinke. Average-case averages: Private algorithms for smooth sensitivity and mean estimation. In Advances in Neural Information Processing Systems 32, pages 181–191, 2019.
- M. Bun, G. Kamath, T. Steinke, and Z. S. Wu. Private hypothesis selection. In Advances in Neural Information Processing Systems 32, pages 156–167, 2019.
- T. Cai and M. Low. A framework for estimating convex functions. Statistica Sinica, 25:423–456, 2015.
- K. Chaudhuri, A. Sarwate, and K. Sinha. Near-optimal algorithms for differentially-private principal components. In Advances in Neural Information Processing Systems 25, 2012. URL http://arxiv.org/abs/1207.2812.
- J. C. Duchi, M. I. Jordan, and M. J. Wainwright. Minimax optimal procedures for locally private estimation (with discussion). Journal of the American Statistical Association, 113(521):182–215, 2018.
- C. Dwork and J. Lei. Differential privacy and robust statistics. In Proceedings of the Forty-First Annual ACM Symposium on the Theory of Computing, pages 371–380, 2009.
- C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3 & 4):211–407, 2014.
- C. Dwork and G. Rothblum. Concentrated differential privacy. arXiv:1603.01887 [cs.DS], 2016.
- C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology (EUROCRYPT 2006), 2006.
- C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Theory of Cryptography Conference, pages 265–284, 2006.
- U. Erlingsson, V. Pihur, and A. Korolova. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In Proceedings of the 21st ACM Conference on Computer and Communications Security (CCS), 2014.
- A. Gonen and R. Gilad-Bachrach. Smooth sensitivity based approach for differentially private PCA. In Algorithmic Learning Theory, pages 438–450, 2018.
- M. Hardt and K. Talwar. On the geometry of differential privacy. In Proceedings of the Forty-Second Annual ACM Symposium on the Theory of Computing, pages 705–714, 2010. URL http://arxiv.org/abs/0907.3754.
- A. Johnson and V. Shmatikov. Privacy-preserving data exploration in genome-wide association studies. In Proceedings of the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1079–1087, 2013.
- M. Kapralov and K. Talwar. On differentially private low rank approximation. In Proceedings of the Twenty-Fourth ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1395–1414, 2013.
- S. P. Kasiviswanathan, K. Nissim, S. Raskhodnikova, and A. Smith. Analyzing graphs with node differential privacy. In A. Sahai, editor, Theory of Cryptography, volume 7785 of Lecture Notes in Computer Science, pages 457–476.
- P. W. Laud, P. Damien, and T. S. Shively. Sampling some truncated distributions via rejection algorithms. Communications in Statistics: Simulation and Computation, 39(6):1111–1121, 2010.
- F. McSherry and K. Talwar. Mechanism design via differential privacy. In 48th Annual Symposium on Foundations of Computer Science, 2007.
- K. Nissim, S. Raskhodnikova, and A. Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the Thirty-Ninth Annual ACM Symposium on the Theory of Computing, 2007.
- R. Okada, K. Fukuchi, and J. Sakuma. Differentially private analysis of outliers. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 458–473.
- M. Reimherr and J. Awan. KNG: The K-norm gradient mechanism. In Advances in Neural Information Processing Systems 32, pages 10208–10219, 2019.
- O. Sheffet. Differentially private ordinary least squares. In Proceedings of the 34th International Conference on Machine Learning, pages 3105–3114, 2017.
- O. Sheffet. Homework 1 for Differential Privacy: Privacy-preserving Data Analysis. University of Alberta course CMPUT651, 2018. URL http://webdocs.cs.ualberta.ca/~osheffet/HW1W18.pdf.
- A. Smith and A. Thakurta. Differentially private feature selection via stability arguments, and the robustness of the Lasso. In Proceedings of the Twenty Sixth Annual Conference on Computational Learning Theory, pages 819–850, 2013. URL http://proceedings.mlr.press/v30/Guha13.html.
- C. Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, pages 197–206, 1956.
- J. Ullman and A. Sealfon. Efficiently estimating Erdos-Renyi graphs with node differential privacy. In Advances in Neural Information Processing Systems 32, pages 3765–3775, 2019.
- A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
- Y. X. Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pages 93–103, 2018.

Tags

Comments

数据免责声明

页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果，我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问，可以通过电子邮件方式联系我们：report@aminer.cn