"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
arxiv(2024)
摘要
Organizations rely on machine learning engineers (MLEs) to deploy models and
maintain ML pipelines in production. Due to models' extensive reliance on fresh
data, the operationalization of machine learning, or MLOps, requires MLEs to
have proficiency in data science and engineering. When considered holistically,
the job seems staggering – how do MLEs do MLOps, and what are their
unaddressed challenges? To address these questions, we conducted
semi-structured ethnographic interviews with 18 MLEs working on various
applications, including chatbots, autonomous vehicles, and finance. We find
that MLEs engage in a workflow of (i) data preparation, (ii) experimentation,
(iii) evaluation throughout a multi-staged deployment, and (iv) continual
monitoring and response. Throughout this workflow, MLEs collaborate extensively
with data scientists, product stakeholders, and one another, supplementing
routine verbal exchanges with communication tools ranging from Slack to
organization-wide ticketing and reporting systems. We introduce the 3Vs of
MLOps: velocity, visibility, and versioning – three virtues of successful ML
deployments that MLEs learn to balance and grow as they mature. Finally, we
discuss design implications and opportunities for future work.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要