LLMs with User-defined Prompts as Generic Data Operators for Reliable Data Processing
2023 IEEE International Conference on Big Data (BigData)(2023)
摘要
Data processing is one of the fundamental steps in machine learning pipelines
to ensure data quality. Majority of the applications consider the user-defined
function (UDF) design pattern for data processing in databases. Although the
UDF design pattern introduces flexibility, reusability and scalability, the
increasing demand on machine learning pipelines brings three new challenges to
this design pattern – not low-code, not dependency-free and not
knowledge-aware. To address these challenges, we propose a new design pattern
that large language models (LLMs) could work as a generic data operator
(LLM-GDO) for reliable data cleansing, transformation and modeling with their
human-compatible performance. In the LLM-GDO design pattern, user-defined
prompts (UDPs) are used to represent the data processing logic rather than
implementations with a specific programming language. LLMs can be centrally
maintained so users don't have to manage the dependencies at the run-time.
Fine-tuning LLMs with domain-specific data could enhance the performance on the
domain-specific tasks which makes data processing knowledge-aware. We
illustrate these advantages with examples in different data processing tasks.
Furthermore, we summarize the challenges and opportunities introduced by LLMs
to provide a complete view of this design pattern for more discussions.
更多查看译文
关键词
Large Language Models,Data Modeling,Data Cleansing,Data Transformations,Design Pattern
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要