InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
CoRR(2024)
摘要
Recent advancements in language technology and Artificial Intelligence have
resulted in numerous Language Models being proposed to perform various tasks in
the legal domain ranging from predicting judgments to generating summaries.
Despite their immense potential, these models have been proven to learn and
exhibit societal biases and make unfair predictions. In this study, we explore
the ability of Large Language Models (LLMs) to perform legal tasks in the
Indian landscape when social factors are involved. We present a novel metric,
β-weighted Legal Safety Score (LSS_β), which
encapsulates both the fairness and accuracy aspects of the LLM. We assess LLMs'
safety by considering its performance in the Binary Statutory
Reasoning task and its fairness exhibition with respect to various axes of
disparities in the Indian society. Task performance and fairness scores of
LLaMA and LLaMA–2 models indicate that the proposed LSS_β metric can
effectively determine the readiness of a model for safe usage in the legal
sector. We also propose finetuning pipelines, utilising specialised legal
datasets, as a potential method to mitigate bias and improve model safety. The
finetuning procedures on LLaMA and LLaMA–2 models increase the LSS_β,
improving their usability in the Indian legal domain. Our code is publicly
released.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要