Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge
CoRR(2024)
Abstract
Large language models (LLMs) have shown great potential for the automatic
generation of feedback in a wide range of computing contexts. However, concerns
have been voiced around the privacy and ethical implications of sending student
work to proprietary models. This has sparked considerable interest in the use
of open source LLMs in education, but the quality of the feedback that such
open models can produce remains understudied. This is a concern as providing
flawed or misleading generated feedback could be detrimental to student
learning. Inspired by recent work that has utilised very powerful LLMs, such as
GPT-4, to evaluate the outputs produced by less powerful models, we conduct an
automated analysis of the quality of the feedback produced by several open
source models using a dataset from an introductory programming course. First,
we investigate the viability of employing GPT-4 as an automated evaluator by
comparing its evaluations with those of a human expert. We observe that GPT-4
demonstrates a bias toward positively rating feedback while exhibiting moderate
agreement with human raters, showcasing its potential as a feedback evaluator.
Second, we explore the quality of feedback generated by several leading
open-source LLMs by using GPT-4 to evaluate the feedback. We find that some
models offer competitive performance with popular proprietary LLMs, such as
ChatGPT, indicating opportunities for their responsible use in educational
settings.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined