Detecting ChatGPT-Generated Essays in a Large-Scale Writing Assessment: Is There a Bias Against Non-Native English Speakers?

A recent study (Liang et al., 2023) showed that publicly available detectors of AI-generated texts are more likely to misclassify essays written by non-native English speakers than those written by native English speakers, leading to significant fairness concerns about using these detectors. Using carefully sampled large-scale data from the Graduate Record Examinations (GRE) writing assessment, we developed multiple detectors of ChatGPT-generated essays based on linguistic features from the ETS e-rater engine and text perplexity features, and investigated their performance and potential bias. Results show that our detectors not only achieve near-perfect detection accuracy, but also show no evidence of bias that disadvantages non-native English speakers.