An Experimental Investigation of Text-based CAPTCHA Attacks and Their Robustness

Text-based CAPTCHA has become one of the most popular methods for preventing bot attacks. With the rapid development of deep learning techniques, many new methods to break text-based CAPTCHAs have been developed in recent years. However, a holistic and uniform investigation and comparison of these attacks’ effects is lacking due to inconsistent choices of model structures, training datasets and evaluation metrics. In this paper, we perform an experimental investigation on the effects of existing attacks on text-based CAPTCHA schemes. We first summarize existing text-based CAPTCHAs using a newly proposed taxonomy based on their resistance mechanisms and systematically review corresponding attacks in terms of methods and pros/cons. Then, we introduce a unified attack framework that contains a number of different attack modules and transfer learning strategies. Applying this framework, we extensively evaluate the performance of known attacks on 20 CAPTCHA schemes in terms of accuracy and efficiency; then, we investigate the robustness of these widely used schemes and discover the effects of previously unexplored attacks. Finally, we discuss future CAPTCHA designs based on our experimental results and findings. Our work also contributes to the CAPTCHA community by offering an open-access dataset that contains 22 different CAPTCHA sample sets.