Pay attention to what you read: Non-recurrent handwritten text-Line recognition

•Novel adaptation of transformers for handwriting recognition tasks, bypassing recurrent neural nets.•Competitive results achieved in low resource scenario with synthetically pretrained model.•Extensive ablation and comparative studies conducted to understand and modify transformer properly for HTR.•Implicit language modelling ability proved.•The state-of-the-art performance achieved on public IAM dataset.