An automatic behavior recognition system classifies animal behaviors using movements and their temporal context.

•ABRS reliably and quickly recognizes fly behaviors such as grooming from video, even in sub-optimal recording conditions.•The algorithm does not require image alignments or labeling of animal's body parts.•Useful orientation/translation invariant spatio-temporal features are extracted from video.•ABRS is freely available on GitHub for usage, modification and future development.•The method is flexible in implementation and can be extended to other behaviors or animal models.