Learning to infer human attention in daily activities

•Different encoder-decoder architectures significantly affect the performance.•The performance improves after considering the task encoding loss.•The pretrain of the neural network contributes to better performance.