Meta-Evaluation Using Impact Machine Translation Evaluation Method At Document Level And Segment Level

Various methods have been proposed and evaluated for the automatic evaluation of machine translation. Their meta evaluations are usually performed only at the document level, which consists of many translated sentences. However, meta-evaluations for methods on the segment level are also extremely important because human judgment can evaluate both the document level and segment level. This paper proposes a calculation method of a document-level score for IMPACT, an automatic evaluation method for machine translation. In previous experiments, IMPACT exhibited the highest correlation with human judgment among several methods at the segment level. It is important for IMPACT to obtain high correlation also at the document level. Furthermore, we describe results of a meta-evaluation for IMPACT at both the document level and segment level through comparison with other methods. Meta-evolutional experiments show that IMPACT can obtain the highest value among several methods in F values based on document-level and segment-level correlations.