An Empirical Analysis of Pronoun Translation Evaluation Metrices from English to Hindi Machine Translation Perspective

Abstract

Background: Recognizing the incompatibility of traditional or general-purpose automatic evaluation metrices in measuring the pronoun translation quality of a machine translation output (MTO), researchers evolved some pronoun specific metrices to measure the pronoun translation quality.

From this point of view, this research paper is aimed: (i) to have a systematic study about such pronoun specific evaluation metrices, (ii) to check their suitability for an English to Hindi machine translation (EHMT) system and (iii) to perform an empirical analysis of these metrices with respect to an EHMT system.

Method: Accounting to three major selection criteria viz. ease of implementation, language independent methodology and scope of performance enhancement, related literature was explored and surveyed. Based on this survey and exploration, out of eight evaluation scripts, two evaluation scripts namely as Precision, Recall and F1-measure (PRF) and Accuracy of Pronoun Translation (APT) metric were adopted for the purpose of this study and analysis. Since both evaluation metrices originally evolved for some other source and target language pair, both metrices were modified appropriately making it suitable for an EHMT system, then conducted an experiment to analyze the performance.

Result: Results show that APT shows very close correlation with human judgements. F-measure attained by PRF and APT was 53% and 68% respectively.

Conclusion: Study and analysis revealed that PRF and APT both metrices are enough suitable for an EHMT system and can be adapted with certain language specific modifications. Experiment results are quite encouraging and show that with accurate word alignment, appropriate configuration setting and case wise weight assignment, APT metric outperform over PRF