2008/2010 NIST Metrics for Machine Translation (MetricsMaTr) GALE Evaluation Set

Item Name: 2008/2010 NIST Metrics for Machine Translation (MetricsMaTr) GALE Evaluation Set
Author(s): NIST Multimodal Information Group, Linguistic Data Consortium
LDC Catalog No.: LDC2011T05
ISBN: 1-58563-575-8
ISLRN: 111-667-828-386-1
Release Date: March 16, 2011
Member Year(s): 2011
DCMI Type(s): Text
Data Source(s): web collection, newswire, broadcast news, broadcast conversation
Project(s): GALE
Application(s): machine translation, machine learning
Language(s): Mandarin Chinese, Arabic, Chinese
Language ID(s): cmn, ara, zho
License(s): LDC User Agreement for Non-Members
Online Documentation: LDC2011T05 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: NIST Multimodal Information Group, and Linguistic Data Consortium. 2008/2010 NIST Metrics for Machine Translation (MetricsMaTr) GALE Evaluation Set LDC2011T05. Web Download. Philadelphia: Linguistic Data Consortium, 2011.

Introduction

2008/2010 NIST Metrics for Machine Translation (MetricsMaTr) GALE Evaluation Set, Linguistic Data Consortium (LDC) catalog number LDC2011T05 and isbn 1-58563-575-8, is a package containing source data, reference translations, machine translations and associated human judgments used in the NIST 2008 and 2010 MetricsMaTr evaluations. The package was compiled by researchers at NIST, making use of Arabic and Chinese broadcast, newswire and web data and reference translations collected and developed by LDC for Phase 2 and Phase 2.5 of the DARPA GALE program.

NIST MetricsMaTr is a series of research challenge events for machine translation (MT) metrology, promoting the development of innovative MT metrics that correlate highly with human assessments of MT quality. Participants submit their metrics to NIST (National Institute of Standards and Technology). NIST runs those metrics on certain held-back test data for which it has human assessments measuring quality and then calculates correlations between the automatic metric scores and the human assessments. Specifically, the goals of MetricsMATR are: to inform other MT technology evaluation campaigns and conferences with regard to improved metrology to establish an infrastructure that encourages the development of innovative metrics to build a diverse community that will bring new perspectives to MT metrology research and to provide a forum for MT metrology discussion and for establishing future directions of MT metrology.

The first MetricsMaTr challenge was held in 2008 the development data from the 2008 program is available from LDC, 2008 NIST Metrics for Machine Translation (MetricsMATR08) Development Data LDC2009T05. The MetricsMaTr10 evaluation plan is included in this release.

Data

This release contains 149 documents with corresponding reference translations (Arabic-to-English and Chinese-to-English), system translations and human assessments. The human assessments include the following: Adequacy7 (a 7-point scale for judging the meaning of a system translation with respect to the reference translation) Adequacy Yes/No (whether the given system segment meant essentially the same as the reference translation) Preference (the judges preference between two candidate translations when compared to a human reference translation) and HTER (Human Targeted Error Rate, human edits to a system translation to have the same meaning as a reference translation).

Updates

Additional information, updates, bug fixes may be available in the LDC catalog entry for this corpus at LDC2011T05.

Sponsorship

This work is supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-003. The content of this publication does not necessarily reflect the position or policy of the Government, and no official endorsement should be inferred.

Samples

Please view this sample.

Available Media

View Fees

Member
Non-Member
Reduced-License
Extra Copy
Login for the applicable fee