Semantic Textual Similarity (STS) 2013 Machine Translation


Item Name: Semantic Textual Similarity (STS) 2013 Machine Translation
Authors: Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo
LDC Catalog No.: LDC2013T18
ISBN: 1-58563-656-8
Release Date: Sep 16, 2013
Data Type: text
Data Source(s): newswire, web collection
Project(s): GALE, NIST MT
Application(s): machine translation
Language(s): English
Language ID(s): eng
Distribution: Web Download
Member fee: $0 for 2013 members
Non-member Fee: US $0.00
Reduced-License Fee: N/A
Extra-Copy Fee: US $
Non-member License: yes
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Eneko Agirre, et al.
2013
Semantic Textual Similarity (STS) 2013 Machine Translation
Linguistic Data Consortium, Philadelphia

Introduction

Semantic Textual Similarity (STS) 2013 Machine Translation was developed as part of the STS 2013 Shared Task which was held in conjunction with *SEM 2013, the second joint conference on lexical and computational semantics organized by the ACL (Association of Computational Linguistics) interest groups SIGLEX and SIGSEM. It is comprised of one text file containing 750 English sentence pairs translated from the Arabic and Chinese newswire and web data sources.

The goal of the Semantic Textual Similarity (STS) task was to create a unified framework for the evaluation of semantic textual similarity modules and to characterize their impact on natural language processing (NLP) applications. STS measures the degree of semantic equivalence. The STS task was proposed as an attempt at creating a unified framework that allows for an extrinsic evaluation of multiple semantic components that otherwise have historically tended to be evaluated independently and without characterization of impact on NLP applications. More information is available at the STS 2013 Shared Task homepage.

Data

The source data is Arabic and Chinese newswire and web data collected by LDC that was translated and used in the DARPA GALE (Global Autonomous Language Exploitation) program and in several NIST Open Machine Translation evaluations. Of the 750 sentence pairs, 150 pairs are from the GALE Phase 5 collection and 600 pairs are from NIST 2008-2012 Open Machine Translation (OpenMT) Progress Test Sets (LDC2013T07).

The data was built to identify semantic textual similarity between two short text passages. The corpus is comprised of two tab delimited sentences per line. The first sentence is a translation and the second sentence is a post-edited translation. Post-editing is a process to improve machine translation with a minimum of manual labor. The gold standard similarity values and other STS datasets can be obtained from the STS homepage, linked above.

Samples

Please view this text sample.

Updates

None at this time.

Content Copyright

Portions 2007 Agence France Presse, Al-Ahram, Al Hayat, Al-Quds Al-Arabi, Asharq Al-Awsat, An Nahar, Assabah, China Military Online, Chinanews.com, Guangming Daily, Xinhua News Agency, 2007, 2013 Trustees of the University of Pennsylvania