The AQUAINT Corpus of English News Text


Item Name: The AQUAINT Corpus of English News Text
Authors: David Graff
LDC Catalog No.: LDC2002T31
ISBN: 1-58563-240-6
Release Date: Sep 26, 2002
Data Type: text
Data Source(s): newswire
Language(s): English
Language ID(s): eng
Distribution: 1 DVD
Member fee: $0 for 2002 members
Non-member Fee: US $300.00
Reduced-License Fee: US $200.00
Extra-Copy Fee: US $200.00
Non-member License: yes
Online documentation: yes
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: David Graff
2002
The AQUAINT Corpus of English News Text
Linguistic Data Consortium, Philadelphia

Introduction

The AQUAINT Corpus, Linguistic Data Consortium (LDC) catalog number LDC2002T31 and ISBN 1-58563-240-6 consists of newswire text data in English, drawn from three sources: the Xinhua News Service (People's Republic of China), the New York Times News Service, and the Associated Press Worldstream News Service. It was prepared by the LDC for the AQUAINT Project, and will be used in official benchmark evaluations conducted by National Institute of Standards and Technology (NIST).

Data

The data files contain roughly 375 million words correlating to about 3GB of data. The text data are separated into directories by source (apw, nyt, xie); within each source, data files are subdivided by year, and within each year, there is one file per date of collection. Each file is named to reflect the source and date, and contains a stream of SGML-tagged text data presenting the series of news stories reported on the given date as a concatenation of DOC elements (i.e. blocks of text bounded by and tags).

All data files are published in compressed form, using the GNU "gzip" utility; as such, all files have a ".gz" extension, and will have null file name extension when uncompressed in the usual way (i.e. just the base file name, consisting of "YYYYMMDD_SRC").

While all the data files are covered by a single DTD, it is not the case that they all have a single pattern of markup. Rather, all files share a core markup structure, with minor variations in the peripheral regions of each DOC element, and the DTD has been written to accommodate the variations.

Updates

19980614_NYT.gz was left off in the conversion from CD to DVD. An update was issued on 09/13/2012. All copies ordered after this date will be complete. Contact ldc@ldc.upenn.edu for more information.

Content Copyright

Portions 1998-2000 New York Times, 1998-2000 The Associated Press, 1996-2000 Xinhua News Agency, 2002 Trustees of the University of Pennsylvania