English Gigaword Second Edition

Item Name: English Gigaword Second Edition
Author(s): David Graff, Junbo Kong, Ke Chen, Kazuaki Maeda
LDC Catalog No.: LDC2005T12
ISBN: 1-58563-350-X
ISLRN: 274-788-133-216-1
DOI: https://doi.org/10.35111/stcf-4x49
Release Date: July 15, 2005
Member Year(s): 2005
DCMI Type(s): Text
Data Source(s): newswire
Project(s): EARS, GALE, TIDES
Application(s): information retrieval, language modeling, natural language processing
Language(s): English
Language ID(s): eng
License(s): LDC User Agreement for Non-Members
Online Documentation: LDC2005T12 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Graff, David, et al. English Gigaword Second Edition LDC2005T12. Web Download. Philadelphia: Linguistic Data Consortium, 2005.
Related Works: View

Introduction

English Gigaword Second Edition was produced by the Linguistic Data Consortium (LDC) and contains approximately 2.3 billion words of English newswire text data that has been acquired over several years by LDC.

This edition includes all of the contents in the first edition, English Gigaword (LDC2003T05), as well as new data from July 2002 through December 2004 from all four sources in the first edition and a new source, the Central News Agency of Taiwan, English Service. This second addition also includes a three-letter language code in the source abbreviations, and minor formatting improvements (mostly line-wrapping).

Data

Here is a table showing the five distinct international sources of English newswire included in this release along with the breakdown of their contents in numbers of documents and K-words (thousands of words):

Source Abbreviation Documents K-words
Agence France-Presse, English Service (afp_eng) 1,202,139 337,792
Associated Press Worldstream, English Service (apw_eng) 1,975,456 736,518
Central News Agency of Taiwan, English Service (cna_eng) 57,999 15,039
The New York Times Newswire Service (nyt_eng) 1,446,256 1,026,533
The Xinhua News Agency, English Service (xin_eng) 1,017,150 201,346
Totals   5,699,000 2,317,228

All the data is organized into zipped files. All text data are presented in SGML form, using a very simple, minimal markup structure; all text consists of printable ASCII and whitespace. Documents are sorted into four different types:

  • Story: a report composed of paragraphs and full sentences; most common
  • Multi: unrelated "blurbs" of several news items
  • Advis: advisories directed at news editors and not intended for publication/general audience
  • Other: intended for publication but not paragraphs or sentences; these are things like lists of sports scores, stock prices, temperatures around the world, etc.

Samples

For an example of the data in this corpus, please view this sample (SGML).

Updates

None at this time.

Available Media

View Fees





Login for the applicable fee