Chinese Gigaword Third Edition

Item Name: Chinese Gigaword Third Edition
Author(s): David Graff
LDC Catalog No.: LDC2007T38
ISBN: 1-58563-455-7
ISLRN: 222-703-436-942-5
Release Date: August 17, 2007
Member Year(s): 2007
DCMI Type(s): Text
Data Source(s): newswire
Project(s): GALE
Application(s): language modeling, information retrieval, natural language processing
Language(s): Mandarin Chinese
Language ID(s): cmn
License(s): LDC User Agreement for Non-Members
Online Documentation: LDC2007T38 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Graff, David. Chinese Gigaword Third Edition LDC2007T38. Web Download. Philadelphia: Linguistic Data Consortium, 2007.

Introduction

Chinese Gigaword Third Edition is a comprehensive archive of newswire text data that has been acquired over several years by the LDC. This edition includes all of the contents in Chinese Gigaword Second Edition (LDC2005T14) as well as new data collected after the publication of that edition. Also, an archive of articles from a new newswire source (Agence France Presse) has been added in the third edition.

The four distinct international sources of Chinese newswire included in this edition are the following:

  • Agence France Presse (afp_cmn)
  • Central News Agency, Taiwan (cna_cmn)
  • Xinhua News Agency (xin_cmn)
  • Zaobao Newspaper (zbn_cmn)

The seven-letter codes in the parentheses above are used for the directory names and data files for each source, and are also used (in ALL_CAPS) as part of the unique DOC "id" string assigned to each news article.

Data

The original data archives received by the LDC from Agence France Presse, Xinhua News Agency and Zaobao were encoded in GB-2312, whereas those from Central News Agency (CNA) were encoded in Big-5. To avoid the problems and confusion that could result from differences in character-set specifications, all text files in this corpus have been converted to UTF-8 character encoding.

New in the Third Edition

  • Over six years worth of articles (October 2000 through December 2006) from Agence France Presse are being released for the first time.
  • Two years worth of new articles (January 2005 through December 2006) have been added to the Xinhua data set.
  • Nearly two years worth of content was added to the CNA data set. There was a gap in the LDC's collection from this source during 2006: no CNA Chinese content was collected between July 27 and December 17 2006, inclusive, so there are no data files for August through November of that year, and the December data file is about half its expected size.
  • A small set of older stories (October through December 1998) have been added from Zaobao; these were previously published by LDC as part of TDT3 Multilanguage Text Version 2.0 (LDC2001T58) and are being included in Gigaword for the first time.

Samples

Please examine this sample(JPEG) for an example of the data in this corpus.

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Available Media

View Fees





Login for the applicable fee