Chinese Gigaword Fourth Edition
| Item Name: | Chinese Gigaword Fourth Edition |
| Author(s): | Robert Parker, David Graff, Ke Chen, Junbo Kong, Kazuaki Maeda |
| LDC Catalog No.: | LDC2009T27 |
| ISBN: | 1-58563-527-8 |
| ISLRN: | 261-416-300-929-8 |
| DOI: | https://doi.org/10.35111/abt0-qy36 |
| Release Date: | September 15, 2009 |
| Member Year(s): | 2009 |
| DCMI Type(s): | Text |
| Data Source(s): | newswire |
| Project(s): | GALE |
| Application(s): | natural language processing, language modeling, information retrieval |
| Language(s): | Mandarin Chinese |
| Language ID(s): | cmn |
| License(s): |
LDC User Agreement for Non-Members |
| Online Documentation: | LDC2009T27 Documents |
| Licensing Instructions: | Subscription & Standard Members, and Non-Members |
| Citation: | Parker, Robert, et al. Chinese Gigaword Fourth Edition LDC2009T27. Web Download. Philadelphia: Linguistic Data Consortium, 2009. |
| Related Works: | View |
Introduction
Chinese Gigaword Fourth Edition, Linguistic Data Consortium (LDC) catalog number LDC2009T27 and isbn 1-58563-527-8, is a comprehensive archive of newswire text data that has been acquired over several years by the LDC. This edition includes all of the contents in Chinese Gigaword Third Edition (LDC2007T38) as well as newly collected data. In addition, four entirely new sources have been added in the fourth edition, Central News Service, Guangming Daily, Peoples Liberation Army Daily, and Peoples Daily.
The eight distinct international sources of Chinese newswire included in this edition are the following:
- Agence France Presse (afp_cmn)
- Central News Agency, Taiwan (cna_cmn)
- Central News Service (cns_cmn)
- Guangming Daily (gmw_cmn)
- Peoples Daily (pda_cmn)
- Peoples Liberation Army Daily (pla_cmn)
- Xinhua News Agency (xin_cmn)
- Zaobao Newspaper (zbn_cmn)
The seven-letter codes in the parentheses above are used for the directory names and data files for each source, and are also used (in ALL_CAPS) as part of the unique DOC id string assigned to each news article.
Data
The original data received by the LDC from AFP, Peoples Liberation Army Daily, Xinhua, and Zaobao were encoded in GB-2312, those from CNA were in Big-5, and those from GMW, CNS, and Peoples Daily were in a combination of GB-2312 and GB-18030. To avoid the problems and confusion that could result from differences in character-set specifications, all text files in this corpus have been converted to UTF-8 character encoding.
New in the Fourth Edition
- Two years worth of new articles (January 2007 through December 2008) have been added to the Xinhua, Agence France Presse, and CNA data sets.
- Four new data sources have been added - Guangming Daily, Central News Service, Peoples Daily and Peoples Liberation Army daily, covering a timespan from November 2006 through December 2008.
Samples
Please view this sample.
Sponsorship
This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.