Tagged Chinese Gigaword Version 2.0

Introduction

Tagged Chinese Gigaword Version 2.0, created by scholars at Academia Sinica, Taipei, Taiwan, is a part-of-speech tagged version of LDC's Chinese Gigaword Second Edition (LDC2005T14). Like the original release, Version 2.0 contains all of the data in Chinese Gigaword Second Edition -- from Central News Agency, Xinhua News Agency and Lianhe Zaobao -- annotated with full part of speech tags. In addtion, this new release removes residual noises in the original and improves tagging accuracy by incorporating lexica of unknown words. The changes represented in Version 2.0 include the following:

• A single-width space is used consistently between two segmented words.
• The position of the newline character remains fixed, better reflecting the source files from Chinese Gigaword Second Edition (LDC2005T14).
• The original coding of partial Latin letters or Arabic numerals is preserved.
• 1,192 documents from Central News Agency (Taiwan) and 13 documents from Xinhua News Agency that were missing from the first publication are included.
• A set of heuristics for building out-of-vocabulary dictionaries to improve annotation quality of very large corpora is incorporated.

Documents in the corpus were assigned one of the following categories:

• story: This type of DOC represents a coherent report on a particular topic or event, consisting of paragraphs and full sentences.
• multi: This type of DOC contains a series of unrelated "blurbs," each of which briefly describes a particular topic or event; examples include "summaries of today's news," "news briefs in ..." (some general area like finance or sports), and so on.
• advis: These are DOCs which the news service addresses to news editors; they are not intended for publication to the "end users."
• other: These DOCs clearly do not fall into any of the above types; they include items such as lists of sports scores, stock prices, temperatures around the world, and so on.

Data

Basic statistics of data from each source are summarized below.

The POS tags and their corresponding explanations are listed below:

Since neither manual checking nor automatic checking against a gold standard is feasible for gigaword size corpora, the authors proposed quality assurance of automatic annotation of very large corpora based on heterogeneous CKIP and ICTCLAS tagging systems (Huang et al., 2008). By comparing to word lists generated from the ICTCLAS version of an automatic tagged Xinhua portion of Chinese Gigaword, a set of heuristics for building out-of-vocabulary dictionaries to improve quality were proposed. Randomly selected texts for evaluating effects of these out-of-vocabulary dictionaries were manually checked. Experimental results indicate that there were 30,562 correct words (about 97.3 %) of tested words. The quality control test result follows:

Samples

Please view this sample.