Chinese Web 5-gram Version 1


Item Name: Chinese Web 5-gram Version 1
Authors: Fang Liu, Meng Yang, Dekang Lin
LDC Catalog No.: LDC2010T06
ISBN: 1-58563-539-1
Release Date: Apr 19, 2010
Data Type: text
Data Source(s): web collection
Application(s): language modeling
Language(s): Mandarin Chinese
Language ID(s): cmn
Distribution: 7 DVD
Member fee: $0 for 2010 members
Non-member Fee: US $150.00
Reduced-License Fee: US $150.00
Extra-Copy Fee: US $150.00
Non-member License: yes
Member License: yes
Online documentation: yes
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Fang Liu, Meng Yang, Dekang Lin
2010
Chinese Web 5-gram Version 1
Linguistic Data Consortium, Philadelphia

Introduction

Chinese Web 5-gram Version 1, Linguistic Data Consortium (LDC) catalog number LDC2010T06 and isbn 1-58563-539-1, was created by researchers at Google Inc. It consists of Chinese word n-grams and their observed frequency counts generated from over 800 million tokens of text. The length of the n-grams ranges from unigrams (single words) to 5-grams. This data should be useful for statistical language modeling (e.g., segmentation, machine translation) as well as for other uses.

Included with this publication is a simple segmenter written in Perl using the same algorithm used to generate the data.

Data Collection

N-gram counts were generated from approximately 883 billion word tokens of text from publicly accessible web pages. This data set contains only n-grams that appeared at least 40 times in the processed sentences. Less frequent n-grams were discarded. While the aim was to identify and collect only Chinese language pages, some text from other languages is incidentally included in the final data.

Data collection took place in March 2008; no text that was created on or after April 1, 2008 was used to develop this corpus.

Preprocessing

The input character encoding of documents was automatically detected, and all text was converted to UTF-8. The data was tokenized by an automatic tool, and all continuous Chinese character sequences were processed by the segmenter.

The following types of tokens are considered valid:

  • A Chinese word containing only Chinese characters.
  • Numbers, e.g., 198, 2,200, 2.3, etc.
  • Single Latin tokens, such as Google, &ab, etc.

Extent of Data

  • File sizes: approx. 30 GB compressed (gzip'ed) text files
  • Number of tokens: 882,996,532,572
  • Number of sentences: 102,048,435,515
  • Number of unigrams: 1,616,150
  • Number of bigrams: 281,107,315
  • Number of trigrams: 1,024,642,142
  • Number of fourgrams: 1,348,990,533
  • Number of fivegrams: 1,256,043,325

Sample

Sample screen shot

Content Copyright

Portions 2008 Google Inc., 2010 Trustees of the University of Pennsylvania