Japanese Web N-gram Version 1

Introduction

Japanese Web N-gram Version 1, Linguistic Data Consortium (LDC) catalog number LDC2009T08 and isbn 1-58563-510-3, was created by Google Inc. It consists of Japanese "word" n-grams and their observed frequency counts generated from over 255 billion tokens of text. The length of the n-grams ranges from unigrams to seven-grams.

The n-grams were extracted from publicly accessible web pages that were crawled by Google in July 2007. This data set contains only n-grams that appear at least 20 times in the processed sentences. Less frequent n-grams were simply discarded. Those web pages requiring user authentication, pages containing "noarchive" or "noindex" meta tags, and pages under other special restrictions were excluded from the final release. While the aim was to process only Japanese pages, the corpus may contain some pages in other languages due to language detection errors. This dataset will be useful for research in areas such as statistical machine translation, language modeling and speech recognition, among others.

Data

Before the n-grams were collected, the web pages were converted into UTF-8 encoding, normalized into Unicode Normalization Form KC (see below), and split into sentences. Ill-formed sentences were filtered out, and the remaining sentences were segmented into "words".

All strings were normalized into Unicode Normalization Form KC (NFKC), which is described in http://www.unicode.org/unicode/reports/tr15/. Japanese strings were normalized according to the following rules:

• Full-width letters/digits were converted to ASCII letters/digits
• Half-width katakana were converted to full-width katakana
• Glyphs for Roman digits were converted to ASCII characters
• Certain Japanese-specific symbols were converted

The vocabulary was restricted to "words" that appeared at least 50 times in the processed sentences.

Statistical information about the corpus is set forth in the following table:

Samples

Japanese Bigram Japanese Trigram