GALE Phase 1 Chinese Blog Parallel Text

Item Name: GALE Phase 1 Chinese Blog Parallel Text
Author(s): Xiaoyi Ma, Stephanie Strassel
LDC Catalog No.: LDC2008T06
ISBN: 1-58563-469-7
ISLRN: 160-931-786-050-6
DOI: https://doi.org/10.35111/7s2f-am34
Release Date: April 22, 2008
Member Year(s): 2008
DCMI Type(s): Text
Data Source(s): weblogs
Project(s): GALE
Application(s): machine translation, language modeling
Language(s): English, Mandarin Chinese
Language ID(s): eng, cmn
License(s): LDC User Agreement for Non-Members
Online Documentation: LDC2008T06 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Ma, Xiaoyi, and Stephanie Strassel. GALE Phase 1 Chinese Blog Parallel Text LDC2008T06. Web Download. Philadelphia: Linguistic Data Consortium, 2008.
Related Works: View

Introduction

Blogs are posts to informal web-based journals of varying topical content. GALE Phase 1 Chinese Blog Parallel Text was prepared by the LDC and consists of 313K characters (277 files) of Chinese blog text and its translation selected from eight sources. This release was used as training data in Phase 1 of the DARPA-funded GALE program.

Source Data

Preparing the source data involved four stages of work: data scouting, data harvesting, formatting, and data selection.

Data scouting involved manually searching the web for suitable blog text. Data scouts were assigned particular topics and genres along with a production target in order to focus their web search. Formal annotation guidelines and a customized annotation tooklit helped data scouts to manage the search process and to track progress.

Data scouts logged their decisions about potential text of interest (sites, threads and posts) to a database. A nightly process queried the annotation database and harvested all designated URLs. Whenever possible, the entire site was downloaded, not just the individual thread or post located by the data scout.

Once the text was downloaded, its format was standardized (by running various scripts) so that the data could be more easily integrated into downstream annotation processes. Original-format versions of each document were also preserved. Typically, a new script was required for each new domain name that was identified. After scripts were run, an optional manual process corrected any remaining formatting problems.

The selected documents were then reviewed for content-suitability using a semi-automatic process. A statistical approach was used to rank a document's relevance to a set of already-selected documents labeled as "good." An annotator then reviewed the list of relevance-ranked documents and selected those which were suitable for a particular annotation task or for annotation in general. These newly-judged documents in turn provided additional input for the generation of new ranked lists.

Manual sentence units/segments (SU) annotation was also performed on a subset of files following LDC's Quick Rich Transcription specification. Three types of end of sentence SU were identified:

  • statement SU
  • question SU
  • incomplete SU

Translation

After files were selected, they were reformatted into a human-readable translation format, and the files were then assigned to professional translators for careful translation. Translators followed LDC's GALE Translation guidelines, which describe the makeup of the translation team, the source data format, the translation data format, best practices for translating certain linguistic features (such as names and speech disfluencies), and quality control procedures applied to completed translations.

Samples

For an example of the data contained in this corpus, please examine these screen captures of source (Chinese) and translation (English) files.

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or policy of the Government, and no official endorsement should be inferred.

Available Media

View Fees





Login for the applicable fee