Morphologically Annotated Korean Text

September 17, 2003

Na-Rae Han

University of Pennsylvania

nrh@babel.ling.upenn.edu

Contributors:

• Seung-yun Yang (syyang@unagi.cis.upenn.edu)
• Mike Maxwell (maxwell@ldc.upenn.edu, LDC)

Contents

• Morphologically analyzed and part-of-speech annotated Korean corpus
• Encoded in ksc-5601 character set
• Size of corpus:
  • 1,574 sentences
  • 41,024 words (with tokenized symbols)
  • 77,173 morphemes
• Data format: one head word per line, word and its morphologically analyzed output are separated by a tab. Within the analyses, each morpheme is followed by "/" and its part-of-speech; morphemes are separated by "+". i.e., 유엔/NPR+은/PAU. ^EOS is a special symbol denoting the end of a sentence.
• The morphologically tagged output is compatible with Klex, the Finite-State Transducer Lexicon of Korean (LDC2004L01).
• The morphologically tagged output conforms to the Korean Treebank POS annotation standards, as found in ftp://ftp.cis.upenn.edu/pub/ircs/tr/01-09/.
• The original text of the corpus is a part of the Korean Newswire corpus (LDC2000T45). The newswire corpus is a collection of Korean Press Agency news articles from June 2, 1994, to March 20, 2000. The portion included in this release consists of a number of hand-picked articles.
• The corpus is a part of the Korean Treebank Phase 2. Between 2001 and 2002 the project was conducted under subcontract from Cogentex Inc, sponsor number Cogentex 5-33436. The original text was tokenized using a tokenization script, which was then automatically analyzed using Klex. Since there can be multiple possible morphological analyses, the output was fed through a statistical ranking system in order to select the best possible analysis for the word in the text environment. The part-of-speech tagged result was then manually corrected by Seung-yun Yang and Na-Rae Han, graduate students in the University of Pennsylvania Linguistics Department.
• The corpus uses a Part-of-Speech tag set which is based on the one employed by the Korean Treebank Project with slight modification. The POS tagging guidelines for the Korean Treebank can be found at ftp://ftp.cis.upenn.edu/pub/ircs/tr/01-09/.