TDT5 Multilingual Text

Item Name: TDT5 Multilingual Text
Authors: David Graff, Junbo Kong, Kazuaki Maeda, Stephanie Strassel
LDC Catalog No.: LDC2006T18
ISBN: 1-58563-417-4
Release Date: Dec 19, 2006
Data Type: text
Language(s): English, Mandarin Chinese, Modern Standard Arabic
Language ID(s): arb, cmn, eng
Distribution: 1 DVD
Member fee: $0 for 2006 members
Non-member Fee: US $1500.00
Reduced-License Fee: US $750.00
Extra-Copy Fee: US $200.00
Non-member License: yes
Online documentation: yes
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: David Graff, et al.
TDT5 Multilingual Text
Linguistic Data Consortium, Philadelphia


The TDT5 corpora were created by Linguistic Data Consortium with support from the DARPA TIDES (Translingual Information Detection, Extraction and Summarization) Program. This release contains the complete set of English, Arabic and Chinese newswire text used in the 2004 Topic Detection and Tracking technology evaluations. The topic relevance annotations corresponding to this publication can be found in LDC Publication LDC2006T19, TDT5 Topics and Annotations.

Topic Detection and Tracking (TDT) refers to automatic techniques for finding topically related material in streams of data such as newswire and broadcast news.

There were four TDT tasks defined for the 2004 evaluation: the tracking of known topics, the detection of unknown topics, the detection of initial stories on unknown topics, and the detection of pairs of stories on the same topic (links). Of these four tasks, the topic tracking task and the link detection task are considered to be "primary." Previous TDT evaluations also included a story segmentation task. This task applied only to broadcast news. Since TDT5 does not include broadcast news, there is no story segmentation task in the 2004 TDT Evaluation.

Complete documentation on the TDT evaluation program can be found on NIST's TDT website.

For further information about corpora and annotations to support the TDT Program visit LDC's TDT information pages.


The images below are samples of the text data contained in this corpus.

Content Copyright

Portions 2003 Agence France Presse, An-Nahar, The Associated Press, The Central News Agency, Los Angeles Times-Washington Post News Service, Inc., The New York Times, SPH AsiaOne Ltd, Ummah Press Service, Xinhua News Agency, 2003, 2004, 2006 Trustees of the University of Pennsylvania