TDT Pilot Study Corpus

Item Name: TDT Pilot Study Corpus
Author(s): James Allan, Yiming Yang, Jaime Carbonell, Jon Yamron, George R. Doddington, Charles Wayne
LDC Catalog No.: LDC98T25
ISBN: 1-58563-140-X
ISLRN: 770-765-444-577-6
Member Year(s): 1998
DCMI Type(s): Text
Data Source(s): transcribed speech, newswire, broadcast news
Project(s): TIDES, TDT, GALE, EARS
Application(s): topic detection and tracking
Language(s): English
Language ID(s): eng
License(s): TDT Pilot Study Agreement
Online Documentation: LDC98T25 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Allan, James, et al. TDT Pilot Study Corpus LDC98T25. Web Download. Philadelphia: Linguistic Data Consortium, 1998.
Related Works: View


The TDT Pilot Study corpus was created to support an initiative in "topic detection and tracking." This initiative is directed toward computer processing of language data, both text and speech. The objective is namely to explore techniques for detecting the appearance of new and unexpected topics and for tracking the reappearance and evaluation of them.


The TDT corpus comprises a set of stories that includes both newswire (text) and broadcast news (speech). Each story is represented as a stream of text, in which the text is either taken directly from the newswire (Reuters) or is a manual transcription of the broadcast news speech (CNN). The corpus spans the period from July 1, 1994 to June 30, 1995. It contains approximately 16,000 stories, with about half taken from Reuters newswire and half from CNN broadcast news transcripts.

An integral and key part of the corpus is the annotation of the corpus in terms of the events discussed in the stories. 25 events were defined that span a variety of event types and that cover a subset of the events discussed in the corpus stories. Annotation data for these events are included in the corpus and provide a basis for training TDT systems.


There are no updates at this time.

Available Media

View Fees

Login for the applicable fee