2000 Communicator Evaluation

Introduction

2000 Communicator Evaluation was produced by Linguistic Data Consortium (LDC) catalog number LDC2002S56 and ISBN 1-58563-258-9.

The original goals of the Communicator program were to support the creation of speech-enabled interfaces that scale gracefully across modalities, from speech-only to interfaces that include graphics, maps, pointing and gesture. The original vision of the Communicator systems included the ability of a user, during one 10-minute session, to plan a three-leg trip, with the three flights/legs on three different days, with rental car and hotel in each of the two "away" cities, plus dictating/sending a voice-mail message.

The actual research that led to the data collections in 2000 and 2001 explored ways to construct better spoken-dialogue sys tems, with which users interact via speech-alone to perform relatively complex tasks such as travel planning. During 2000 and 2001 two large data sets were collected, in which users used the Communicator systems built by the research groups to do travel planning. The researchers improved their systems intensively during the ten months between the two data collections. This distribution consists of all the data from the 2000 collection.

All the Communicator implementations used a common software architecture, called Galaxy-II, which was designed by a research team at MIT and adapted for Communicator in collaboration with a team at MITRE. The architecture supported detailed logging of the interaction between users a nd the systems.

Data

Nine sites participated in this project: ATT, BBN, Carnegie Mellon University, IBM, MIT, MITRE, NIST, SRI and University of Colorado at Boulder.

In 2000, each user called the nine different automated travel-planning systems to make simulated flight reservations. The order in which the users encountered the systems was counterbalanced, for statistical analysis purposes. All aspects of the reservations were simulated in 2000.

Each user was to make nine calls. The first seven calls had an assigned hypothetical travel task, which the user got via th e web. The last two calls asked the user to make simulated travel reservations for a trip that they might wish to take: they were asked to make travel plans for a vacation or pleasure trip on the eighth call and a business trip paid for by an employer on the ninth call.

All audio files are in SPHERE format, recorded in 8-bit u-law and pcm, at 8 KHZ. The files consist of the sites' recordings and the NIST recordings. The sites' recordings are utterance level (one channel) while the NIST recordings are a continuous recording of the whole call (both channels: user and system). The two-channel sphere files total ~62 hours of audio (3415 MB), representing ~317K words in transcription. The caller side of the calls have had sample_checksums added to the files headers submitted by the sites.

Updates

There are no updates available at this time.

NIST and DARPA have an Interagency Agreement by which funds are transferred to NIST. The funds to support NIST's DARPA Communicator Role were transferred under ARPA Order No. G270.