\documentstyle[urcsreport,fullname,11pt,twoside,psfig]{article}
\raggedbottom

\begin{document}

\title{The {T}{\sc rains} 93 Dialogues}

\author{Peter A. Heeman and James Allen}
\funding{Funding gratefully received from the Natural Sciences and 
Engineering Research Council of Canada, from NSF under Grant
IRI-90-13160, and from ONR/DARPA under Grant N00014-92-J-1512.}
\techrepname{TRAINS Technical Note}
\techrepnum{94-2}

\maketitle

\begin{abstract}

\noindent
This report describes a corpus of task-oriented dialogues set in the
{\sc Trains} domain.  A user collaborates with a planning assistant to
accomplish some task involving manufacturing and shipping goods in a
railroad freight system.  We include a description of the task,
collection situation, and transcriptions conventions.  The audio
files, along with time-aligned word and phoneme transcriptions are
available on CD-ROM from the Linguistic Data Consortium.  Altogether,
there are 98 dialogs included, collected using 20 different tasks and
34 different speakers.  This amounts to six and a half hours of
speech, about 5900 speaker turns, and 55000 transcribed words.
\end{abstract}

\pagenumbering{roman}
\tableofcontents
\cleardoublepage
\pagenumbering{arabic}

%=====================
\section{Introduction}
%=====================

One of the goals that we are pursuing at the University of Rochester
is the development of a conversationally proficient planning
assistant, which helps a user construct a plan to achieve some task
involving the manufacturing and shipment of goods in a railroad
freight system.  To do this, we need to know what kinds of phenomena
occur in such dialogues, and how to deal with them.  To provide
empirical data, we have collected a corpus of dialogues in this
domain with a person playing the role of the system.  The collection
procedure was designed to make the setting as close as to
human-computer interaction as possible, but was not a ``wizard''
scenario, where one person pretends to be a computer.  Thus these
dialogues provide a snapshop into an ideal human-computer interface
that would be able to engage in fluent conversations.

In this paper, we present our current dialogue collection endeavor.
Unlike our previous collections, we have used speech analysis tools in
order to produce time-aligned word transcriptions.  We supply both the
transcriptions and the audio files on CD-ROM available from the
Linguistic Data Consortium.  This allows this corpus to be used for speech
analysis purposes, such as speech recognition, and prosodic analysis.

This collection also expands on the number of different tasks, and the
number of different speaker pairs.  We have 20 different problem
scenarios, and 25 different pairs of conversants.  As well, for each
pair of conversants, we have collected up to seven dialogues, each
involving a different task.  This should make this corpus suitable for
studying problem solving strategies, as well as how agents collaborate
in solving a task.

Since our corpus contains dialogues in which the conversants work
together in solving the task, it provides natural examples of dialogue
usage that computer systems will need to handle in order to carry on a
dialogue with a user.  For instance, our corpus contains instances of
overlapping speech, back-channel responses, and turn-taking: phenomena
that do not occur in collections of single speaker utterances, such as
ATIS \cite{madcow:92:snlp}.  Also, even for phenomena that do occur in
single speaker utterances, such as speech repairs, our corpus allows
the interactions with other dialogue phenomena to be examined.

In comparison to the HCRC Map Task corpus \cite{anderson-etal:91:ls},
ours is also unique.  The map task involves one agent trying to
explain his route to another agent.  Our domain involves two agents
working together to construct a plan that solves some stated goal.
So, the agents must do high-level domain planning in addition to
communicative planning.  Hence, our corpus allows researchers to
examine language usage during collorative domain-planning---an area
where human-computer dialogues will be very useful.

%=======================================
\section{Overview of the Trains Project}
%=======================================

The {\sc Trains} project is a long-term research project to develop an
intelligent planning assistant that is conversationally proficient in
natural language \cite{allen-etal:95:jetai}. In particular, the {\sc
Trains} agent helps a person (the manager) construct and monitor plans
about a railroad freight system.  The manager is responsible for
assigning cargo to trains and scheduling shipments, scheduling various
simple manufacturing tasks, and for revising the original plan when
unexpected situations arise during plan execution.  The system aids
the manager in all aspects of this task by interacting in natural
language and (eventually) through a graphical interface. In
particular, the system typically will perform the following tasks:

\begin{itemize}
\item	evaluating suggested courses of action, such as calculating
       expected completion times, detecting conflicts that might
       interfere with the actions, and so on; 
\item	filling in details of the proposed plan that do not require
        the manager's attention; 
\item	suggesting ways to solve particular subproblems as they arise;
\item	presenting and describing the current state of the world and
       how the proposed plan  may affect it, including answering
       questions from the manager; 
\item	dispatching the plan to the different (simulated) agents in
the world;
%
\item interpreting reports back from the agents in the world in
order to monitor the progress of the plan and to anticipate problems
before they arise;
%
\item coordinating the correction and modification of plans with the manager.
\end{itemize}

While we aim to produce a functional system, the system itself is not
really the goal of the effort. Rather, the domain is a tool for
forcing our research to address the problems that arise in building a
complete dialogue system. For instance, the dialogue module must be
able to handle a wide range of everyday discourse phenomena rather
than handling a few selected problems of theoretical interest. While
this approach focuses our research directions, the solutions that we
seek are general solutions to the phenomena rather than specific
solutions that happen to work in the {\sc Trains} domain. The {\sc
Trains} project currently has several main foci of research:

\begin{itemize}
\item parsing and semantically interpreting utterances as they arise
in spoken language, including sentence fragments, repairs and
corrections, and long sequences of utterances that incrementally
provide information that eventually combines to form a complete
interpretation;

\item accounting for the discourse behavior present in natural
dialogue, {\em including} local coherence issues of reference and
scope ambiguity, and global discourse issues such as topic flow and
structure;

\item representing the reasoning and control of the discourse agent,
including reasoning about plans in the {Trains} domain as well as the
reasoning processes that drive the system's behavior in the dialogue
itself; and

\item providing the knowledge representation and reasoning tools
that are required to support planning and scheduling in realistic size
domains, including reasoning about time, events, actions and plans.
\end{itemize}


%================================================
\section{Role of Dialogues in the Trains Project}
%================================================

There are a wide range of problems that we intend to use this data to
help solve. These dialogues have been extensively used in our work on
plan-based dialogue models and speech acts (e.g., Traum and Hinkelman,
1992\nocite{traum-hinkelman:92:ci}), on discourse models for handling
co-reference and scoping (e.g., Poesio, 1992\nocite{poesio:92:kr}), on
detecting and correcting speech repairs (e.g., Heeman and Allen,
1994\nocite{heeman-allen:94:acl}), and on other parts of the {\sc
Trains} project.  Here are a few examples of how we have used, or are
planning to use the data.

\begin{itemize}
\item parsing spontaneous dialogue. It is well known that spoken
language is quite different in structure than written text. In
addition, spontaneous dialogue seems very different than spoken
language in a setting where speakers have time to plan and prepare
their utterances in advance. This database will allow one to study the
structure of interactive dialogue.

\item speech repairs.  In-line speech repairs are very common 
in interactive dialogue. We find about a quarter of the turns contain
repairs: repetitions, corrections, and cancelations.

\item prosody in spontaneous dialogue. While the study of intonation and
prosody is an area of growing interest, there is little work done so
far on spontaneous dialogue. More frequently, studies involve read
speech, or utterances that are planned and prepared in advance of the
utterance.  The database will provide a rich source of data on
repairs; 

\item turn-taking strategies. With the precise timing information
available, more detailed studies of turn-taking will be possible than
previously could be done using only analyses based on timing by hand.
Effects of intonation, use of certain phrases, and so on, can be
studied in relation to the turn-taking behavior observed; 	

\item the use of referring expressions in dialogue. With a large database of
dialogue, and some additional annotation on co-reference phenomena,
various theories of reference resolution, such as centering, will
be able to be empirically tested. In addition, the effect of
segmentation into intonational phrases on reference phenomena can be
studied; 	

\item global discourse structure. Several theories of the global structure
of discourse have been proposed, but none have been the subject of
systematic study because of a lack of enough data in a well-behaved
domain.

\item speech act analysis. It is believed that common ways of phrasing
utterances conventionally signal certain (indirect) speech acts. The
database will allows us to study such issues quantitatively.

\item the effect of general world knowledge on dialogue structure. A
number of theories of topic flow and change have been proposed using
plan-based models, but there has never been a systematic study of such
theories based on a large corpora of dialogues.

\end{itemize}

%===========================================================
\section{Previous dialogue collections in the Trains Domain}
%===========================================================

There have been two prior dialogue collection efforts as part of the
{\sc Trains} project.  \namecite {nakajima-allen:93:tr} collected
dialogues in the {\sc Trains} domain as part of a study that examined
prosodic clues to discourse structure.  For the dialogues,
a human played the role of the system, and both system and
user were out of sight of each other communicating through microphones
and headphones.  The only common domain knowledge that the conversants
shared was a map of the {\sc Trains} domain, which included the
cities, and locations of warehouses and factories.

The second dialogue collection, the {\sc Trains} 91 dialogues, was
undertaken by \namecite{gross-allen-traum:93:tr}.  This collection
employed the same collection paradigm as Nakajima and Allen, but with
a simpler domain.  Gross, Allen, and Traum found that this domain
provided a complex enough setting for extended dialogues.  For the
dialogue collection, eight different users were recorded, with the
same person playing the role of the system throughout.  The system had
information about the distance between cities and restrictions for
shipping freight.  Each pair did two problems: a warmup problem and a
test problem.  The same warmup and test problems were used by all
pairs of participants.


%==========================================
\section{The Trains 93 Dialogue Collection}
%==========================================

The present dialogue collection is very similar to the second dialogue
collection.  The {\sc Trains} map used in this collection, shown in
Figure~\ref {fig:newmap}, differs only slightly from the one used in
the 91 dialogues.  However, there are some differences between the two
corpora.  First, more attention was payed to minimizing outside noise,
and obtaining high-quality recordings.  Second, the dialogues were
transcribed using the Waves software \cite {entropic:93:waves},
resulting in time-aligned transcriptions.  Third, a larger number of
speaker pairs and problems were used, resulting in a much larger
corpus of dialogues.  Third, less attention this time was spent in
segmenting the dialogues into utterance units.  Rather, we used a more
pragmatically oriented view for segmenting the dialogues into
reasonable sized audio files, suitable for use with Waves.  This
convention is described in Section~\ref {sec:segmenting}.

The dialogues in this study were collected during three efforts.  The
first set were collected in the spring of 1992, encompassing dialogue
{\bf d92-1}, and was part of a trial run.  In the fall of 1992, after
small changes in the problem scenarios and in the user and system
instructions, the {\bf d92a} dialogues were collected.  Then during
the spring of 1993, after some further refinements in the
instructions, we collected the bulk of the dialogues, the {\bf d93}
dialogues.
%
\begin{figure}[htbp]
\psfig{figure=93map.eps,width=6.0in}
\caption{{\sc Trains} Domain for 93 Dialogues\label{fig:newmap}}
\end{figure}


%-----------------
\subsection{Setup}
%-----------------

The dialogues were collected in an office which had partitions
separating the two conversants.  Dialogues were collected with
Sennheiser HMD 414 close-talking microphones and headphones.  The
dialogues were recorded by a Panasonic SV-3900 Digital Audio Tape deck
at a sampling rate of 48 kHz.  In addition to the system and user, a
coordinator, who ran the experiments, was also present.  All three
could communicate with each other over microphones and headphones.
But only the system and user's speech was recorded, each on a separate
channel of a DAT tape.  Both the user and system knew that the
coordinator was overhearing, but would not participate in the
dialogues, even if a problem arose.

At the start of the session, the user was given a copy of the consent
form (Appendix~\ref {app:consent}) to read and sign, as well as a copy
of the user instructions (Appendix~\ref{app:user}) and user map
(Appendix~\ref {app:usermap}).  The user was not allowed to write
anything done.

The system was given a copy of the system instructions
(Appendix~\ref{app:system}) as well as copies of the system map
(Appendix~\ref{app:systemmap}).  The system was also given blank
paper and a pen, and was encouraged to use these to help them
remember the plan, and answer the user's queries.

Once the user and system had read over the instructions, the
coordinator had them practice on the warmup problem given in the
user's instructions.  For the {\bf d93} dialogues, the warmup problem
was not recorded, but was used to test the recording levels.

The participants then proceeded to do anywhere between two and seven
more problems, depending on how many they could complete in the thirty
minute session.  All problems were arranged into three piles on the
user's desk, with each pile corresponding to a different level of
difficulty (see Appendix~\ref{app:problems}).  When ready to begin a
dialogue, the coordinator would instruct the user to take a problem
from the top of a certain pile.  The first problem, after the warmup,
was always from the easiest pile.  For latter problems, the level of
difficulty was chosen on the basis of how well the participants
handled the previous problem and how much time remained.

After a problem was chosen, the user was given time to read the
problem over (less than a minute).  Once this was done, the user would
signal by saying ``ready''.  The coordinator would then set the DAT
deck into record mode and push a button that would cause a green light
to turn on at the user's and system's desk, which would signal the
system to begin the conversation.

The coordinator would record the conversation until it was clear that
the two participants had finished the dialogue.  At that point, the
user would hand the problem card to the coordinator, who would write
the problem number (written on the back of the card) down on the
recording log.


%------------------------
\subsection{The subjects}
%------------------------
\label{sec:subjects}

The role of the system was played primarily by graduate students from
the department of Computer Science and the department of Linguistics,
except for CK, an undergraduate who was transcribing the {\sc Trains}
91 dialogues using Waves.  About half of the systems were involved in
the {\sc Trains} project.  As for the users, we mostly used naive
subjects who did the experiment as course credit for a cognitive
science course.  All participants were native speakers of North
American English.  Most dialogue partners were not known to each
other, except in the case of dialogues {\bf d92-1}, {\bf d93-21}, and
{\bf d93-24}.  The list of speakers is given in
Appendix~\ref{app:speakers}.


%-------------------------------------------
\subsection{Comments on Collected Dialogues}
%-------------------------------------------

Participants sometimes made meta-comments about the task, or about the
instructions.  They also tended to make mistakes.
Appendix~\ref{app:errors} gives a list of the errors that were made,
and Appendix~\ref{app:dialoglist} lists which errors occurred in each
dialogue.  These errors were discovered by reading throught the
transcript of each dialogue.  One particular problem was that the
instructions were not very clear of what it meant for engines to be on
the same track.

Of the 98 dialogues collected, 3 of them ended with the participants
giving up, 23 of them ended with a plan that was not valid, and 14
more of them ended with a plan in which the execution time was
miscalculated, but where the plan could execute within the required
time constraints.


%==============================
\section{Segmenting a Dialogue}
%==============================
\label{sec:segmenting}

In order to work with a dialogue, we have segmented each dialogue into
conveniently sized single-speaker segments that capture the sequential
nature of the dialogue.  These guidelines are taken from our
technical note on dialogue transcription tools \cite
{heeman-allen:94:tn-tools}, in which we also give worked examples, and
describe tools that assist in this process.  For the first pass, we
made use of an automatic tool that segmented by silences.  After the
words were transcribed, we went back over the segmentation, and fixed
up any problems by hand.

The guidelines rely on the fact that the two conversants are
participating in a dialogue.  So, the conversants tend to follow a
turn-taking protocol and tend to break their speech into distinct
units.  The aim of the segmentation is the following.
%
\begin{description}
%
\item[A1:\hspace*{.25em}] Each utterance file should be short enough 
so it is easy to analyze (not more than 12 seconds long), and so it
does not include effects due to interactions from the other
participant.
%
\item[A2:\hspace*{.25em}] Each utterance file should be long enough 
so that local phenomena are not split across utterance-file
boundaries.
%
\end{description}

The first guideline should ensure that the sequence of single-speaker
utterance files captures the sequential nature of the dialogue, thus
allowing the flow and development of the dialogue to be preserved.  In
order words, the single-speaker utterance files should not contain or
overlap a contribution by the other speaker.  The second guideline
ensures that the segments allow local phenomena to be easily studied,
since they will be in a single file suitable for ToBI annotation.
There can be conflicts between these two aims.  If this happens, the
first aim, (A1), should take priority.

In using the above guidelines for segmenting dialogues, we have 
operationalize what constitutes as a suitable place to break up a
speaker's speech.  We propose the following list of conditions, of
which only one needs to hold.  These are ordered by the
appropriateness of the resulting break.

\begin{description}
\item[C1:\hspace*{.25em}] A suitable break occurs in a speaker's
speech whenever she stops and the other speaker starts (or continues),
without the first trying to continue.
%
\item[C2:\hspace*{.25em}] A suitable break occurs whenever all of the
following criteria hold.
%
\begin{enumerate}
\item There is an intonational phrase boundary.
\item There is a major syntactic category (i.e. NP or S) boundary.
\item There is a breath or pause.
\end{enumerate}
%
\item[C3:\hspace*{.25em}] A suitable break occurs whenever the first 
two criteria of (C2) hold, and there is a break between the words (but
shorter than a pause).
%
\item[C4:\hspace*{.25em}] A suitable break occurs whenever two of the 
three criteria of (C2) hold.
%
\end{description}

In segmenting a dialogue, utterance unit boundaries should be chosen
so that (A1) and (A2) are satisfied using the strongest breaks possible.


%===========================
\section{Word Transcription}
%===========================

After the dialogues were segmented into utterance files, we used the
Waves software to transcribe each utterance file, according to the
conventions given in Appendix~\ref{app:annotations}.  

For each dialogue, we first did a rough transcription, in which the
words were transcribed without regard to the time-alignment of each
word.  After the rough transcriptions were finished, we used a program
that scattered the word annotations evenly throughout the length of
the utterance, and then hand-aligned each word using Waves.  Halfway
through this effort, we switched to using an automatic word aligner
from Entropic Research Laboratories \cite {entropic:94:aligner}, which
we fed the rough transcriptions.  The result of the aligner was then
hand-checked and corrected if necessary.

To test the accuracy of our transcriptions, we selected a set of
utterances that spanned across the dialogue corpus.  (This set of
utterances consisted of 287 words.)  We then extensively hand-checked
the alignment, using spectrographs to increase the accuracy.  We will
refer to this alignment as the {\em reference} alignment.  All
silences, regardless if they were perceptable, were transcribed in the
reference alignment.  We then compared the reference alignment against
our original hand-aligned transcription and the output of the aligner,
ignoring discrepancies that fell within the silence markings of the
reference annotations.  Figure~\ref{fig:lag} shows the percentage of
words whose end-points were aligned within a given amount of time of
the reference end-point for the word.  From this, we can see that 95\%
of the words were hand-transcribed to be within 50 ms of the reference
end-point, and 83\% of the automatically aligned words.
%
\begin{figure}[htbp]
\centerline{\psfig{figure=lag.ps,width=5.0in}}
\caption{Difference in end points \label{fig:lag} with respect to the reference alignments}
\end{figure}

Measuring the differences in end-points is not very informative.  Some
words, especially function words, can have a short duration, even less
than 50 ms.  Hence, in Figure~\ref {fig:overlap}, we measured the
percentage of overlap.  For each word, we took the overlap in the
reference duration and the hand-transcribed duration of the word, and
compared this to the average of the reference duration and the
hand-aligned duration of the word.  Again, differences that fell in
the silence markings of the reference alignment were not counted.
From the figure, we see that for the hand-alignment, 95\% of the words
overlap with the reference alignment by at least 77\%; and for the
automatic alignment, 78\% of the words overlap with the reference
alignment by at least 77\%.
%
\begin{figure}[htbp]
\centerline{\psfig{figure=overlap.ps,width=5.0in}}
\caption{Overlap in word annotations with respect to reference alignment\label{fig:overlap}}
\end{figure}

%==============================
\section{Phoneme Transcription}
%==============================
\label{sec:phones}

We have also provided phoneme transcriptions.  These transcriptions
were all automatically obtained from the aligner, and are not
hand-corrected.  So, the accuracy of the phoneme alignment can be
gauged by the accuracy of the automatic word alignment (from which
they were obtained).  As mentioned above, Figure~\ref {fig:lag} and
Figure~\ref {fig:overlap} give the results of how well the automatic
word alignment did in comparison to the reference alignment.  Since,
83\% of the words have their end-points aligned within 50 ms of the
reference alignment, the phoneme alignment should be about as
accurate.


%=========================
\section{Using the Corpus}
%=========================
\label{sec:using}

The dialogue collection is available on CD-ROM from the Linguistics
Data Consortium.  Each dialogue is in a separate directory under the
directory {\bf dialogs}.  The name of the dialogue directory is in the
form {\bf $x$-$y$.$z$}, where $x$ indicates which of the three
collections, $y$ denotes the system and user pair, and $z$ denotes the
dialogue number within the system and user's set of dialoges.  The
three collections are named {\bf d92}, {\bf d92a}, and {\bf
d93}.\footnote{The system and user pair indicators for the {\bf d92}
and {\bf d92a} dialogs starts at 1, but for the {\bf d93} dialogs, we
made this number the same as the DAT tape number.}  Note that the {\bf
d92} dialogues consists of only one dialogue, which was named {\bf
d92-1}.

For each dialogue, we have included a text transcription, audio files,
and time-aligned annotations.  Table~\ref{tab:org} gives a complete
listing of the files associated with each dialogue.  All of these
files are speaker specific, except the dialogue transcription.  The
audio files have been formatted using the NIST Sphere header
structure, and are stored in compressed form.  The annotation files
are in the ESPS (Xlabel) format.  Each annotation file has a header
followed by a `\#' symbol, followed by the annotations.  Each
annotation is on a separate line.  The first column has the
time-alignment (in seconds), the second a colour to display the symbol
in (can be ignored), and the third has the actual symbol.

\newcommand{\desbegin}{\vspace{.3em}\noindent}
\newcommand{\des}[2]{\parbox[t]{10em}{{\bf #1}}\parbox[t]{29.6em}{#2} \vspace{.2em}\\}
%
\begin{table}
\hrule
\vspace{.5em}
\desbegin
\des{speaker$s$.fea}{Sphere (NIST) Audio file for the speaker $s$.
Speaker 0 is the system and speaker 1 is the user.  Recorded at 16
kHz (except for {\bf d92-1}, which was recorded at 8 kHz).}  
\des{speaker$s$.utts}{ESPS Annotation file that marks the utterance-level
file boundaries for speaker $s$.}
\des{$dialog$.txt}{Formatted listing of the transcription of the
dialogue, where $dialog$ is the name of the dialogue.  This is a
standard ASCII file, and can be easily viewed.}
\des{speaker$s$.words}{Annotation file that contains all of the
hand-aligned words in the dialogue for speaker $s$.}
\des{speaker$s$.display}{Annotation file that contains all of the
words in the dialogue for speaker $s$, combined with marks that
indicate the start and end of each utterance file (marked with {\bf
utt$i$<} and {\bf utt$i$>}.  These files are provided so that the user
can easily display the dialogue feature files along with the word and
utterance segmentation (see Section~\ref{sec:using}).}
\des{speaker$s$.awords}{Annotation files that contains the word
transcriptions from an automatic aligner.  These time alignments will
be in agreement with the phonetic transcriptions.}
\des{speaker$s$.phones}{Annotation files that contain the phonetic
transcriptions produced by the automatic aligner (see
Section~\ref{sec:phones}).}
\caption{Dialogue Files\label{tab:org}}
\vspace{.5em}
\hrule
\end{table}


The simplest place to start using the corpus is with the text
transcriptions of the dialogue (in ASCII).  These are stored in the
subdirectory {\bf transcripts} as {\em dialog}{\bf .tex}, where {\em
dialog} is the name of the dialogue (they are also in each dialogue
directory).

To use the audio files, you will need to install the NIST Sphere
toolset, which is included on the CD-ROM, in order to decompress the
audio files.  If you have access to Waves, the tool {\bf
displaydialog} can be run in the dialogue directories to display the
audio files (after it decompresses it) and the annotations ({\bf
speaker$s$.display}).  The Waves tools can display NIST Sphere files.
Of course, any tool capable of handling NIST Sphere audio files can be
used with the speaker audio files.

A third way of examining the dialogues is to segment them into
single-speaker utterance files that preserve the sequential nature of
the dialogue (see Section~\ref{sec:segmenting}).  A segmentation of
the speaker files is included in each dialogue directory, and can be
used for segmenting a dialogue into such a sequence of utterance
files.  In the {\bf tools} subdirectory on the CD-ROM is a toolkit for
managing such a segmentation.  This toolkit makes use of Waves, and is
described in a technical note \cite{heeman-allen:94:tn-tools} included
in the {\bf docs} subdirectory.  The program {\bf expanddialog}, which
makes use of the toolkit, can be used to create the utterance
segmentation from the CD-ROM dialogue.  This program is also in the
{\bf tools} subdirectory.

%========================
\section{Acknowledgments}
%========================

This dialogue collection was made possible by the help of many
contributors.  First, we thank Derek Gross and David Traum, authors of
the report on the Trains 91 dialogues.  We used their instructions,
{\sc Trains} map, and procedures as a starting point for this effort.
We wish to the participants of James' seminar on dialogue issues, and
in particular, Cheryl Beach, who revamped the problem scenarios and
instructions.  We thank Alice Kyburg who set up the 92 dialogue
collection.  We thank Brad Miller for overseeing the setup of the
speech lab.

We thank the many participants, especially the graduate
students who volunteered several hours of their time to play the role
of the system.

We also thank Bin Li and Tsuneaki Kato for their help in formulating the
heuristics for breaking up a dialogue.  We also thank Bin, Greg
Mitchell, Andrew Simchik, and Mia Stern for their help in annotating
the dialogues.
% 
\bibliographystyle{fullname} 
\input{bib} 

\appendix
\addcontentsline{toc}{section}{Appendices}

\input{consent}
\newpage

\input{user}
\newpage

\input{usermap}
\newpage

\input{system}
\newpage

\input{systemmap}
\newpage

\input{problems}
\newpage

\input{errors}
\newpage

\input{speakers}
\newpage

\input{dialoglist}
\newpage

\input{annotations}
\newpage

\end{document}
