Protocol
Specification for Interaction Between
Communicator
Data Collection System and
Remote
Communicator System
This document describes how the Communicator Data Collection System (CDCS) interacts with the caller and a remote Communicator system. It also gives the specifications that a remote Communicator system must conform to for successful interactions with the CDCS.
CDCS Functions:
The Communicator Data Collection System (CDCS) will automatically route incoming calls from subjects to specific remote Communicator systems and record the caller and system speech on two separate channels. Both channels will be stored in 8-bit μlaw at 8 KHertz sampling rate. The caller/system orderings will be pre-assigned. As such, systems must be available to handle calls when they come in. Note that this also means that any system which cannot handle all the callers who need to talk to it simultaneously will be penalized by having “no calls” for some subjects. The CDCS will provide an optional handshaking protocol for sites who wish to ensure proper call identification and to enable later automatic logging/CDCS recording synchronization.
Remote
Communicator System Registration:
Before being permitted to participate in the Communicator data collection experiments, sites must transmit ALL of the following information to NIST via email to audrey.le@nist.gov:
Site Name: (ATT, BBN, CMU, Colorado, IBM, MIT, MITRE, NIST, SRI, etc.)
System Name: (identifier for your system)
Contact Name: (name of primary contact at your site)
Contact Email: (email for primary contact)
Administrator: (name of on-duty system administrator during data collection)
Administrator Email: (administrator email)
Administrator Phone: (administrator phone number)
Trouble Email: (email address for error messages)
System Phone1: (phone number of first copy of system)
System Phone2: (phone number of second copy of system)
System Phone3: (phone number of third copy of system)
Handshaking Protocol: (Yes/No)
Web URL: (URL for subject instructions/enrollment -- only fill if you want NIST to publish this on subject instruction page)
Protocol:
The following scenario describes a typical data collection session.
Prior to each call to the CDCS, the caller must log into a common web site (the URL for this web site will be given out later), hosted by NIST, to get the instructions for the next scenario. Per prior agreement with the sites, some web pages will contain a link to the site’s instruction/enrollment pages. After having read the instructions for the scenario and follow the site’s link (if any), the caller will then call the CDCS.
When the CDCS detects an incoming call, it answers the call and performs some internal tests. If the tests fail, the CDCS will ask the caller to call back and terminate the call. If the tests pass, the CDCS prompts the caller to enter his/her PIN followed by a pound sign (#). The PIN is a unique four-digit number that NIST pre-assigns to each caller. If the entered PIN is invalid, the CDCS prompts the caller to re-enter his/her PIN. If the entered PIN is invalid after the third attempt, the CDCS will ask the caller to recheck his/her PIN before calling back and terminate the call. If the PIN is valid, the CDCS calls one of the phone numbers for a remote Communicator system as specified in the site registration form.
If all the CDS gets a busy signal or no answer from all of the registered phone numbers for a remote Communicator system, the CDCS will ask the caller to try again at a later time and terminate the connection between the CDCS and the caller. Note that this counts as one of the three strikes allowed for this remote Communicator site for this user and session (see Error Handling).
Optional
CDCS/Remote Communicator System Handshaking Protocol:
A handshaking protocol is available for sites who wish to ensure proper subject-scenario identification and log synchronization. Sites must indicate whether they intend to use this protocol in the system registration form.
The following
actions will occur for sites participating in the protocol:
Once the CDCS detects that a remote Communicator system has answered a call, the CDCS constructs a call ID followed by a pound sign and sends it to the remote Communicator system as a series of DTMF signals.
The call ID consists of four fields: 1) a 4-digit caller's PIN, 2) a 2-digit session number, 3) a 2-digit scenario number, and 4) a 2-digit site ID. An example is given below:
1234020301# indicates that caller with PIN 1234 is making his/her second call performing scenario number 3 with system 1.
After sending the remote Communicator system the call ID, the CDCS waits for 5 seconds for the remote Communicator system to echo back the call ID. The remote Communicator system must echo the call ID followed by the pound sign as a series of DTMF signals. If the CDCS does not receive the call ID from the remote Communicator system within 5 seconds from the time that the call ID was sent, the CDCS will resend the remote system the call ID. If the CDCS does not receive the call ID back on the second try, the CDCS will the caller to try again at a later time and terminate the connections between the CDCS and the remote Communicator system and between the CDCS and the caller. Note that this counts as one strike against a site for this user (see Error Handling).
Once the CDCS receives the call ID from the remote Communicator system, the CDCS compares the call ID received with the call ID sent to ensure proper logging. If they do not match, the CDCS will terminate the connection between the CDCS and the remote Communicator system and redial for a second try. If the CDCS receives an incorrect call ID, the CDCS will ask the caller to try again at a later time and terminate the connections between the CDCS and the remote Communicator system and between the CDCS and the caller. Note that this counts as one strike against a site for this user (see Error Handling).
Once
the call ID is verified, the CDCS starts recording and connects the caller to
the remote Communicator system. The
CDCS then sends the remote Communicator system the DTMF pound key (#) to signal
that the CDCS is ready and recording.
When the remote Communicator receives the DTMF pound key, it sends the
DTMF star key (*) to indicate that it is beginning its interaction with the
caller.
The
CDCS senses for and records the DTMF star key as part of the audio. The DTMF star key is used for later
synchronization of the audio collected by the CDCS with the time stamps in the
log files created by the remote Communicator systems.
Note
that no caller/remote Communicator system audio will be recorded prior to
handshaking.
No Handshaking Protocol
Option:
Sites
must indicate in the system registration form if they elect to not participate
in the handshaking protocol.
The following actions will
occur for sites who do not participate in the handshaking protocol:
The
CDCS will simply act as a switchboard and connect users with remote
communicator systems and record both sides of the dialogue. Note that no
automatic logging synchronization will be possible with calls recorded by the
CDCS using this mode.
Since
no Call ID is sent to the remote systems, these systems must make their own
arrangements to identify the caller’s PIN and properly name their log files so
that they can be combined later with the CDCS recordings. We suggest that the sites record the caller
PIN in their own Web-based user profiling procedure and use the CDCS PIN when
the user calls in to their system.
Call Completion:
When
the CDCS detects a hang-up signal, it will stop the recording and end the
session. Once the session is underway,
the call can be terminated by a hang-up from either the user or the remote
Communicator system. The hang-ups will
be logged in a CDCS file. If a hang-up
occurs within 20 seconds of the start of recording, the CDCS will consider the
call a failure and automatically reschedule the call. A remote site participating in the handshaking protocol will know
a call has been rescheduled if it receives the same call ID twice. Non-participating sites must make this
determination from direct subject/system interactions. For rescheduled calls, the remote
Communicator systems are to overwrite (replace) any audio or log files from
earlier calls with the same ID.
(However, it is suggested, for safety, that remote Communicator systems
rename and preserve older files.)
Post Call Questionnaire:
After
the caller has completed each call, he/she must go to the NIST web page to fill
out a questionnaire about the call he/she just completed and retrieve the
instructions for the next call. The
NIST web site/database will automatically serve the caller the proper
questionnaire/instructions based on its call logging.
Error Handling and “Down”
Communicator Systems:
Since each subject has a limited time to make his/her calls and we do not want to lose subjects due to frustration, each Communicator system is allowed to have up to three "strikes" before a particular subject is directed to move to the next session-scenario. Items counted as strikes are given below:
1) all registered phone numbers for a system are busy or do not answer
2) the system echoes an incorrect call ID after two redials
3) the system echoes no call ID within the given amount of time after two tries
Note that items 2 and 3 only apply to sites participating in the handshaking protocol.
Note that systems which are chronically unresponsive during the data collection period will be under-represented in the data collection do to “no calls” and possibly receive a poor evaluation.
For each strike, the CDCS will send an email to the
registered Trouble Email address of the remote system to indicate that an error
has occurred. The CDCS will inform the
caller to wait at least 10 minutes before calling back. It is advised that site administrators
closely monitor the state of their systems during the evaluation data
collection period so that their systems are not skipped over because of such
problems.
Log File Transmission:
During the data collection period, the remote sites are responsible for sending NIST their log files. The remote sites will use comm2000@nist.gov to email NIST their log files. The log files should be sent as the body of the email message in plain ASCII text. For sites that participate in the handshaking protocol, the subject of the email should be the identifier string which includes the 10-digit call ID (e.g., 1234020301), the 8-digit date, and the label “auto” for auto-generated log files and “annotate” for human-generated annotation files. For sites that do not participate in the handshaking protocol, the subject of the email should be an identifier string which includes the 4-digit caller's PIN, the string “****” (to replace the scenario and session number), the 2-digit site's ID as given in the table below, 8-digit date, and the label “auto” for auto-generated log files and “annotate” for human-generated annotation files The information given in the subject of the email will be used to name the session log files. For sites that do not participate in the handshaking protocol, NIST will infer the session and scenario number since in this experiment each subject only calls each site once. If possible, sites should use this naming convention in all their locally created files. NIST will use the call IDs to name the log files as follows:
<PIN>_<SESSNUM>_<SCENNUM>_<SYSID>_<DATE>_<TYPE>.<EXT>,
where
-
PIN
is the unique 4-digit ID number assigned to each caller
-
SESSIONNUM
is the 2-digit session number for this caller
-
SCENARIONUM
is the 2-digit ID of the scenario
-
SYSID
is the 2-digit ID of the remote Communicator system
-
DATE
is the 8-digit date of the recording
-
TYPE
is either “auto” for auto-generated log files or “annotate” for human-generated
annotation files
-
EXT
is the 3-character extension used for log files. In this case, “xml” is used
Example :
1234_02_03_01_20000619_auto.xml – indicates the remote system
log of dialogue for caller 1234 making 2nd call and working on scenario number
3 with system 1 on June 19, 2000.
1234_02_03_01_20000619_annoate.xml – indicates the annotation file for dialogue for caller 1234 making 2nd call and working on scenario number 3 with system 1 on June 19, 2000.
Currently, the remote Communicator systems are assigned the following system IDs:
|
Communicator System Name |
Communicator System ID |
|
ATT |
01 |
|
BBN |
02 |
|
CMU |
03 |
|
COLORADO |
04 |
|
IBM |
05 |
|
MIT |
06 |
|
MITRE |
07 |
|
LU |
08 |
|
SRI |
09 |
Audio Filename Convention:
Although
remote sites are not required to submit their audio recordings, it is suggested
that they follow the naming convention given below.
<PIN>_<SESSNUM>_<SCENNUM>_<SYSID>_<DATE>_<CHAN>.<EXT>,
where
-
PIN
is the unique 4-digit ID number assigned to each caller
-
SESSNUM
is the 2-digit session number for this caller
-
SCENNUM
is the 2-digit ID of the scenario
-
SYSID
is the 2-digit ID of the remote Communicator system
-
DATE
is the 8-digit date of the recording
-
CHAN
is either “cal” for recording of caller speech or “sys” recording of system
speech
-
EXT
is the 3-character filename extension used for audio files (e.g., “sph” for
SPHERE-formatted audio files)
Example :
1234_02_03_01_20000619_cal.sph – indicates this is the caller's audio in SPHERE format for caller 1234 making 2nd call and working on scenario number 3 with system 1 on June 19, 2000.
1234_02_03_01_20000619_ sys.sph – indicates this is the system's audio in SPHERE format for caller 1234 making 2nd call and working on scenario number 3 with system 1 on June 19, 2000.
Schedule:
|
|
From |
To |
Description |
|
Alpha
Test |
April
24 |
April
28 |
2
or 3 sites; 2 or 3 subjects with toy scenarios |
Beta Test |
May
24 |
May
26 |
All
sites; 9 NIST staff with dev-test scenarios |
|
Evaluation* |
June
21 |
June
23 |
Cluster
1 with first 24 subjects |
|
Evaluation* |
June
28 |
June
30 |
Cluster
2 with second 24 subjects |
|
Evaluation* |
July
5 |
July
7 |
Cluster
3 with third 24 subjects |
Deliverables
Dates:
Item
|
Due
Date |
Auto-generated log files |
July
10 |
|
Human-generated
annotation files |
July
14 |
Scenario
Definition:
NIST will come up with a set of suitable scenarios. The scenarios may include international airports but will not include rental car and hotel reservation as these options are not evaluated for this round of evaluation.