[ATT]
One or two days before the subjects come in, call them to make sure
they are coming, and ask if they need directions to get here. Ask
them to arrive (if possible) about fifteen minutes before the hour.
The experiment will take around an hour and a half from the time
you meet them downstairs to the time you walk them out. They should
park in the visitor's parking lot. Make sure they have my phone
number, and also give them the number in the wizard room. The phone
numbers should all be on the whiteboard.
Tell ORACLE to talk to LUCID.
saruman% sock_ora foo
Start up EMACS. You will need to use a different window than the one you ran sock_ora foo in.
saruman% xemacs &
To start LUCID, type the following in the EMACS window:
M-x lucid <return>
Once LUCID is running, load ATIS and tell LUCID to talk to ORACLE.
> (load "go")
> (in-package :tina)
> (recordinit #)
Where # is the port number sock_ora foo is listening on.
A few other things:
When they have selected cities for each scenario, call the computer (3112), have them put on the headset (microphone on the left if they are left handed, on the right if they are right handed) and make sure the microphone is properly positioned. Go through the stuff in the red box on the whiteboard:
> (wizard-atis "<subject's initials>" "<wizard's name>" :book-flights? t)
At the "Hit return to start recording. " prompt, you may type
quit exits ATIS redo "would you mind repeating .. . ." last TTS repeats the last single sentence it said. This should be used to repeat any single sentence response.At the "Recording. . . Hit return to stop. " prompt, you may type
quit exits ATIS redo "would you mind repeating .. . ."After an "I'm ready to begin a scenario," always type "begin scenario n" (where n is the scenario number) no matter what the person actually says.
If there is clipping, go in and turn down the gain box by one notch (to +42).
If they finish the two short scenarios before twenty minutes have elapsed, and they don't go on to the third, have them select another short scenario or go on to the long one (if there is time).
Let them finish whatever scenario they're working on. If they are floundering, you can stick your head in the door and tell them to say "End scenario."
Ask if they want a drink of water and ask how they're doing with the system. If they were having problems, you are not to give them any examples of what the computer likes. You may repeat the stuff about speaking naturally, or ask them to think about what they would say if they were trying to get information from you. If they already did all three scenarios, have them choose another long one for the second session.
Before you go out to start the second session, make sure they have the headset on with the microphone properly positioned, and remind them to say "Begin scenario."
Before you begin the next scenario, get rid of any stopped play processes. In a UNIX window, type
saruman% psg play
saruman% kill -2 # # #
Where each # is a process id of a stopped play process.
> (quit-sender)
> (user::quit)
Resist the temptation to "help" the user by rephrasing their questions. They are probably less frustrated than you are.
If you cut off any part of their speech, make them say it again. Have no mercy.
Be extra careful about when and how many times you hit return. I'd rather you lose half a second and hit return at the right time than have to ask the subject redo, or hang the machine, or make the SURFboard complain.
If you feel comfortable starting recording a word before the end of TTS, fine. However, if you repeatedly record the end of the TTS, I would ask that you wait until TTS is done before starting recording.
Try not to start recording if playback has not finished. This creates a stopped processes and can really slow the machine down.
There are solutions to a variety of system problems on the whiteboard in the "In case of error:" box. Here are some additions:
> (restart-user "xyz" "wizard")
where xyz are the speaker's initials and wizard is your name.
saruman% ttsbgstop
until you get the response "tts background processes have been terminated" then type
saruman% ttsbg
saruman% ttsgo "testing 1 2 3"
and make sure both you and the subject hear it talk.
saruman% replay
saruman% ttsgo "Please speak just one sentence at a time, then wait for me to respond."
saruman% ttsgo "I keep losing the beginnings of your sentences. Please wait a bit longer
after I finish talking."
You are not to use ttsgo for anything else.
[BBN]
WIZARD CHECKLIST
A Day before Data Collection
Make sure your subject will have the SUBJECT INSTRUCTIONS a day before the
data collection session.
Tell subject:
When the subject arrives:
Tell subject:
- "Solving a scenario means that you have obtained from ATIS the flight information specified in the scenario. You may use the notepad for jotting down that information."
Hand the subject the Scenario list:
Tell subject:
Consult (when needed) the: "Tips for success in interacting with Delphi".
[CMU]
%% NOTE: original is formatted in 12pt Helvetica, makes use of bold fonts, etc
Script for the Participants Contact
Greet the participant. Present them with the "ATIS" participant folder.
Go over each sheet inside the folder, paraphrasing each section. The wording you might use as you review each sheet follows:
I. Fast Facts Sheet
a) What
"This system is know as the Air Travel Information System, or ATIS. It is a prototype of a voice input information system that contains the same information as the Official Airline Guide. The information contained in the system is the same information that you could get from a travel reservationist if you called an airline. While the information is the same, in its experimental stage ATIS can not make a reservation for you."
b) How
"To use ATIS you make your request by talking into the microphone on the headset while simultaneously holding down the mouse "Talk" button. Once you let up on the button, the computer will begin to process your request. While the computer searches the data base for the information the "Talk" button will be disabled. Once the computer has completed its search the information it has gathered will be printed onto the screen before you.
If you should make a mistake and let go of the button before you are through talking, the computer will try and answer the portion of the request that it received. If it does not understand you, or if the information was incomplete it will respond with an error message to the screen, and you should respond accordingly."
c) System Limitations
"There is a great deal of information contained in this system's database. however, remember that you are talking to a computer that is looking for "key" words in your speech and will take all of your requests quite literally. Ambiguous questions may result in a great deal of information being displayed on the screen, or may result in an error message. We suggest that you try to phrase your questions with very specific information requests".
d) Screen Dimming
"If you suddenly realize that the screen has dimmed considerably just hit the green command button and it will brighten up again."
e) Where
"On the second page of Fast Facts we have listed the cities that ATIS has programmed into its database. There are nine airports currently in this experimental database, and we ask that you limit your requests to these airports."
f) Other information
"In addition to requesting specific flights we encourage you to request other information form the system which might include: Information about abbreviations that you do not understand, airline and airport information, Inflight meals, airfares, and the other information that you see listed here."
II. The Twenty Four Hour Clock Sheet.
a) "The flight departure, and arrival time that you will see displayed on the screen will be written in the Twenty Four hour clock, or in Military time. We have included a sheet for you to help you decode this clock, and there is also one pinned to the wall beside the terminal.
A.M. is displayed inside the clock, and P.M. is displayed outside the clock."
III. The Scenario's
"We have provided you with a series of travel scenario's to help you in your effort to obtain information from ATIS. These scenario's are a loose outlines of travel plans that you can use as a basis for your travel requests. These are only loose sketches, and there are no right or wrong flights to choose from. We have provided these outlines as an aid to you, in an effort to make these requests as realistic, and entertaining as possible. You are welcome to deviate from these scenario's if you wish."
IV. Itinerary Tally Sheet
a) {Please have the participant fill out their Name and the date on the tally sheet.}
"We would like you to use this sheet to mark down the flights that you choose, based on the scenario's that we have given you. Mark down the flights that you would have chosen from the list that ATIS provides. There are extra Tally sheets on the desk if you should need more. [MIT]
Following are the instructions given to the experimenter running the session for MIT ATIS Data Collection:
Only enter an utterance if the subject has recorded it and you have heard it played back in its entirety. The subject has recorded something when the words "Utterance Recorded" appear on your screen and the prompt changes from "ATIS:" to TEXT:" Type in what the subject said, omitting false starts and filled pauses (e.g., "um," "er," etc.). When you hit return, you will see a line containing "Sentence:" followed by the text of what you have just typed on the screen. You will then see the ":" prompt. At this point, you may change what you have typed, if, for example, you misspelled a word or the subject incompletely recorded an utterance. After you have hit return on the line with the ":" prompt, what you have typed, either at the "TEXT:" prompt or the ":" prompt will be processed by the system.
After the system has processed the utterance, you will see the ":" one last time. If the utterance was correctly typed into the system, simply hit return here to get back to the "ATIS:" prompt. If, however, you have made a typo, you can recover here by typing in the correct utterance. The system will re-process what you have typed and record it with the correct digitized utterance. Please make a note of typos so that the incorrect utterance can be removed from the log.
If a subject tries to record an utterance when the system is working, he/she will hear a beep. If the subject ignores the beep and thinks that he/she HAS, in fact, recorded an utterance, type "redo" at the "ATIS:" prompt. This will produce the message "Please record again." If the subject has incompletely recorded an utterance (either the beginning or the end of the utterance is cut off when played back) type "redo" at either the "TEXT" or ":" prompts. This will produce the message "I'm sorry, I didn't record everything you said. Please record again." If the subject records silence, type "sil" at either the "TEXT:" or ":" prompts. This will produce the message "I'm sorry, you inadverdently recorded silence. Please try again."
Always ask a subject if he or she has participated in the experiment
before. If he/she has, look up the subject's name in the file:
/dynamic/user/joe/atis/subjects.text
When you start wizard-atis, add the following keyword/value pair:
:speaker-id "xx"
where "xx" is the two character speaker id associated with the subject in the file subjects.text.
If the system crashes and you see a prompt like this:
->
type
:a
at the prompt. You should then see the prompt:
>
If that doesn't work and you still see:
->
try typing :a again. (If that doesn't work, look for Joe or Stephanie.) When you see the > prompt, retype the wizard line as you did to begin the session, adding the
:speaker-id "xx" [where "xx" is the two character speaker id associated with this speaker]
keyword/value pair, as well as the following keyword/value pair:
:restart? t
The complete wizard-atis line should look something like:
(wizard-atis "inits" "wizard-name" :book-flights? nil :AtoD "new" :speaker-id "xx" :restart? t)
If the system dies completely and you see a message saying that lisp has stopped itself, type:
jobs
at the Unix prompt. If you something like:
[1] - Suspended AtisWorld | tee ....
type:
kill %x
at the Unix prompt, where x is the number associated with the job (in the example above, x = 1). Restart the Lisp as you would at the beginning of a session. When you restart wizard-atis, use both the :speaker-id keyword (the speaker-id should be the same as before the system crashed for the same speaker) and also type the following keyword/value pair:
:restart? t
Before beginning a session, type:
(demo t)
in the Lisp Listener. This will create a status window on the subject's machine (you may need to move it to the bottom of the screen) and display the "ATIS:" prompt. Have the subject practice recording (anything) to get used to the idea of pushing, talking, and releasing. There is a short delay built into the demo and you can point out that the status window displays "System working", the record icon is gray, and the "ATIS:" prompt has gone away. After a few seconds, things will return to record mode (record icon black, status window "Ready for input", and "ATIS:" prompt displayed) and the subject can once again record. The "demo" program runs through three iterations of recording.
Type verbatim what the subject said while the push-to-talk button was depressed (what you hear played back after digitization), even if you know the system will not be able to understand it. Do not enter non-speech hesitations (e.g., ``um,'' ``er'') or stammers. If a subject stutters over or mispronounces a word, enter the normal spelling for the word. Do not type in false starts that the subject corrects. For example, if the subject says, ``what are the show me the flights from boston to denver,'' you should type ``show me the flights from boston to denver'' to the system. Do not coach the subject on the best way to phrase utterances or offer example utterances of your own that you know work. Remember to have the user mark the beginning and end of scenarios by speaking ``Begin scenario'' and ``End scenario.''
The utterances that mark the beginning and ending of the scenario are not saved by the system, so it doesn't really matter if they are completely digitized or not. No matter what the subject says, ALWAYS type "Begin scenario xx" where "xx" is the number of the scenario (e.g., "one", "two", etc.) to begin a scenario and "End scenario" (with no number) to complete a scenario.
ATIS is an air travel information system under development at AT&T that will eventually provide travel assistance using spoken input to a computer. Thank you for coming in to talk to ATIS. The speech and dialogue information we collect during this session will be used to improve the system.
You will be carrying on a dialogue with a computer, asking the computer questions which it will answer as best it can. The computer may also ask you questions. Please speak to the computer as you might to a human, but be aware that it will not be able to answer all your questions, and may answer some questions incorrectly. If you have a series of questions, please ask them one at a time, and wait for a response after each one. If the system gets stuck processing your sentence, it will provide a response explaining its problem.
The information provided to you is extracted from the Official Airline Guide, or the OAG. Though the system currently knows about only 11 cities (9 airports) in the U.S., and 9 airlines serving these cities, it does have the information you will need to solve the scenarios we have provided. For example, it knows about departure and arrival times, fares, meals, and aircraft types for each flight in the database. A summary of what the system currently knows can be found on a separate page of these instructions.
You are asked to select and solve several of the twelve travel planning scenarios listed on the last two pages. Category A scenarios are considered solved when you have found the requested information. Category B scenarios are more open ended, and you can book flights if you wish to do so. You may end a scenario whenever you choose to, whether you have solved it or not.
Whenever you start a new scenario, please speak the canned sentence, "Begin scenario x," where x is the number of the scenario you have chosen.
When you are done with a scenario, please end by saying "End scenario," before you move on to the next one. This will enable us to identify which scenario you are exploring.
Here are some hints and suggestions.
The computer will answer every question with a verbal response. Please wait for this response before continuing.
A series of shorter questions often works better than a single long question.
The computer will provide detailed flight and fare information only when you have narrowed down your choices to three or fewer flights/fares.
The computer can keep track of only one flight at a time.
The computer cannot tell you when you will arrive at or depart from an intermediate stopping point. If you want that information, you must book separate flights from origin to stopping point and stopping point to destination.
The computer tries to emulate humans in the following ways:
It remembers various pieces of information, such as your destination and any fare or time restrictions you give it.
It can infer dates relative to today or a previously mentioned date.
Thanks and good luck!
Knowledge Base of Current System
ATIS knows about flights between a limited set of cities. For each
flight, ATIS knows about the following:
Airlines
Originating city
Destination city
Classes of service
Other relevant information
faresCities that ATIS knows about:
meals
ground transportation between airport and downtown
aircraft
number, location, and duration of stops
Airlines that ATIS knows about (not all airlines serve all cities):
American
Continental
Delta
Eastern
Lufthansa
Midway
TWA
United
USair
Travel Planning Scenarios NOT
E: Since the scenarios have quite a bit of flexibility, you may want to write down the specifics of the problem that you have selected to solve on a sheet of paper. This may avoid getting you and the system confused!
[Scenarios follow]
[BBN]
ATIS DATA COLLECTION
Instructions to Subjects
Thank you for participating in our data collection project.
1. Getting Acquainted
What is ATIS?
ATIS, or Airline Travel Information System, is a voice-operated system under development at BBN. The system is designed to give answers to spoken queries about flight information. The flight information that the system knows about is a subset of the information contained in the Official Airline Guide (OAG). (See Section 2 entitled "What kind of information is in the ATIS system".)
What is the purpose of this data collection effort?
Because the system we have is still under development, it will be able to answer many (we hope) but not all of your queries. The purpose of this data collection is to give us a larger sample of queries from actual users who are trying to use the system. We plan to use the collected data to help us improve the performance of the system.
What am I supposed to do?
Your task is to use the ATIS system to help you in solving several flight-related scenarios given to you by the data collector at the time of data collection. In the process of solving the scenario, don't make a special effort to restrict yourself to ask queries that are the most relevant to that scenario. In particular, when you are just starting a scenario you may want to explore the information that ATIS contains.
How to use the system
See Section 3 for a complete set of instructions on how to use the system. In addition, you will receive instructions on the use of the system from the data collector. The data collector will be monitoring the whole data collection session.
Briefly, you request information from ATIS by using the microphone to talk to the system. The screen will show what the system thinks you said and what the answer to your query is.
Remember: The purpose of this process is to collect speech data; the more queries you ask the better. Don't try to think of the shortest way you can get your information. Don't be afraid to explore.
2. What Kind of Information is in the ATIS System?
ATIS knows about flights between a limited number of cities (see list below). For each flight, ATIS knows about the following:
Cities and Airports that ATIS knows about:
CITY/AIRPORT AIRPORT CODE ALTERNATE AIRPORT NAME Atlanta ATL William B. Hartsfield Baltimore/Washington BWI Boston BOS Logan Dallas/Fort Worth DFW Denver DEN Stapleton Oakland OAK Philadelphia PHL Pittsburgh PIT San Francisco SFOThere are 11 cities served by 9 airports. Baltimore and Washington are served by the same airport; likewise for Dallas and Fort Worth. The airports in Oakland and San Francisco serve both cities.
3. How to Use the ATIS System
3.1 Scenarios After an initial training period, the data collector will give you several scenarios to solve using the ATIS system. Below is the procedure to follow for each of the scenarios:
Begin Scenario: To begin a new scenario, use the mouse to click (with the left-most mouse button) the Begin Scenario button in the top-left of the ATIS screen (see Figure 1). A menu of scenario numbers will pop up. Click the identification number of the scenario that you will be working on. (The Begin Scenario button will now say End Scenario; see below.)
Note: Make sure you write down the specifics of the problem you have selected to solve. This may avoid getting you and the system confused! Use the notepad provided to you by the data collector to write down notes for yourself as you wish throughout the data collection session.
Query Sequence: You solve the scenario by asking the ATIS system a sequence of queries. To ask a new query:
Click New Query button in the top-left corner of the ATIS screen. The System Status display will show the word "Listening", when the system is ready to accept speech input.
Speak your query into the microphone.
After a short wait, the system will show on a new card what it thought you said (see Figure 1).
After another (usually longer) wait, the system displays on the card an answer to your query. (Sometimes, this wait can be quite long; please be patient.)
Repeat the above sequence for every new query until you feel you have finished working with that scenario. Solving a scenario means that you feel you have obtained from ATIS the flight information specified in the scenario.
If after saying a query you decide that you wish to abort that query (e.g., you made a false start or you changed your mind about what you wanted to say), simply click on the Abort Query button.
End Scenario: To end a scenario, click the End Scenario button on the ATIS screen. Now, you can begin another scenario by following the above procedure.
3.2 ATIS Screen
Figure 1 shows the basic ATIS screen, which you will use to control the session and the system uses to communicate with you. The ATIS screen has two main displays: The top level menu and the cards display.
Top Level Menu: This menu is shown in the top-left part of the ATIS screen. It contains a number of buttons and status windows. We have already described how several of the buttons and status windows are used.
Cards Display: The cards display contains your queries and their answers. Each card corresponds to a single query. The top line of each card shows the card number and what the system thought you said. In the main body of the card, the system displays the answer to the query. As you say the different queries, the cards appear automatically and stack themselves in the lower right part of the screen. The stacking is done such that the old queries are visible but not their answers; only the most recent card is fully visible. After about ten cards have been created, the oldest one will be deleted automatically whenever a new one is created.
To the left of each card is a scroll bar that allows you to scroll that card vertically in case the answer occupies more space than allotted.
You will often find the need to look back at the answers to some of the previous queries. The answers to older queries can be seen by simply clicking on the push/pop button in the top-left corner of the card of interest. Clicking that button will bring forward (pop) that card and you will be able to see the whole card. Clicking the same button again will push that card back where it came from. You can pop a number of cards in that manner. To avoid having to push them all back by clicking the push/pop button, simply click on the Restack button in the top-level menu.
Note: The system only remembers the ten most recent queries. Cards that disappear from the screen can never be retrieved.
3.3 Clone Cards
In addition to being able to look back at previous queries and their answers, it is often convenient to be able to make a copy of a card for later reference. This is particularly important if you suspect that the card of interest correponds to an old query that soon may disappear from the screen.
To make a copy of a card, you clone it. To do that, click the clone button in the top-left part of the card and drag the card-skeleton that appears to the desired spot on the screen, then click the mouse again. Figure 2 shows a card that has been cloned. A clone card looks the same as a card in the Cards Display except that it has an additional top bar and different buttons. Unlike the cards in the Cards Display, the clone card can be resized using the Resize button in the top-right corner of the card.
The clone card can be deleted from the screen by clicking the Delete Clone button in the top-left corner of the card. A clone of the same card can be made again from the Cards Display, but only if that card is still visible in the Cards Display. To make the clone card disappear temporarily, click on the Iconize Clone button in the left part of the query line of the clone card. The clone card will disappear but an icon corresponding to that card will appear in the top right part of the screen with the name of the card in it. The clone card can be made visible again by clicking in the icon for that card.
You may clone as many cards as you like, within the limits of the size of the screen.
3.4 Messages from the system
If the system is unable to understand or answer your request, you will receive a brief message on the screen. Read the message and click in the box under it that says "OK" when you are ready to continue.
4. Tips for Success
Remember, this is an experimental system. It can be quite slow at times and it may not be able to answer all your queries. Please be patient. Here are some tips on how to make the session go smoothly.
Be specific. The system tends to take what you say quite literally.
Don't be long winded. Many short queries generally work better than few long ones.
Don't pause. Short pauses are OK, but if you pause too long in the middle of a question, the system will think that you have finished speaking, and will stop listening to you.
Think before you talk. Compose your query, press the New Query button on the screen, and begin talking soon after the system status changes from "ready" to "listening".
Be natural. Don't try to speak "computerese", just ask the questions in ordinary English and using your natural voice.
Refer to notes. You might want to keep this document turned to the page titled "What kind of information is in the ATIS System?" to see the kinds of things you can ask about.
Ask it! If you're not sure whether to ask a particular question or not, go ahead and give it a try!
Don't get overwhelmed when receiving large amounts of information from ATIS. When ATIS responds with an answer that contains too much information, you may want to ask additional queries that will constrain that information.
[CMU]
%%NOTE: Original includes graphics, uses 12pt Helvetica for text, and has %% additional formatting (e.g., for columns of information).
Airline Travel Information Systems
Fast Facts
What The Air Travel Information System (ATIS) is a prototype of a voice-input information retrieval system has the same information that is contained in the Official Airline Guide (OAG). It simulates an airline reservation system, although at this time it cannot make reservations for you.
How
Using ATIS is easy. When the Computer is ready to receive input from you it will enable the "talk" button.
When the button looks like this ATIS is ready, and waiting for you to request information.
When you are ready to make a request, position the mouse arrow on the talk button, and hold down the left mouse button. The button will then appear as:
[[[ GRAPHIC HERE ]]]
When you are finished with your request let go of the mouse button. ATIS will begin processing your request, and the button will look like this:
[[[ GRAPHIC HERE ]]]
This is a signal to you that ATIS is gathering the information that you requested, and will soon provide your answer to the screen before you. The information will look similar to the information in the OAG itself.
Please wait for the button to say "Push Down to Talk" before making another request.
Beginning and Ending a Session As you end each scenario, please hold down the "talk button" and state: "Ending scenario number _____". Then once the computer has enabled your "talk button" you can say "Beginning scenario number _____". This will help us label our voice recordings.
Systems limitations:
The system has a great deal of information, but will take every request literally. If you ask to see all the flights from one city to another you will be given ALL the appropriate flights. We suggest that you be as specific as possible, whenever possible. At this time reservations and seating assignments cannot be made. Where
For testing purposes, ATIS has been programmed with only a few major cities which are:
Other Information to Request
In addition to the suggested questions that will be provided on your travel scenario you are encouraged to request additional information about:
A tally sheet is provided for your convenience. Please use it to mark down the flights that make up your itinerary.
There are no right or wrong answers, this is not a test. So relax and enjoy yourself!!!
[MIT]
ATIS Data Collection Instructions to Subjects
ATIS is an air travel information system under development at MIT that will eventually provide automated travel assistance using speech as input. The data that we collect from you today will be used to develop this computer system. Your help is greatly appreciated.
During this data collection session, you will try to solve several different problem scenarios. In each scenario, you will ask the computer questions, and it will try to answer you. The system that you will be using is in its infancy and will not be able to answer all of your questions. It may even answer some questions incorrectly. Whenever it gets confused, please try asking your question in a different way.
The information that the system uses was extracted from a small subset of the Official Airline Guide. Thus, while this information is all real, the system is quite limited in its knowledge. For example, it currently knows about only 11 U.S. cities, 9 airports, and 9 airlines serving these cities. The system does know some important information to help you solve the scenarios on pages 4, 5 and 6. For example, it knows about departure and arrival times of each flight, the type of aircraft used, and whether a meal is served. A summary of what the system currently knows can be found on page 3 of these instructions.
During your conversation with the computer, it will try to interpret your questions in the context of what it has understood so far. For example, once you mention a place of departure or a destination, it will remember them in understanding later questions. The computer will answer you by both speaking and displaying a table on the screen in front of you. If you don't understand certain symbols and abbreviations that show up in the table, you can ask the computer what they mean.
We would like to emphasize that there is no "right" way to ask a question. If what you try doesn't work, try rephrasing what you have said.
TALKING TO THE SYSTEM
On pages 4, 5, and 6 there are several scenarios having to do with travel planning, and we would like you to solve some of them. When you begin a scenario, please say "Begin scenario" and the number of the scenario you have chosen. e.g. "Begin scenario 1". When you are done with each scenario, please end by saying "End scenario" before you move on to the next one.
You will now be asked to practice recording several utterances.
Thank you very much for your help.
ATIS Data Collection Information Sheet
USEFUL COMMANDS: Speak these commands as needed:
Begin scenario X (To begin a new scenario; "X" is the scenario number.) End scenario (To end a scenario) Scratch that (To wipe out the last sentence that the system understood.) Clear history (To wipe out the entire past conversation in a given scenario.)WHAT THE ATIS SYSTEM KNOWS ABOUT:
ATIS knows about flights between a limited set of cities. For each flight, ATIS knows the following:
[NOTE]: Since the scenarios have quite a bit of flexibility, you may want to write down the specifics of the problem that you have selected to solve on a sheet of paper. This may avoid getting you and the system confused!
[Scenario lists follow]
[SRI] SRI has three versions of the instructions they have given to their subjects. The first version seemed to be difficult for the subjects to understand and the second version had to be modified when they (SRI) changed their system. All three versions are included below.
[SRI-1] 1) Instructions with no accompanying demonstration: (Used for subjects q0 through q3) Subject Instructions, No Demonstration
Overview:
We are developing a computer system that allows you to ask questions
to obtain information concerning air travel planning. You are being
asked to use this initial system. You will be asked to find the best
solution to a travel-planning task, for example, to find the cheapest
fare between two cities. You can do this by asking the system for
information about various flights and fares, about ground
transportation, about meals, plane types, arrival and departure times
of flights, and so on.
Please note that although the system contains the same information the "real-life" travel agents use, the system cannot ACTUALLY make reservations for you.
Speaking to the System:
To speak to the system, you must place the cursor over this bar, and click the mouse button [experiment demonstrates]. You will see that on the left of the screen is a description of what the system is doing at any one time. Before you speak, "Ready for Speech" will be highlighted. While you are talking, "Listening" will be highlighted. After you have finished speaking, the system will detect that you are done, and the highlighting will move to "Recognizing Speech". Note that if you pause for too long, (i.e. for over one second) while you are speaking, the system may think you are done, and move to the "Recognizing Speech" stage.
Please speak naturally and clearly. The system will interpret what you say, search for the information, and sent its reply to your screen. After the system has finished recognizing your utterance, you will see what the system thought you said. This will be printed in large, bold letters. Underneath that will be a description of what the system thought you were asking, for example, "all flights from Boston to San Francisco". Finally, in the small, lower window, you will see the information that the system has determined should answer your request. This information will be in the form of tables of information.
Using the Scroll Bar:
If you make a request and get back more information than fits on one screen, you may have to use the scroll bar to see the information that does not fit on the first screen. To use the scroll bar, move the cursor to the dark bar at the side of the window on the screen [experimenter points this out], and hold down the mouse button as you move the cursor up or down. Information that was previously off the screen will now appear.
If the System Does Not Understand Your Request:
The system may not always understand your request, or be able to answer it. This could happen for three reasons. The first is because it could not correctly identify the words you used. In that case, you will see that the words it prints out first, in large letters, do not correspond to what you said. When this happens, however, it is still possible that the system will return the correct information to you. That is, despite the fact that some of the individual words you said were not correctly recognized by the system, it is still possible that the paraphrase produced by the system will be correct. In this case, you will get the information you want. However, if enough of your words are not recognized correctly, the system may not be able to answer your request. In this case you may try REPEATING YOUR REQUEST.
The second reason the system may not be able to answer your request is if it correctly recognized the words you said, but does not know how to interpret them. For instance, there may be two or more ways to interpret something you said. Since the system cannot choose between alternative interpretations, it will give you an error message. You should then REPHRASE YOUR REQUEST.
The third reason the system may not be able to answer your request is if it correctly recognizes the words you said, and knows how to interpret them, but there is no answer for your request. For instance, if you ask to see the flights that cost less than five dollars, you will get a message in the lower window that will tell you that there are no flights in the database that satisfy that condition. You should then MAKE A DIFFERENT REQUEST.
Context-Dependency of Requests Unless you tell it to do otherwise, the system will "remember" what you specified in preceding requests. For example, if you ask to see flights from Boston to Dallas, you will get a listing of those flights. If you then ask to see flights leaving after five, what you will get will be a listing of flights from Boston to Dallas that leave after five, not of all flights in the database that leave after five.
If you do NOT want the system to take previous information into account when you make a request (for instance, if you DID want to see all the flights in the database that leave after five, NOT just the ones from Boston to Dallas) then you will have to let the system know this. To eliminate any preceding context for a request, use the mouse to move the cursor to the "Start Over" box [experimenter points to this], and click once. A line will be drawn below the previous query, showing you that the system will not take into account any specifications you made in earlier queries. Remember, UNLESS you tell the system to start over in this way, all specifications you make in previous queries will be carried forward to the next one.
You may at times wish to resume using a previous context, after having started over. To go back to a previous context, move the cursor back to the sentence in your series of requests that corresponds to the previously-established context you desire, and simply use the mouse button to click once on that utterance. You will notice that the system responds by opening up a new line below the one that you clicked on. You will also notice that utterances after that one are faded out. The fading out is to show you what the system is carrying over from earlier requests. The specifications you make in any utterance that is NOT faded out will be taken into account in your new utterance.
Completing the Experimental Session:
When you think you have found the best flight(s) for the task, please write the flight number(s) down on the piece of paper provided. Inform the experimenter that you are finished with your task. You will then receive a gift certificate for your participation
[SRI-2]
2) Instructions with accompanying demonstration:
(Used for subjects q4 through qa)
SRI Subject Instructions (to accompany demonstration)
In this experiment, your task will be to speak to the system to elicit travel information in order to solve a scenario like this one. [Show training Scenario] In order to explain how to use the system, I'm going to give you a short demonstration using this training scenario.
To speak to the system, you first use the mouse to position the cursor on the "click here to talk" button, and then press the left mouse button. When you do so, you will see that the highlighting at the upper left-hand side of the screen moves from "Ready for Speech" to "Listening", and you can begin talking. So if I were working on this scenario, I could say something like this: "Show me flights from San Francisco to Dallas on September sixth"
As you can see, the system indicates what it is doing by highlighting the processes listed at the left of the screen. This black bar at the top shows you how much of the recognizing task it has completed.
The system determines when you are finished speaking by listening for a pause of more than a second. This means, of course, that if you pause for too long in the middle of your utterance, it will assume you are finished and move on to the recognizing part. If this should happen, you have to wait for it to come back before it can accept more speech.
When it comes back, it will show you three things: a transcription of your utterance, in bold blue letters, a paraphrase of how it interpreted the utterance in smaller green letters, and the answer, in the form of a table, here in the answer window below.
It's quite possible that it will make mistakes. If the wording of the transcription isn't right, you should check the paraphrase to see what the interpretation was, since it may sometimes get the words wrong but the interpretation right.
The tables that come up in the answer window are often longer than the window. To see the rest of the table, use the middle button on the mouse to move this scroll bar. Just hold the button down and drag the dark gray part in the direction you want to scroll.
When you speak to it, the system assumes that everything you've said before still holds. So for example, I could now say: "Which flights leave before nine in the morning." Now it will assume I'm still talking about flights from San Francisco to Dallas. Unless you indicate otherwise, it will always make these assumptions about the previous context. If you want to change the context, there are two ways to do it. You can click on the "start over" button [show] which eliminates all previous context. As you can see, the lines that are no longer relevant are faded out on the screen. The other way is to click on a previous context. [show] When you do that, you will go back to the context of that line.
Do you have any questions?
[At this point the microphone is given to the subject, who is asked to speak to the system. If all goes well, the subject is then given the real scenario and left alone in the room.]
[SRI-3]
3) Instructions with accompanying demonstration (modified for new interface): (Used for Subjects qb through qg)
SRI Subject Instructions (to accompany demonstration)
In this experiment, your task will be to speak to the system to elicit travel information in order to solve a scenario like this one. [Show training Scenario] In order to explain how to use the system, I'm going to give you a short demonstration using this training scenario.
To speak to the system, you first use the mouse to position the cursor on the "click here to talk" button, and then press the left mouse button. When you do so, you will see that the highlighting at the upper left-hand side of the screen moves from "Ready for Speech" to "Listening", and you can begin talking. So if I were working on this scenario, I could say something like this: "Show me flights from San Francisco to Dallas on September sixth"
As you can see, the system indicates what it is doing by highlighting the processes listed at the left of the screen. This black bar at the top shows you how much of the recognizing task it has completed.
The system determines when you are finished speaking by listening for a pause of more than a second. This means, of course, that if you pause for too long in the middle of your utterance, it will assume you are finished and move on to the recognizing part. If this should happen, you have to wait for it to come back before it can accept more speech.
When it comes back, it will show you a paraphrase of how the system interpreted your utterance, here at the top, and just below it it will give the answer in the form of a table. Down below will be a transcription of your utterance. If the paraphrase seems to be wrong, you can check the transcription to see if it misrecognized your words. It's quite possible that it will make mistakes.
The tables that come up in the answer window are sometimes longer than the window. To see the rest of the table, use the middle button on the mouse to move this scroll bar. Just hold the button down and drag the dark gray part in the direction you want to scroll.
When you speak to it, the system assumes that everything you've said before still holds. So for example, I could now say: "Which flights leave before nine in the morning." Now it will assume I'm still talking about flights from San Francisco to Dallas. Unless you indicate otherwise, it will always make these assumptions about the previous context. If you want to change the context, you should use the buttons here at the bottom. "Start Over" gives you a new context and wipes out everything you've done before. Once you press "Start Over", you can't go back to the sentences you've said before. The "Previous" button allows you to see the last sentence you said, and "Next" shows you the next sentence after the one currently on the screen. If you go up to a previous sentence, however, and then talk to the system, it erases all the next sentences after that point, so you can't get back down to them. Also, if you press "Next" when there is no next sentence, you will get an error message. Just click on "continue".
Do you have any questions?
Why don't you try speaking to it, to make sure you're comfortable with it? Ask it something about the fares.
[At this point the microphone is given to the subject, who speaks to the system. If all goes well, the subject is then given the real scenario and left alone in the room.]