; File: logfile-subject-instructions.doc., Updated 11/04/92 ; Logfile-dry-run subject instructions ; [BBN] BBN ATIS - Subject Instructions MADCOW Introduction ATIS (or Air Travel Information System) is a travel planning application that uses speech as input. We are studying how such systems are used by asking people such as yourself to use it for solving a set of problems in travel planning. The database used by the ATIS system is very similar to one that a travel agent might use to help you plan a trip. However, due to current limitations in the technology we have available, the version you will be using today is in comparison very limited. Specifically, the system knows only the following information: Cities and airports Atis will know about flights between the nine airports are listed below. You can refer to an airport by city name (e. g., BOSTON), by the airport code (e. g., BOS or B O S) or by the airport's name, if there is one, (e. g., LOGAN). The cities are as follows: City or Airport Name Airport Code Alternate airport name Atlanta ATL William B. Hartsfield Baltimore/Washington BWI --- Boston BOS Logan Dallas/Fort Worth DFW --- Denver DEN Stapleton Oakland (California) OAK --- Philadelphia PHL --- Pittsburgh PIT --- San Francisco SFO --- The system also contains a limited amount of information about ground transportation between an airport and the city (or cities) it serves, for example if a limousine service is available and what it costs. Airlines The system knows about the nine airlines listed in the table below. You can refer to them by name, for example DELTA or by abbreviation D L. Airline Abbreviation American AA Continental CO Delta DL Eastern EA Lufthansa LH Midway ML TWA TW United UA USair US If you've been following the news, you will notice that some of the airlines we list no longer exist (the database was constructed at a time when they did). For purposes of this session, please assume that they still exist and are still flying. Information about a flight The system has a great deal of information about each flight in its database. The information includes the following categories: Flight number (e.g., USAir 123). Departure and arrival times (e.g., 9:30 AM). Type of aircraft (e.g., jet, 727, etc.) and their seating capacity. Fares (for one-way, first class, with restrictions, etc.) Meal availability (breakfast, snack, etc.) Availability for days of the week (Monday, etc.) and for specific dates (September 21st, etc). Number of stops (if the flight is not direct). Summary Remember that ATIS is a limited system and knows only the information noted above. If you ask it questions about other topics (for example, the weather), ATIS will not be able to answer them. In any case, you should limit your questions to ones that bear directly on the solution of your current problem. Hints for operating the system The ATIS system is quite capable but it is nevertheless limited in its abilities. To use it most effectively, please refer to the HOW TO USE THE ATIS SYSTEM document. In addition, you should remember the following advice: When posing a query, try to speak naturally, not too fast and not too slow. Avoid using ``computerese''. The system has been trained on naturally spoken language and will respond best to such speech. Short and to-the-point questions will generally work better than long, complex queries. If the system doesn't appear to be understanding you, try rephrasing your question. If that doesn't work, there is a chance that the particular type of information you're after may lie outside the system's knowledge base. You should then consider changing your line of questioning. The scenarios Please remember the following as you do the scenarios: Begin each scenario with clicking on the SCENARIO: button. You can then choose your assigned scenario from a list of scenario identifiers. When you are finished (you either have the answer to the problem (or want to give up), call me (at x3753) and I will come in. Then click on the TYPE ANSWER button. An answer window will apprear. Type your answer. Use the right/left arrow keys to advance/back your cursor; and the backspace key to delete characters. After you have entered your answer, click on the DONE bar at the top of that window. The answer window will disappear and you may continue with your next scenario. Work exclusively on finding a solution to the problem at hand. Your goal is to identify what you believe to be the correct solution to each problem. If you encounter any difficulties in operating the system, please alert me immediately by calling x3753 or notifying me in my office (at #15/150). Note that I will not be able to help you find a solution to the problem! Call only if the system malfunctions or if you feel you don't understand something about the procedure. If you would like to rest or leave the room during the session, let me know when you've just completed a problem. (We will only be able to interrupt the session between problems.) Please try to solve every problem. If after several reasonable attempts you feel you are unable to complete a problem, please notify me. This is a plain text, unformatted version of the document used in the October 1992 dry-run of the ATIS logfile evaluation. How to Use the BBN ATIS System 1. ATIS Screen Figure 1 shows the basic ATIS screen, which you will use to control the session and which ATIS uses to communicate with you. All mouse operations are done with the left button unless otherwise indicated. The ATIS screen has three main displays: The Control Buttons, Status Indicators and the Query Display. o Control Buttons: The Control Buttons are located at the top of the ATIS screen (See Figure 1). They include a number of buttons with which you control the start/end of scenarios and the start/cancel of your query. Figure 2 provides a brief description of each button. o Status Indicators: The Status indicators are located under the Control Buttons (See Figure 1). They provide you with the current status of the ATIS system with respect to the processing of your query. Figure 2 provides a brief description of each status indicator. o Query Display: The Query Display is divided into 3 parts (as shown in Figure 1): The top part shows what the system thought you said; The middle part contains an explanation of the information the system is trying to provide you; and the bottom part contains the answer to your query. The answer contains a heading and data. To the left of each part there is a scroll bar that allows you to scroll that card vertically by using the middle mouse button (in case the answer occupies more space than allotted). The answer section also has a scroll bar along the bottom to let you scroll horizontally. There are 2 additional useful displays: The Control Panel and the Query History displays: o Control Panel: The Control Panel is used mainly to monitor the display of the query history. To open the Control Panel click the right arrow button (located on the top-right of the ATIS screen). The Control Panel includes six selectable items (see Figure 3), out of which only 2 are of interest to you. These are: Show Control Panel - Show/Hide the control panel. Click on the Box (of the left) in order to toggle the Show/Hide status. A box filled with an "X" indicates that the Control Panel will be displayed. Show History - Show/Hide the query history display. Click on the Box (of the left) in order to toggle the Show/Hide status. A box filled with an "X" indicates that the query history will be displayed. o Query History: As you speak the different queries, the queries stack themselves in the History display which may be useful to you when you need to look back at the answers to some of your previous queries. The History display (shown in Figure 4) shows snapshots of three queries at a time. Initially, snapshots of the three most recent queries will be shown. However, you may scroll up/down with the middle button to obtain snapshots of previous queries. Each snapshot is numbered and includes the query and one row of the answer to the query. When you have spotted a query for you wish to see the full answer, you may open a full display of that query by clicking (with the left-most mouse button) on the right arrow button to the right of the query. A scrollable full Query Display will appear, which you may move to a desired location on the screen. You can open a number of query displays in that manner. 2. Scenarios After an initial training period, the experimenter will give you several scenarios to solve using the ATIS system. Below is the procedure to follow: o User: Enter your first name by using the mouse (with the left-most button) to click on the User: button. A window will appear in which you can type your first name. Use right/left arrow keys to advance/backspace the cursor, and the backspace key to delete characters. Close the window by clicking on the Done bar at the top of that window. Remember, you need to enter your name only once. o Scenario: To begin a new scenario, use the mouse to click (with the left- most mouse button) the Scenario: button (see Figure 1). A menu of scenario identifiers will pop up. Select the identifier of the scenario you have been assigned to work on (by clicking on that identifier). This identifier should match the identifier specified in your hardcopy that contains the scenario description. The text of that scenario will be displayed. Note: Make sure you write down the specifics of the problem you have selected to solve. This may avoid getting you and the system confused! Use the notepad provided to you by the experimenter to write down notes for yourself as you wish throughout the experiment. o Query Sequence: You solve the scenario by asking the ATIS system a sequence of queries. To ask a new query: o When the System Status Display has the Ready bar highlighted you may click the Click To Talk button in the top-left corner of the ATIS screen. The Listening bar will then get highlighted indicating that the system is ready to accept speech input. o Speak your query into the microphone. o After a short wait, the system will show on the upper part of the screen what it thought you said (see Figure 1). o After another (usually longer) wait, the system displays on the screen an answer to your query and a sentence that describes the information the system is trying to provide. Repeat the above sequence for every new query until you feel you have finished working with that scenario. o Cancel: If during or after saying a query you decide that you wish to abort that query (e.g., you made a false start or you changed your mind about what you wanted to say), simply click on the Cancel button. o Reset: To start a scenario over (if you or the system gets confused) click on the Reset button. The system will acknowledge the restart of the scenario with a voice message: "OK, let's begin again". o Answer: Enter the answer you have obtained by clicking with the left-most mouse button on the Type Answer button. A window will appear in which you can type your answer. Use right/left arrow keys to advance/backspace the cursor, and the backspace key to delete characters. Close the window by clicking on the Done bar at the top of that window. o To indicate that you have ended a scenario and wish to begin another scenario, notify the experimenter by phone (see section 4 for details). The experimenter will hand you your next scenario. When you are ready to solve the problem described in the scenario click again the Scenario: button. Now, you can begin another scenario by following the above procedure. o When you have completed your four scenarios notify the experimenter. You don't need to click any more on the Scenario button. 3. Tips for Success Don't pause. Short pauses are OK, but if you pause too long in the middle of a question, the system will think that you have finished speaking, and will stop listening to you. Think before you talk. Compose your query, press the Click To Talk button on the screen, and begin talking soon thereafter. Be natural. Don't try to speak "computerese", just ask the questions in ordinary English and using your natural voice. Refer to notes. You might want to keep the ATIS crib sheet at your sight to see the kinds of things you can ask about. Ask it! If you're not sure whether to ask a particular question or not, go ahead and give it a try! Empty fields in the answer. If you encounter an empty field in the answer, do not consider the corresponding line as part of your answer. Blank line at the bottom of a scrollable answer. Be aware that a blank line may (occasionally) appear at the bottom of a vertically scrollable answer. Do not assume that there is no more information displayed as part of the answer. Rather, observe the scroll bar and if it indicates that there is more to view, scroll down/up as appropriate. 4. To Contact the Experimenter To contact the experimenter just call Varda Shaked at x3753 or walk down the hall to her office (at 15/150). ; ; [CMU] Subject Instructions for End-to-End Evaluation MADCOW This document describes the standard subject instruction procedure to be used in end-to-end evaluation. The materials consist of the following items: 1. text of subject instructions for experimenter 2. subject's version of instructions (for reading along) 3. subject's crib sheet The instructions used at a particular site will need to be customized to reflect differences in speech input procedures and display configuration. Points in the instruction sequence at which this should be done have been marked with . All sites should submit a complete version of the instructions as used at their site to NIST as part of evaluation data submission. It is to a site's advantage to have properly instructed subjects use their system. Confused subjects will not be able to make effective use of a system and the reported performance will suffer accordingly. Sites should take the necessary steps to properly train the experimenter. The experimenter should be thoroughly familiar with the subject instructions (as well as the experimental procedure described in the companion document Experimenter Instructions for End-to-End Evaluation) prior to the start of data collection. The experimenter should be able to fluently deliver the instructions to the subject, reading from the text in a practiced manner. All experimenters should practice their instruction delivery a minimum of two times before the start of data collection. The practice sessions should be ``full-dress rehearsals'' and should (within reason) use participants who are from the same subject population as the eventual subjects. The entire session does not need to be rehearsed, but the rehearsal should continue up to the completion of the first real scenario, this should be followed by administration of the questionnaire. The following sections represent the order of instructions to be given to the subject. The subject should have a copy of these instructions to follow along with the spoken delivery. The material in Roman face should be read to the subject. Material in [brackets] is for the benefit of the experimenter and should not be shared with the subject. 1 Experimenter version of Instructions 1.1 Introduction ATIS (or Air Travel Information System) is a travel planning application that uses speech as input. We are studying how such systems are used by asking people such as yourself to use it for solving a set of problems in travel planning. The database used by the ATIS system is very similar to one that a travel agent might use to help you plan a trip. However, due to current limitations in the technology we have available, the version you will be using today is in comparison very limited. Specifically, the system knows only the following information: 1.1.1 Cities and airports Atis will know about flights between the nine airports that are listed on your crib sheet. The cities are as follows: [read down the column of the table] City or Airport Name Atlanta Baltimore/Washington Boston Dallas/Fort Worth Denver Oakland (California) Philadelphia Pittsburgh San Francisco The system also contains a limited amount of information about ground transportation between an airport and the city (or cities) it serves, for example it can tell you if a limousine service is available and what it costs. 1.1.2 Airlines The system knows about the nine airlines listed on your crib sheet. [read down the column] Airline American Continental Delta Eastern Lufthansa Midway TWA United USAir If you've been following the news, you will notice that some of the airlines we list no longer exist (the database was constructed at a time, February 1990, when they did). For purposes of this session, please assume that they still exist and are still flying. 1.1.3 Information about a flight The system has a great deal of information about each flight in its database. The information includes the following categories: Read only the first part of each item and not the parenthesized portion. o Abbreviations (ATL = Atlanta, AA = American Airlines, etc.) o Aircraft (names, seating capacity, etc.) o Airfares (class of fare, restrictions, price, etc.) o Airports (Logan Airport is in Boston, etc.) o Classes of Service (one-way, coach, first class, etc.) o Flight Numbers (USAir 123, etc.) o Ground Transportation (between airport and downtown) o Meals (served on the flight, e.g breakfast, snack, etc.) o Number of Stops (zero or more) o Schedules (arrival, departure time, days of week, dates, etc.) 1.1.4 Summary Remember that ATIS is a limited system and knows only the information noted above. If you ask it questions about other topics (for example, the weather), ATIS will not be able to answer them. In any case, you should limit your questions to ones that bear directly on the solution of your current problem. 1.2 Becoming familiar with the system To acquaint you with the operation of the ATIS system, you will first complete a practice problem, which we will refer to as a ``scenario''. We will begin with an explanation of how to operate the system. 1.2.1 The Display [Site-specific instructions that explain to the subject the various components of the visual display. These presumably include windows or fields that display such elements as the recognized utterance, perhaps a paraphrase of it, the retrieved information, the location and meaning of error messages. Other interface attributes, such as additional display elements or synthetic speech feedback should also be explained at this time. The goal is to ensure that the subject will not experience any surprises during the data collection proper.] The display contains five separate fields. System Status The dark grey field provides information about the state of the system, for example whether it's ready to listen to you or whether it's processing. If you're not quite sure what the system is doing or what you can do, you should look here. Heard This field tells you what the system has heard. This field will not necessarily correspond to what you have said, but it will correspond to what the system has recognized. Result The result of your query (that is, information from the database) will appear in this field. Query Context This field provides you with the assumptions the system is making when it goes about answering your query. Note that combines information from your current utterance with context setting information from your previous utterances. If you feel that the system has made erroneous assumptions, you have the option to clear the context and starting over by clicking the Reset Context button. Scenario The Scenario window will show the problem you are currently working on. Start Scenario When you are ready to start a problem, click this button; it will display a problem in the Scenario window and enable speech input. 1.2.2 Speech input [Site-specific instructions that explain to the subject how to enter a spoken query. These should presumably explain the input protocol, microphone positioning, reasonable sound level and any other relevant details. Ideally the subject should actually speak something into the system at this point.] Microphone Place the microphone on your head. Note that the microphone should be about an inch away from your mouth. It should be to the side, out of the way of the air stream coming out of your mouth. Please be aware of microphone position, as the ability of the system to understand what you're saying depends on it. If you notice it slipping out of position, please readjust it. If the system is having difficulties understanding you, you might check the microphone position. Button The Alternate button on the keyboard is use to make the system attend to your speech. Please follow the following procedure in addressing the system: -- Think of what you want to say. -- When you're ready, press down the Alternate button and wait for the word Listening to appear in the System Status box. -- Say your sentence. -- When you're done speaking (and only then) let go of the button. It's important to wait until after you've stopped speaking to let go of the button, as doing so any earlier may cut off your sentence, possibly leading to an error. 1.3 Hints for operating the system The ATIS system is quite capable but it is nevertheless limited in its abilities. To use it most effectively, you should remember the following advice: o When posing a query, try to speak naturally, not too fast and not too slow. Avoid using ``computerese''. The system has been trained on naturally spoken language and will respond best to such speech. o Short and to-the-point questions will generally work better than long, complex queries. o If the system doesn't appear to be understanding you, try rephrasing your question. If that doesn't work, there is a chance that the particular type of information you're after may lie outside the system's knowledge base; you should consider changing your line of questioning. 1.4 Practice scenario We will now do a practice scenario. The purpose is to better acquaint you with the ATIS system and to give you an idea of the type of problem you will be solving. I will be able to help you with any problems you may have with the equipment or with understanding the display, but I will not be able to answer any questions about the problem itself. If you find you need to be reminded of anything, such as which cities ATIS knows about, please refer to your copy of the instructions or to the crib sheet. To start this scenario, click on the Start Scenario button. When you feel you have found the answer to the problem, click the End Scenario button. This will bring up a panel into which you should type in your answer. You will need to do this for each of the scenarios you carry out. Please go ahead. [Place the scenario in the subject's view; do not initiate any interaction with the subject, wait for them to ask questions. Only the subject should be talking to the system, since we wish to eliminate, as much as possible, the opportunity for subjects to model appropriate behavior. If they are doing something clearly wrong (e.g. misusing the equipment, you may interrupt and correct them).] 1.5 The scenarios We are now ready to start the main set of scenarios. Before we do, I would like to remind you of the following: o Begin each scenario by clicking the Start Scenario button. o Please work exclusively on finding a solution to the problem at hand. Your goal is to identify what you believe to be the correct solution to each problem that I give you. o If you encounter any difficulties in operating the system, please alert me immediately calling out. Note that I will not be able to help you find a solution to the problem! Call only if the system malfunctions or if you feel you don't understand something about the procedure. o If you would like to rest or leave the room during the session, let me know when you've just completed a problem. (We will only be able to interrupt the session for at most five minutes at a time and only between problems.) o Please try to solve every problem. If after several reasonable attempts you feel you are unable to complete a problem, please click the End Scenario button. For your answer enter none. o When you have completed a scenario, please enter your answer in the box that will appear on the screen. Use the answer template shown at the bottom of the scenario to determine what information your answer should contain. Do you have any last questions about the procedure? Remember that we are interested in how well the system helps you solve a problem. You should at all times concentrate on problem solution. If you are interested in exploring the system's capabilities we will be happy to arrange an additional session for this purpose. Good luck! [ Wait for the subject to correctly initiate the scenario, then leave the experimental enclosure. Do not return until the subject has terminated the scenario. Remove the just-completed scenario and replace it with the next one. Continue until all required scenarios have been completed. ] 1.6 Questionnaire administration You have completed your assigned problem set. I would now like you to fill out the following questionnaire. Please let me know when you are done. [Leave the enclosure; return when signalled by the subject.] 1.7 Session conclusion At this point, formal data collection is at an end and the experimenter should proceed to complete the session as outlined in the Experimenter Instructions: debriefing and site-specific administrative procedures.] 2 Subject Instructions 2.1 Introduction ATIS (or Air Travel Information System) is a travel planning application that uses speech as input. We are studying how such systems are used by asking people such as yourself to use it for solving a set of problems in travel planning. The database used by the ATIS system is very similar to one that a travel agent might use to help you plan a trip. However, due to current limitations in the technology we have available, the version you will be using today is in comparison very limited. Specifically, the system knows only the following information: 2.1.1 Cities and airports Atis will know about flights between the nine airports that are listed on your crib sheet. The cities are as follows: City or Airport Name Atlanta Baltimore/Washington Boston Dallas/Fort Worth Denver Oakland (California) Philadelphia Pittsburgh San Francisco The system also contains a limited amount of information about ground transportation between an airport and the city (or cities) it serves, for example it can tell you if a limousine service is available and what it costs. 2.1.2 Airlines The system knows about the nine airlines listed on your crib sheet. Airline American Continental Delta Eastern Lufthansa Midway TWA United USAir If you've been following the news, you will notice that some of the airlines we list no longer exist (the database was constructed at a time, February 1990, when they did). For purposes of this session, please assume that they still exist and are still flying. 2.1.3 Information about a flight The system has a great deal of information about each flight in its database. The information includes the following categories: o Abbreviations (ATL = Atlanta, AA = American Airlines, etc.) o Aircraft (names, seating capacity, etc.) o Airfares (class of fare, restrictions, price, etc.) o Airports (Logan Airport is in Boston, etc.) o Classes of Service (one-way, coach, first class, etc.) o Flight Numbers (USAir 123, etc.) o Ground Transportation (between airport and downtown) o Meals (served on the flight, e.g breakfast, snack, etc.) o Number of Stops (zero or more) o Schedules (arrival, departure time, days of week, dates, etc.) 2.1.4 Summary Remember that ATIS is a limited system and knows only the information noted above. If you ask it questions about other topics (for example, the weather), ATIS will not be able to answer them. In any case, you should limit your questions to ones that bear directly on the solution of your current problem. 2.2 Becoming familiar with the system To acquaint you with the operation of the ATIS system, you will first complete a practice problem, which we will refer to as a ``scenario''. We will begin with an explanation of how to operate the system. 2.2.1 The Display The display contains five separate fields. System Status The dark grey field provides information about the state of the system, for example whether it's ready to listen to you or whether it's processing. If you're not quite sure what the system is doing or what you can do, you should look here. Heard This field tells you what the system has heard. This field will not necessarily correspond to what you have said, but it will correspond to what the system has recognized. Result The result of your query (that is, information from the database) will appear in this field. Query Context This field provides you with the assumptions the system is making when it goes about answering your query. Note that combines information from your current utterance with context setting information from your previous utterances. If you feel that the system has made erroneous assumptions, you have the option to clear the context and starting over by clicking the Reset Context button. Scenario The Scenario window will show the problem you are currently working on. Start Scenario When you are ready to start a problem, click this button; it will display a problem in the Scenario window and enable speech input. 2.2.2 Speech input Microphone Place the microphone on your head. Note that the microphone should be about an inch away from your mouth. It should be to the side, out of the way of the air stream coming out of your mouth. Please be aware of microphone position, as the ability of the system to understand what you're saying depends on it. If you notice it slipping out of position, please readjust it. If the system is having difficulties understanding you, you might check the microphone position. Button The Alternate button on the keyboard is use to make the system attend to your speech. Please follow the following procedure in addressing the system: -- Think of what you want to say. -- When you're ready, press down the Alternate button and wait for the word Listening to appear in the System Status box. -- Say your sentence. -- When you're done speaking (and only then) let go of the button. It's important to wait until after you've stopped speaking to let go of the button, as doing so any earlier may cut off your sentence, possibly leading to an error. 2.3 Hints for operating the system The ATIS system is quite capable but it is nevertheless limited in its abilities. To use it most effectively, you should remember the following advice: o When posing a query, try to speak naturally, not too fast and not too slow. Avoid using ``computerese''. The system has been trained on naturally spoken language and will respond best to such speech. o Short and to-the-point questions will generally work better than long, complex queries. o If the system doesn't appear to be understanding you, try rephrasing your question. If that doesn't work, there is a chance that the particular type of information you're after may lie outside the system's knowledge base; you should consider changing your line of questioning. 2.4 Practice scenario We will now do a practice scenario. The purpose is to better acquaint you with the ATIS system and to give you an idea of the type of problem you will be solving. I will be able to help you with any problems you may have with the equipment or with understanding the display, but I will not be able to answer any questions about the problem itself. If you find you need to be reminded of anything, such as which cities ATIS knows about, please refer to your copy of the instructions or to the crib sheet. To start this scenario, click on the Start Scenario button. When you feel you have found the answer to the problem, click the End Scenario button. This will bring up a panel into which you should type in your answer. You will need to do this for each of the scenarios you carry out. Please go ahead. 2.5 The scenarios We are now ready to start the main set of scenarios. Before we do, I would like to remind you of the following: o Begin each scenario by clicking the Start Scenario button. o Please work exclusively on finding a solution to the problem at hand. Your goal is to identify what you believe to be the correct solution to each problem that I give you. o If you encounter any difficulties in operating the system, please alert me immediately calling out. Note that I will not be able to help you find a solution to the problem! Call only if the system malfunctions or if you feel you don't understand something about the procedure. o If you would like to rest or leave the room during the session, let me know when you've just completed a problem. (We will only be able to interrupt the session for at most five minutes at a time and only between problems.) o Please try to solve every problem. If after several reasonable attempts you feel you are unable to complete a problem, please click the End Scenario button. For your answer enter none. o When you have completed a scenario, please enter your answer in the box that will appear on the screen. Use the answer template shown at the bottom of the scenario to determine what information your answer should contain. Do you have any last questions about the procedure? Remember that we are interested in how well the system helps you solve a problem. You should at all times concentrate on problem solution. If you are interested in exploring the system's capabilities we will be happy to arrange an additional session for this purpose. Good luck! Subject's crib sheet ATIS: Air Travel Information Service OVERVIEW The database installed in this system is a subset of data from the Official Airline Guide from February 1990. The traveler is forewarned that the data are quite out of date! Cities Served Airlines Atlanta American Baltimore/Washington Continental Boston Delta Dallas/Fort Worth Eastern Denver Lufthansa Oakland (California) Midway Philadelphia TWA Pittsburgh United San Francisco USAir Other Information Abbreviations (ATL = Atlanta, AA = American Airlines, etc.) Aircraft (names, seating capacity, etc.) Airfares (class of fare, restrictions, price, etc.) Airports (Logan Airport is in Boston, etc.) Classes of Service (one-way, coach, first class, etc.) Flight Numbers (USAir 123, etc.) Ground Transportation (between airport and downtown) Meals (served on the flight, e.g breakfast, snack, etc.) Number of Stops (zero or more) Schedules (arrival, departure time, days of week, dates, etc.) ; ; [MIT] Subject Instructions for End-to-End Evaluation MADCOW October 9, 1992 1 Subject version of Instructions 1.1 Introduction ATIS (or Air Travel Information System) is a travel planning application that uses speech as input. We are studying how such systems are used by asking people such as yourself to use it for solving a set of problems in travel planning. The database used by the ATIS system is very similar to one that a travel agent might use to help you plan a trip. However, due to current limitations in the technology we have available, the version you will be using today is in comparison very limited. Specifically, the system knows only the following information: 1.1.1 Cities and airports ATIS will know about flights between the nine airports that are listed on your sheet. You can refer to an airport by city name (for example, PITTSBURGH), by the airport code (for example, ATL for Atlanta) or by the airport's name, if there is one, (for example, LOGAN airport in Boston). The cities are as follows: 1 ______________________________ |__City_or_Airport_Name__|____ | Atlanta | | Baltimore/Washington | | Boston | | Dallas/Fort Worth | | Denver | | Oakland (California) | | Philadelphia | | Pittsburgh | |________San_Francisco________| The system also contains a limited amount of information about ground transportation between an airport and the city (or cities) it serves, for example it can tell you if a limousine service is available and what it costs. 1.1.2 Airlines The system knows about the nine airlines listed in the table below. You can refer to them by name, for example DELTA or by abbreviation, for example AA. ________________ |____Airline____| | American | | Continental | | Delta | | Eastern | | Lufthansa | | Midway | | TWA | | United | |____USAir____|_ If you've been following the news, you will notice that some of the airlines we list no longer exist (the database was constructed at a time, February 1990, when they did). For purposes of this session, please assume that they still exist and are still flying. 2 1.1.3 Information about a flight The system has a great deal of information about each flight in its database. The information includes the following categories: o Abbreviations (ATL = Atlanta, AA = American Airlines, etc.) o Aircraft (names, seating capacity, etc.) o Airfares (class of fare, restrictions, price, etc.) o Airports (Logan Airport is in Boston, etc.) o Classes of Service (one-way, coach, first class, etc.) o Flight Numbers (USAir 123, etc.) o Ground Transportation (between airport and downtown) o Meals (served on the flight, e.g breakfast, snack, etc.) o Number of Stops (zero or more) o Schedules (arrival, departure time, days of week, dates, etc.) 1.1.4 Summary Remember that ATIS is a limited system and knows only the information noted above. If you ask it questions about other topics (for example, the weather), ATIS will not be able to answer them. In any case, you should limit your questions to ones that bear directly on the solution of your current problem. 1.2 Hints for operating the system The ATIS system is quite capable but it is nevertheless limited in its abilities. To use it most effectively, you should remember the following advice: o When posing a query, try to speak naturally, not too fast and not too slow. Avoid using "computerese". The system has been trained on naturally spoken language and will respond best to such speech. 3 o Short and to-the-point questions will generally work better than long, complex queries. o If the system doesn't appear to be understanding you, try rephrasing your question. If that doesn't work, there is a chance that the particular type of information you're after may lie outside the system's knowledge base. You should then consider changing your line of questioning. 1.3 The MIT ATIS system The version of the ATIS system you will be using keeps a record of what it has understood so far and will interpret your queries in that context. There are two ways in which the user can modify this discourse history, using the Scratch last sentence or Clear history buttons. They are described below. In addition, the MIT ATIS system assumes that you may want to book flights, in which case the system will guide you in making certain choices. 1.4 The display The display you will be using is divided into four parts: o At the top, a ticket window, where information about the current status of your flight plans is displayed; o Below the ticket window, four system status icons, which will flash depending upon what part of the system is currently working; o The system response window, where the result of each query will be displayed. This window will show you the words that were understood by the automatic speech rec- ognizer, a summary of what the system understood (which will also be spoken by a speech synthesizer), and the information that was retrieved from the database. This window will also display error messages that are produced by the system. o The control panel, which you will use to interact with the system. You will use the buttons in this window to: - Provide speech input to the recognizer (using the microphone icon). - Begin, answer, or cancel a scenario, described in more detail below. - Scroll backward (using the ^ key) or forward (using the _ key) in the display window, or scroll to the end of the display (using the End key). 4 - Erase the effects of the immediately preceding query (using Scratch last sentence) or erase the entire discourse history of the scenario (using Clear history). (Note: These keys will be inactive while the system is processing your utterance. If you misspeak, please wait until the system responds before deciding to use one of these keys.) The number of the scenario you are currently working on is also displayed on the control panel. Only the currently active buttons are displayed at any given time. To begin a scenario, click on Begin scenario. This will activate (and display) the other buttons. Once you have recorded an utterance, all interactive buttons disappear until a response has been generated. No further input to the system is possible during this period. Once the microphone button reappears, the system is ready for your next query. 1.5 Speech input When you are ready to talk to the computer: o Click and HOLD DOWN the left mouse button INSIDE the microphone button. o Wait until the icon turns GREEN. o Hold it down the entire time you are speaking. o Release the button when you are done speaking. The system can "hear you" only when you hold the mouse button down, inside the micro- phone icon. If you forget to click on the microphone button, don't worry - just start over, clicking on the microphone. If you release the mouse button before you are done speaking, the system will only hear a portion of what you said. You may want to use the Scratch last sentence button and start your question over. 1.6 The scenarios We are now ready to start the main set of scenarios. Before we do, I would like to remind you of the following: 5 o Begin each scenario by clicking on the "Begin scenario" button in the control panel of your screen. The text of the scenario will then appear on your screen. You may begin speaking to the system at this point using the microphone icon. o When you feel you have found the answer to the problem, you should click on the "Answer scenario" button in the control panel on your screen. A pop-up window will appear where you can type in your answer to the scenario. o Please work exclusively on finding a solution to the problem at hand. Your goal is to identify what you believe to be the correct solution to each problem that I give you. o If you encounter any difficulties in operating the system, please alert me immediately by calling me (just talk into your headset, you don't need to push the microphone button). Note that I will not be able to help you find a solution to the problem! Call only if the system malfunctions or if you feel you don't understand something about the procedure. o If you would like to rest or leave the room during the session, let me know when you've just completed a problem. (We will only be able to interrupt the session for at most five minutes at a time and only between problems.) o Please try to solve every problem. If after several reasonable attempts you feel you are unable to complete a problem, please click on the "Cancel scenario" button on the control panel. Good luck! 6 2 Subject's crib sheet ATIS: Air Travel Information Service OVERVIEW The database installed in this system is a subset of data from the Official Airline Guide from February 1990. The traveler is forewarned that the data are quite out of date! Cities Served Airlines Atlanta American Baltimore/Washington Continental Boston Delta Dallas/Fort Worth Eastern Denver Lufthansa Oakland (California) Midway Philadelphia TWA Pittsburgh United San Francisco USAir Other Information Abbreviations (ATL = Atlanta, AA = American Airlines, etc.) Aircraft (names, seating capacity, etc.) Airfares (class of fare, restrictions, price, etc.) Airports (Logan Airport is in Boston, etc.) Classes of Service (one-way, coach, first class, etc.) Flight Numbers (USAir 123, etc.) Ground Transportation (between airport and downtown) Meals (served on the flight, e.g breakfast, snack, etc.) Number of Stops (zero or more) Schedules (arrival, departure time, days of week, dates, etc.) ; ; [SRI] Instructions: The document "Subject Instructions for End-to-End Evaluation" was augmented in the following manner: Sections 1.1 through 1.2 were read aloud by the experimenter, with the subject following along on a separate copy. The experimenter read section 1.3 before proceeding to 1.2.1 so that subjects would know how to speak before they were asked to say something to the system. At Section 1.2.1, the experimenter began to show the subject how to use the SRI SLS by introducing the practice scenario. These sections were read to the subject: We will now work on a practice scenario. The purpose is to better acquaint you with the ATIS system and to give you an idea of the type of problem you will be solving. I will be able to help you with any problems you may have with the equipment or with understanding the display, but I will not be able to answer any questions about the problem itself. If you find you need to be reminded of anything, such as which cities ATIS knows about, please refer to your copy of the instructions. Here is a sample scenario. First read it through and when you are ready to ask your first query to the system, let me know. The experimenter then proceeded to give the site-specific instructions for system use. When the subject was ready to ask the first query, the experimenter showed the subject how to use the Push-to-Talk button To speak to the system, move the mouse cursor into the "Push and Hold Here to Talk Button". Then push any mouse, hold it down while you are speaking, release it when you are finished. The experimenter then let the subject ask this first query, correcting the subject if the subject had any problems. While the query was being recognized, the experimenter explains the highlighting of sections on the screen reflecting the state of the system. You'll notice that the system highlights what it's doing over here on the left. When the system is ready for speech again, "Ready" will be highlighted in green. This bar shows you how much of the recognition task has been done so far. When the system came back with the answer, the experimenter explained how to interpret it. You will then see the output of the recognizer in this part of the screen (point to location). Since this is an automatic system, and it's not as smart as a human listener, you will see some errors in the recognized string. Up in this part of the screen (point to location) is a sentence or paraphrase which expresses what the system THOUGHT you were asking for, given that the system assumes you are trying to get information about air travel. Underneath the paraphrase (point to location) is a table of information from the database. Often, the system will be able to determine what type of information you were asking for despite some errors in the recognized string. In other cases, however, errors in the recognized string may lead to a paraphrase that you do not desire, and you'll need to ask again for the information you desire. Then the experimenter explained the Context mechanism. When asking again for the information you desire, it's important to tell you about a major aspect of this system, which is that the system takes CONTEXT into account each time you ask a question. For your first question, there is no previous context, but once you have asked a question, the specifications in that question will be carried over to the next question. For example, if you have a list of all flights from San Francisco to Boston on the screen, and then you ask for just nonstop flights, the system will give you nonstop flights from San Francisco to Boston, rather that all the nonstop flights in the entire database. It will assume that these specifications should hold unless you change the context or ask about different cities. It's important to keep this in mind when errors occur, because if you currently have some undesired information on the screen, it will be carried over into your next query--UNLESS you switch your context. The system allows you to switch context by using the "Previous Question" and "Next Question" buttons (point to location and demonstrate by pushing buttons). You may also press "Start Over", which wipes out all previous context. If you press "Start Over" you will no longer be able to get back to the previous questions. There is a scroll bar here which you can use if there is ever too much information in the answer to fit into the window. This version of the system is sometimes used for other purposes. For this experiment, please don't press these buttons: "Abort," "Microphone switch," "Playback," "Vocabulary," "Overview," or "Quit." When you feel that you have found the answer to the problem, you should write down the answer on the scenario sheet. Do you have any questions? Questions pertaining to the display or mechanics of system use were answered, and then the experimenter read section 1.5, including the following site specific protocols: Begin each scenario with your first query. If you encounter any difficulties in operating the system, please alert me by speaking into the microphone. I will be monitoring your speech, but please note that I will not be able to help you find a solution to the problem! Call me only if the system malfunctions or if you feel you don't understand something about the procedure. Please try to solve every problem. If after several reasonable attempts you feel you are unable to complete a problem, please say so into the microphone. You will find that that the scenarios differ greatly in how difficult they are to solve. Some may be quite easy, while others may have solutions that aren't obvious. If you cannot find a solution to the scenario in a straightforward manner, it may help to try less obvious ways to solve it. The experimenter then read the rest of the instructions. ;