File pofi.doc Version 3.0 10/26/94

INTERPRETING ATIS QUERIES RE THE DATABASE

1. General Principles:

1.1. Only reasonable interpretations will be used.

The annotators must decide if a linguistically possible interpretation is reasonable or not. If applying the Principles of Interpretation strictly results in an interpretation that in judgement of the annotators is clearly not what was intended by the speaker, they have the discretion to add additional interpretations that they believe do reflect the intentions of the speaker. The strict interpretation according to the Principles of Interpretation will always be allowed, however.

1.2. The context will be used in deciding if an interpretation is reasonable.

Rules below which call for certain ambiguities are always overridden by context; e.g., although "8 o'clock" may be ambiguous in general between A.M. and P.M., if the context makes it clear that only one of these interpretations is reasonable, only that one will be used, and the phrase will not be considered ambiguous.

1.3 Each interpretation must be expressible as one SQL statement, unless another form of answer is explicitly allowed (e.g., yes/no questions. See 2.11.). (In such a case, however, the query must also have at least one answer producible by a single SQL statement, to ensure that systems that answer with only database contents still have a way to respond to every answerable query. See 1.6.)

1.4 All interpretations meeting the above rules will be used by the annotators to generate possible reference answers.

A query is thus ambiguous iff it has at least two interpretations that are fairly represented by distinct SQL expressions. (For practical reasons, if a query has more than six interpretations, we consider it hopelessly vague rather than testably ambiguous.) The reference SQL expression stands as a semantic representation or logical form. If a query has two interpretations that result in the same SQL, it will not be considered ambiguous. The fact that the two distinct SQL expressions may yield the same answer given the database is immaterial.

The annotators must be aware of the usual sources of ambiguity, such as structural ambiguity, exemplified by cases like "What is the earliest flight leaving Boston and arriving in Atlanta on November eleventh?", in which the modifying phrase "on November eleventh" may be attached to modify either "arriving in Atlanta" or "leaving Boston and arriving in Atlanta". More generally, if structural ambiguities like this could result in different (SQL) interpretations, they must be treated as ambiguous.

1.5 An attribute X of a type of entity Y may be used in connection with another type of entity Z, with the interpretation "Z's bearing the relation R to Y's with attribute X," for some particular relation R. Uncertainty as to what relation R is intended will be treated as an ambiguity.

For example, modifiers such as "discount", "first class", or "round trip" always refers to fares; when they are used as a modifier of something else, like seats or flights, they must be interpreted as referring to a set of associated fares. Thus "discount (or first class or round trip) flights" would be interpreted as meaning "flights with "discount (or first class or round trip) fares."

1.6 Some queries are considered unanswerable. Such a query will be classed "X" in its categorization file to indicate this. Detailed rules for classing queries "X" may be found in the documentation of the classification system. In general, in order to be answerable, a query must have a valid SQL interpretation whose evaluation with respect to the data base produces a valid answer, and which contains only information that a system under test has valid access to. As an example of the latter case, a query is considered unanswerable if, in order to answer it, a system would have to know a certain incorrect date that had been given previously in the recording session only in an answer. But if the subject picks up on the date and repeats it in a query, that query is perfectly well answerable.

1.7 Interpretations may incorporate certain pieces of prior context. In constructing an interpretation, an annotator may include, in addition to information in the query itself, information that has either appeared in the dialog in prior questions or that could be derived from the correct minimal answers to prior questions plus the contents of the database. (Such an interpretation is called "context-dependent" and will be tagged in its categorization file with an abbreviated pointer to the nearest prior question or answer that supplies the information.) The notion of inclusion intended here is quite close to a cut-and-paste of words or phrases, with appropriate adjustments for morpho-syntactic correctness. Information that is only implied by or deducible from this prior context is not allowed.

Note the congruence of this principle with the prior one: an interpretation is invalid if it requires additional information from an incorrect prior answer or from some aspect of the display that is not included in the correct answers, such as column headings.

1.8 In making interpretations, the annotators should incorporate the minimal constraints from context needed to generate the correct minimal answer, with the exception noted below in principle 2.29. This general principle is justified by the fact that, given the mechanism for generating a maximal answer (documented separately in "min_max_eval.txt.), using more contextual information than is necessary to generate the correct minimal answer would result in that information getting into the maximal answer, and sites supplying answers could not predict which fields could be in the maximal answer and which couldn't. When exceptions to this generality are agreed on, they will be explicitly documented, as in principle 2.29, so that sites won't have this problem.

1.9 If a subject uses terminology in a way that violates the Principles of Interpretation after being exposed to this usage in a system display or response, then the utterance will either be reinterpreted in accordance with the Principles of Interpretation, if it is possible to do so coherently, or be classified X under Principle 1.7 if it is not possible to interpret it coherently in accordance with the Principles of Interpretation.

If a subject spontaneously uses terminology in a way that violates the Principles of Interpretation, unprompted by such a use by the system, then the utterance will be interpreted as the subject intended, despite the violation of the Principles of Interpretation.

1.10 Any utterances that have been significantly affected by a data collection system error will be classified as X:from-wizard-error.

2. Specific Principles:

In this arena, certain English expressions have special meanings, particularly in terms of the database distributed by TI in the spring of 1990 and revised in November 1990 and May 1991. Here are the ones we have agreed on: (In the following, "A.B" refers to field B of table A.)

2.1. Requests for enumeration.

A large class of tables in the database have entries that can be taken as defining things that can be asked for in a query. In the answer, each of these things will be identified by giving a value of the primary key of its table. These tables are:

   Name of Table      English Term(s)        Primary Key
   aircraft           aircraft, equipment    aircraft_code
   airline            airline                airline_code
   airport            airport                airport_code
   city               city                   city_code
   fare_basis         fare code              fare_basis_code
   class_of_service   class, class code      booking_class
   fare               fare                   fare_id
   flight             flight                 flight_id
   food_service       meals                  meal_code,meal_number,
                                             compartment
   ground_service     ground transportation  city_code,airport_code,
                                             transport_type
   month              months                 month_number
   restriction        restrictions           restriction_code
   state              names of states        state_code
   time_zone          time zones             time_zone_code
   dual_carrier       dual carrier service   main_airline,
                                             low_flight_number,
                                             high_flight_number
   flight_stop        (intermediate) stops   flight_id, stop_number    
 
2.2 Flights.

2.2.1 A flight "between X and Y" means a flight "from X to Y".

2.2.2 The "itinerary" of a flight refers to the set of all nonstop legs of that flight. When an "itinerary" is asked for, each leg of the flight will be identified by the set of tuples for the flight from the flight_leg table, including all fields in that table.

2.2.3 Stops.

2.2.3.1 A request for a flight's stops will be interpreted as asking for the intermediate stops only, from the flight_stop table.

2.2.3.2 In answering other questions about stops, the flight.stops field alone will be used if at all possible; the flight_stop table will only be used if it must be, and in that case, the SQL will include its primary key fields. (This means that fields from the flight_stop table are allowed in the maximal answer only in the latter case.)

Examples of queries that can be answered using only the stops field of the flight table are: "Is that flight nonstop?", "Which of those flights have stops?", "How many stops does that flight have?", and the interpretation of the yes/no question "Does that flight have a stop?" that is "How many stops does that flight have?".

Examples of queries that must be answered using the flight_stop table are: "Does that flight have a stop in Atlanta?", "What is the stop for that flight?", "Does that flight have a stop that's longer than 30 minutes?", "Show me flights that stop in Denver.", and the interpretation of the yes/no question "Does that flight have a stop?" that is "What is the stop for that flight?".

2.2.3.3 "Stopovers" will mean "stops" unless the context clearly indicates that the subject intended "stopover", as in "Can I make a two day stopover on that flight?". In that case the query is answered using the stopover column of the restrictions table.

2.2.4 Multi-leg flights.

2.2.4.1 In case of an inconsistency in the flight table between the information in a tuple for a multi-leg flight and the information in the tuples for its component legs, the information in the tuple for the overall flight will take precedence in answering queries ABOUT THE OVERALL FLIGHT over the information in the tuples for the component legs.

2.2.4.2 In deciding whether a multi-leg flight has a given property when some or all of its legs have the property, these rules apply:

A flight serves a particular meal if AT LEAST ONE of its legs serves the meal.

A flight is a dual carrier flight if AT LEAST ONE of its legs is a dual carrier flight.

A flight is on a given airline if ALL of its legs are on the airline.

A flight is on a particular kind of aircraft if AT LEAST ONE of the aircraft in the aircraft_code_sequence for the flight is that kind of aircraft.

2.2.4.3 References to the "legs" of a flight will generally be ambiguous between the nonstop legs of a flight and the maximal direct flight segments of the flight.

2.2.5 If a subject has been shown a flight number sequence for a connecting flight, subsequent references to one of the flight numbers in the sequence may be ambiguous between just the direct flight segment with that flight number or the entire connecting flight. (Generally only references to the flight number of the first direct flight segment are ambiguous; see the interpretation guidelines.)

2.2.6 A "red eye" flight is one that leaves between 9 P.M. and 3 A.M. and arrives between 5 A.M. and 12 noon.

2.2.7 An "overnight" flight is equivalent to a "red eye" flight.

2.2.8 The location of a departure, stop, or arrival should always be taken to be an airport.

2.2.9 The "schedule" of a flight is the arrival and departure time for every nonstop leg of the flight.

2.2.10 Temporal modifiers of flights, with no explicit indication of departure or arrival, such as "5:00 flight", "morning flight", or "late flight", are to be interpreted as referring unambiguously to the departure time of the flight, unless it is clear from context that the speaker intended to refer to the arrival time of the flight. For example, if the speaker says "the 4:15 flight" and there's one in the previous answer that arrives at 4:15 but none that leaves at 4:15, then surely they meant arrival_time. (For precise definition of the time modifiers, see 2.4 "Time".)

2.2.11 "What are the stops?" is ambiguous between an "event" interpretation of "stop" and a "location" interpretation. The "event" interpretation is answerable by enumerating the stops, as per 2.1 above; the "location" interpretation is answerable by listing the airports, as per 2.2.8 above.

2.2.12 "The times for those/that flight(s)" may be ambiguous between departure times ONLY, and BOTH departure and arrival time(s). If the subject is clearly asking for more than one time for a flight, as in "What are the times for that flight?", both departure and arrival times will be given. If just one time for a flight is asked for, as in "What is the time for that flight?", only departure time will be given. If it is unclear, as in "What are the times for those flights?", there is an ambiguity, and either type of answer may be given.

2.2.13 Queries asking for a flight that returns on the same day as another flight whose date is not specified, such as "I'd like to fly a round trip from A to B and return on the same day.", are currently considered to be uninterpretable.

2.2.14 A flight is from/to a city iff it is from/to an airport serving that city.

2.2.15 Change of planes.

2.2.15.1 A flight is considered to have a change of planes if the value of the aircraft_code_sequence field for the flight contains a "/". This will be true just in case the value of the aircraft_code_sequence field does not occur in the aircraft_code of the aircraft table.

2.2.15.2 Queries about the location of changes of planes will be considered unanswerable.

2.3 Fares.

2.3.1 A "one way" fare is a fare for which round_trip_required = "NO".

2.3.2 A "round trip" fare is a fare with a nonnull value for fare.round_trip_cost.

2.3.3 References to, or restrictions on, the cost of fares will be interpreted as applying to the one direction cost, unless it is clear from the context that the round trip cost is intended.

2.3.4 A "coach" fare is one whose fare_basis.class_type = "COACH". Similarly, the fare modifiers "first class", "business class", and "thrift class" refer to values of the fare_basis.class_type field.

2.3.5 A reference to ranking of fares, e.g. "fares that are Y class or better", will be interpreted as a reference to the rank of the associated booking class (class_of_service.rank).

2.3.6 A "discounted fare" is one whose fare_basis.discounted = "YES".

2.3.7 Both "excursion fare" and "special fare" mean fares with nonnull restriction codes (fare.restriction_code).

2.3.8 A "cheap" fare is an economy fare that is not class Y; that is, with economy = "YES" and booking_class NOT = "Y" in the fare_basis table.

2.3.9 For requests for airline fares, it will always be permissible to also include flights having those fares in the answer.

2.3.10 Restrictions on fares are not to be applied implicitly in answering queries. For example, a request for fares for a flight on a date near the session date will not omit those fares whose advance purchase restriction cannot be met.

2.3.11 A fare can be said to be on a day, available on a day, etc., if and only if the fare has a fare basis code whose basis_days field includes the day in question, AND the fare applies to a flight whose flight_days field includes the day in question.

2.4 Times.

2.4.1 The normal answer to otherwise unmodified "when" queries will be a time of day, not a date or a duration.

2.4.2 The answer to queries like "On what days does flight X fly" will be a list of days.day_name fields.

2.4.3 Time expressions (other than the "eight hundred hours" style) with a number from 1 to 12 in the hour position which don't specify "a.m." or "p.m.", such as "8:50", are ambiguous and may be interpreted as either a.m. or p.m.

2.4.4 Periods of the day.

The following table gives precise interpretations for some vague terms referring to time periods. The time intervals given include the end points. Items flagged with "*" are in the current (rdb4) database interval table. Each period is a contiguous block of time, so when an END TIME is less than a corresponding BEGIN TIME, it refers to the time on the next day.


            PERIOD       BEGIN TIME   END TIME
           morning*          0000       1200
           afternoon*        1200       1800
           evening*          1800       2200
           day*               600       1800
           night*            1800        600
           early morning*    0000        800
           mid morning*       800       1000
           late morning*     1000       1200
           early afternoon*  1200       1400
           mid afternoon*    1400       1600
           late afternoon*   1600       1800
           midday*           1100       1500
           early evening     1800       2000
           late evening      2000       2200
           early night       1800       2200
           late night        2200       0300
           early [flight]    0000       1000
           late [flight]     2000       0300
           breakfast time    0600       0900
           lunch time        1100       1400
           dinner time       1700       2000
           late night        2200       0300
           A.M.              0000       1159
           P.M.              1200       2359
2.4.5 "Earliest", when the relevant period of time is not further specified, will be taken to mean "earliest on any day (after midnight)", as opposed to earliest after the time of utterance or earliest after the beginning of a day (that is, 6 a.m.) as defined in the table above.

2.4.6 Interval endpoints.

Here is a list of a number of terms for time intervals, showing whether or not the end points of the intervals are to be including in the interval. The "periods of the day" referred to here are shown explicitly in 2.4.4.

                             INCLUDES
                TERM        ENDPOINTS?
             before T1          N
             after T1           N
            between T1 and T2   Y
            arriving by T1      Y
            departing by T1     Y
            periods of the day  Y
2.4.7 "Noon" means strictly 1200 hours.

2.4.8 Flights "arriving on X": to decide if a flight's arrival day is the same as it's departure day, first calculate the hours gained due to time zone changes:

        hours_gained =   hours_from_gmt(from_airport)
                       - hours_from_gmt(to_airport)
and then use this to calculate the flight's arrival day:
       IF arrival_time + 100*hours_gained < departure_time,
         THEN arrival_day = departure_day + 1
         ELSE arrival_day = departure_day
  
2.4.9 When a day or date modifier is used with a period of the day that overlaps a day/date boundary, as in "Wednesday night", the period referred to is the period beginning on that day/date, not ending on it.

2.4.10 A date without a specified year will be interpreted as being in the period from and including the date of utterance up to but not including a year from the date of utterance. For instance, "March third", when uttered on a March third, must mean the date of the session that includes the utterance, not a year later. (Note that the year may be specified indirectly via allowable prior context or certain time phrases like "last March third" or "next March third".)

2.4.11 Times in interpretations are represented as a 24-hour clock in the range 0000 - 2399 inclusively.

2.4.12 "Before" means numerically less than, and "after" means numerically greater than.

2.4.13 The interpretation of "between (time) A and (time) B", when B refers to a time earlier than A according to the preceding principles, as in "between eleven P.M. and one A.M.", is "at or after A or at or before B". 2.4.14 "Midnight" means strictly 0000 hours.

2.5 Units.

All units will be the same as those implicit in the database (e.g. feet for aircraft.wing_span, but miles for aircraft.range_miles, durations in minutes).

2.6 Meals.

2.6.1 For purposes of determining flights "with meals/meal service", snacks will count as a meal.

2.6.2 "list the types of meal" should produce one tuple per meal, not a single meal_code string.

2.7 "With" clauses.

"with"-modification clauses: "show me all the flights from X to Y with their fares" will require the identification of both flights and their fares (so if there are 2 flights, each with three fares, the answer will have 6 tuples, each with at least the flight_code and fare_code). In general, queries asking for information from two or more separate tables in the database will require the join of fields that would identify each table entry separately.

2.8 "Class".

"Class" refers to the contents of the "booking_class" field in the "class_of_service" table.

2.9 Meaning requests.

2.9.1 With the particular exceptions noted below, requests for the "meaning" of something will only be interpretable if that thing is a code with a canned decoding definition in the database. In case the code field is not the key field of the table, information should be returned for all tuples that match on the code field. Here are the things so defined, with the fields containing their decoding:

         Table          Code Field       Decoding Field
       aircraft         aircraft_code    aircraft_description 
       airline          airline_code     airline_name
       airport          airport_code     airport_name
       city             city_code        city_name
       class_of_service booking_class    class_description
       code_description code             description
       days             days_code        day_name
     equipment_sequence aircraft_code_sequence
                                         *aircraft_description
       fare_basis       fare_basis_code  booking_class, class_type,
                                           premium, economy, discounted,
                                           night, season, basis_days,
                                           fare_basis_code
       food_service     meal_code        meal_description
       interval         period           begin_time,
                                           end_time
       month            month_number     month_name
       state            state_code       state_name
       time_zone        time_zone_code   time_zone_name
       restriction      restriction_code advance_purchase, stopovers, 
                                           saturday_stay_required,
                                           minimum_stay, maximum_stay,
                                           application, no_discounts,
                                           restriction_code
* - The aircraft_description field that answers this type of query is found in the aircraft table, joined with the equipment_sequence table on the aircraft_code fields.

(Note that for the fare_basis_code and restriction_codes, all columns in their tables are to be included in the minimal answer.)

2.9.2 "Define X" and "explain X" should be interpreted the same as "What does X mean."

2.9.3 Requests for more information.

2.9.3.1 Requests for additional information about X or all information about X will be answered with ALL columns from the database table for X, if X refers to a type of entity that corresponds to a table; otherwise the request will be classified as unanswerable.

2.9.3.2 Requests for "specifics", "details", or "particulars" about X, or requests to "describe" X that are not treated as requests for the meaning of X, will be treated as requests for all information about X.

2.9.3.3 Requests for information that do not fall under the previous two principles, but which occur in situations that suggest that more than the usual amount of information is desired, will be treated as ambiguous between normal requests for information and requests for all information.

2.9.4 "What kind of X is Y" queries, where Y is clearly a kind of X, will be interpreted as equivalent to "what does Y mean?", where Y is a code field value for the table referred to by X.

2.10 Meaning/enumeration ambiguity. Vague queries such as "What is/are X?", "Describe X.", or "Give me information about X." may be treated as meaning questions ("What does 'X' mean?"), or as requests for enumeration ("list X"), or ambiguous between the two, according to the judgement of the annotators as to what the speaker meant in a particular case.

2.11 Yes/no questions.

2.11.1 Queries that are literally yes-or-no questions are considered to be ambiguous between interpretation as a yes-or-no question and interpretation as the corresponding wh-question. For example, "Are there flights from Boston to Philly?" may be answered by either a boolean value ("YES/TRUE/NO/FALSE") or a table of flights from Boston to Philadelphia. Annotators creating reference answers will provide what they feel, as native English speakers, are the entities being asked about.

2.11.2 A yes-or-no question about the existence of a difference, such as "Is there a difference in advance purchase requirements between AP/80 and AP/57?" has a corresponding wh-question whose treatment is described below in 2.20.

2.11.3 Presupposition failures (excluding the number presupposition specifically allowed in 2.25) on yes/no questions make the yes/no interpretation unreasonable, so that only the wh-question interpretation, giving a tabular answer, exists.

2.11.4 A yes-or-no question about a feature of individual members of a set, phrased as a question about the set as a whole, as in "Are those flights one way?", can be answered by a table of only the positive ("yes") instances, e.g. the subset of those flights that are one way. In other words, "Are those Xs Y?" has the corresponding wh-question "Which Xs are Y?". Note that in questions like this, there is a presupposition that all the members of the set have the same value for the feature, and if this is not true, the yes/no question interpretation is unreasonable and will be ignored, according to principle 2.11.3 above, although the wh-question interpretation is still valid.

2.12 A city and an airport will be considered "near" (or "nearby") each other iff the city is served by the airport, and two cities will be considered "near" (or "nearby") each other iff there is an airport that serves them both.

2.13 When it is clear that an airline is being referred to, the term "American" by itself will be taken as unambiguously referring to American Airlines.

2.14 References to "the city" or "downtown" are ambiguous, and may be interpreted as referring to any city that seems reasonable. If there is no indication which city is being referred to, the query may also be taken as referring to the set of all cities served by the airport.

2.15 When a query refers to an aircraft type by giving an identifying code, such as "737", the answer consists of all the aircraft that have that code in either the aircraft.code field or the aircraft.basic_type field or both. If a manufacturer (such as "BOEING") is also given, it must match the value of aircraft.manufacturer.

2.16 Utterances whose answers require arithmetic computation are not now considered to be interpretable; this does not apply to arithmetic comparisons, including computing the maximum or minimum value of a field, or counting elements of a set of tuples.

2.17 Tolerances.

When a query is hedged by use of "around X" or "about X", where X is numeric, as in "Show me the flights that leave around 4 o'clock", these plus-or-minus tolerances should be used on the different types of quantities:

       Times ............. 30 minutes
       Dates ............. zero
       Days of Week ...... zero
       Other Quantities .. 10%
       (such as fares)
If none of these rules apply, then the tolerance is zero. Time periods defined by these tolerances include the end points.

2.18 Constraints on Flights and Fares.

When a constraint such as date, day, or class is given in a query that asks for flights and fares, as in

"Flights available Saturdays along with their fares"
"Flights and their fares on 7/23"
"Flights on 7/23", "What are the fares?"
the query must be interpreted with the constraint applying to both flights and fares. (Given the current structure of our database, if the constraint is taken to apply only to flights, fares will be returned that are not available on the date/day mentioned in the question.)

2.19 Prices and costs will be treated as asking for fares rather than as asking for dollar amounts.

2.20 Queries asking for a "difference" may on some occasions be interpreted as requesting a general comparison between attributes of entities, and in these cases, they may be answered by listing the entities in question along with the values for the attribute in question. For example, the query "What is the difference in advance purchase requirements between AP/80 and AP/57?" may be answered by listing the two fare classes along with their advance purchase requirements.

On other occasions, "difference" specifically means arithmetic subtraction, as in "What is the time difference between Dallas and Denver?", and the query will be counted as an unanswerable arithmetic query (see 2.16 above).

If an instance of "difference" seems ambiguous between the two interpretations, only the nonarithmetic, general comparison interpretation will be considered. (Note that this is an exception to the general rule that the existence of a valid but unanswerable interpretation make the whole query unanswerable.)

2.21 Questions about the calendar, such as "What is the date of next Thursday?", are not now considered to be interpretable.

2.22 References to these holidays are to be interpreted as referring to the dates shown:

         Holiday     Date
        Christmas    12/25
        New Year's   01/01
        4th of July  07/04
2.23 In the answer to a request for a set of counts, such as "How many flights on each airline have business class?", inclusion of the tuples for which the count is 0 is optional.

2.24 The measure of the size of an airplane is its seating capacity.

2.25 All presuppositions about the number of answers (either existence or uniqueness) will be ignored.

2.26 "Ticket", in addition to its literal meaning, is basically ambiguous between "fare" and "flight". For example, in "Show me tickets from Dallas to Boston serving lunch.", it would have to be taken to mean "flight", while in "How much is a ticket from Dallas to Boston?", it must refer to fares.

2.27 The terms "Dallas Fort Worth," "Baltimore Washington," "Minneapolis St. Paul," and "Seattle Tacoma" are basically ambiguous and may refer to either the cities or the airports so named.

2.28 The set of possible interpretations for a code is limited to the fields in the database that the code actually occurs in, but it may be additionally restricted by context.

Note that this implies that if there are just two code fields in the database having the value "Y", then the query "What is code Y?" with no contextual restriction is just two ways ambiguous and perfectly well answerable.

2.29 In response to a request for "departures" or "arrivals", the minimal answer should contain the identifications of the flights composing the answer, and the maximal answer may contain, in addition, the departure or arrival times and the origin or destination airports. This is not to be taken as defining what "departure" and "arrival" mean.

2.30 Tag questions with negative main clauses, such as "You don't know of a flight on Thursday, do you?" are not considered to be interpretable.

Tag questions with positive main clauses, such as "There are nonstop flights from Boston to Dallas, aren't there?" are considered yes/no questions such that "yes" indicates agreement with the main clause, and "no" indicates disagreement with the main clause.

2.31 "A to B and C" in the utterance "I want to go from A to B and C" has three interpretations:

  1. A to C stopping in B
  2. the union of A to B and A to C
  3. the union of A to B and B to C

2.32 Expressions of preference in queries are to be treated as absolute requirements; that is, in queries like "I want X and Y and preferably Z" or "... Z if possible", "preferably" and "if possible" have no force and Z is as definite a requirement as X and Y.

2.33 When it is clear that an airline is being referred to, the term "Canadian" by itself will be taken as unambiguously referring to Canadian Airlines.

2.34 "Seat" should be interpreted as equivalent to "ticket" unless it is literally about seats, as in "How many seats are there on a 747?".

2.35 Queries asking "how many ..." are interpreted as asking for a number, not a set of tuples.

2.36 "Bay area" will be taken to refer to San Francisco, Oakland, and San Jose, California, unless it is clear that this is not what the speaker intended.

2.37 Ground transportation.

2.37.1 If it is clear that a user is talking about ground transportation between an airport and a city, "train" will be interpreted to mean rapid transit in the ground_service table. otherwise, the possibility exists that the user is talking about some other type of train, and the query will be considered unanswerable.

2.37.2 Queries whose interpretation requires knowing the name of a particular city or metropolitan area transit system will be considered unanswerable.

2.38 An airline serves an airport (or city) if the airline has a flight TO the airport (or TO an airport that serves the city).

2.39 Since flight IDs and fare IDs are intended only for internal database use and have no significance outside the database, they should not be displayed to users, and utterances that refer to flight IDs or fare IDs will be considered uninterpretable.

2.40 Airports (in the 46 city ATIS database).

2.40.1 The following interpretations of the expressions

" airport"
"the airport"
will always be allowed for these specific cities:
	    Los Angeles		Los Angeles International
	    San Francisco	San Francisco International
	    St. Petersburg	St. Petersburg/Clearwater International
	    Tampa		Tampa International
[NOTE: The annotators have said they want to be allowed to make exceptions to this principle. In view of the lateness of this objection being raised, the strict version of the principle will be in effect for the December 1994 evaluation, but the issue will be re-opened before any subsequent evaluations.]

2.40.2 For any city, if there is a specific airport serving the city that is uniquely salient in the context of the utterance, the expressions

" airport"
"the airport"
will be allowed to be interpreted as referring to that airport.

2.40.3 The interpretations of the expressions

" international airport" "the international airport"
will be as follows for the specified cities:
	    Los Angeles		Los Angeles International
	    San Francisco	San Francisco International
	    St. Petersburg	St. Petersburg/Clearwater International
	    Tampa		Tampa International
	    Chicago		O'Hare International
	    Houston		Houston Intercontinental
	    Dallas		Dallas/Fort Worth International
	    Montreal		Dorval International
	    Toronto		Lester B. Pearson International
2.40.4 In all other cases, the expressions
"<city name> airport"
"the <city name> airport"
"<city name> international airport"
"the <city name> international airport"
will be interpreted as referring to the set of all airports serving the city.


File: guidelin.doc, Updated 08/16/93 Version 2.0

Guidelines for the Principles of Interpretation

The principles of interpretation as stated allow some leeway for annotators and adjudicators to use their own best judgement in formulating interpretations of questions. For example, there is a general rule that context may make one interpretation of an ambiguous query unreasonable, and therefore ignorable; but deciding in particular cases that context makes an interpretation unreasonable is largely left up to human judgement. This document contains individual cases that exemplify judgements made by annotators and adjudicators, along with some general rules-of-thumb, to serve as guidelines for system builders who want to build the same kind of judgement into their systems. When possible, cases will be keyed to the principle that seems most relevant; at the bottom of this document, the section "Miscellaneous Cases" contains some isolated cases that haven't yet yielded a general rule-of-thumb.

1.1. Only reasonable interpretations will be used.

Guideline: if there are several possible interpretations of an utterance, and the subject should have known that one of them would produce a null answer, that one is probably unreasonable.

More generally, if one interpretation violates a pragmatic principle or maxim, such as "Be relevant", and another interpretation is available that has no such violations, then the first is probably unreasonable and may therefore be ignored.

   Case #1: (hypothetical)

       A1: [a list of flights]
       Q2: "Which ones serve breakfast?"
       A2: [none of them]
       Q3: "Which ones have stops?"
"Ones" in Q3 is basically ambiguous between referring to a.) the list of flights in A1; and b.) the subset of them that serve breakfast. If there had been some flights which served breakfast, both interpretations would have been reasonable; but because the b.) interpretation would yield a null answer, and the speaker must have known this, it was rejected as unreasonable.

Guideline: if there are several possible interpretations of an utterance, and one of them is a context-dependent equivalent of the immediately preceding query, that one is probably unreasonable.

  Case #1: (Nov. '92 test data)

     ii0023 Q: "What flights are available from Boston to Denver today?"
            A: [did not understand]
     ii0033 Q: "What flights are available from Boston to Denver?"
The question in this case was whether or not ii0033 could be interpreted in a context-dependent manner as totally equivalent to the preceding query, ii0023, carrying over the "today". The annotators and adjudicators decided against allowing this inter- pretation. We include a suggestion that one feature of a context dependent interpretation that might make it unreasonable is if it is context-dependent on the immediately preceding utterance, and that interpretation would make it a request for the same information.

2.2.5 References to legs for connecting flights are basically ambiguous between referring to the leg or to the entire flight.

Guideline: references to second or later legs of a flight are almost certainly not reasonably interpreted as referring to the entire flight.

Because some annotators believe that a case might conceivably arise in which reference to a second leg would reasonably be interpreted as referring to the whole flight, this rule has not been made absolute. In fact, no such case has yet (5/15/92) occurred in real data. Here is their hypothetical example of such a case, in which Q4 might be taken as referring to the whole flight:

      Q1: "Show me flights from Boston to San Francisco."
      A1: [a table of flights]
      Q2: "What United Airlines flights leave in the afternoon?"
      A2: 
             DAYS       FROM TO  LEAVE ARRIVE AIRLINE + FLIGHT #
             ---------- ---- --- ----- ------ --------------------
             DAILY      BOS  SFO  1745   2127 UA93
             DAILY      BOS  OAK  1720   2205 UA281/UA673
             DAILY      BOS  SFO  1720   2214 UA281/UA297

      Q3: "What is the fare for the flight leaving at 5:20?"
      Q4: "What is the fare for United Airlines, flight 297?"

2.11.1 Queries that are literally yes-or-no questions are considered to be ambiguous between interpretation as a yes-or-no question and interpretation as the corresponding wh-question.

Guideline: if it seems clear that the subject was requesting information FROM the database, and not information about the system's ABILITY to provide that information, then a yes/no interpretation is unreasonable.

Some hypothetical cases where the yes-or-no interpretation is unreasonable are:

      "Can you show me the morning flights?"
      "Can you tell me if there are morning flights?"
MISCELLANEOUS CASES 1. The expression "... flight 915 ..." can't reasonably be interpreted as the flight that leaves or arrives at 9:15, as in utterance qx0061ss:
      A5: 
          FARE   ONE  ROUND
        RESTRICT WAY  TRIP   FROM TO  LEAVE ARRIVE AIRLINE + FLIGHT # 
          ---    ----  ----  ---- --- ----- ------ --------------------
           F     $428  $856   ATL BWI  9:15  10:55 EA202
           Y     $286  $572   ATL BWI  9:15  10:55 EA202
           F     $428  $856   ATL BWI 10:02  11:45 DL1204
           Y     $286  $572   ATL BWI 10:02  11:45 DL1204
           .       .     .     .   .    .      .     .
           .       .     .     .   .    .      .     .
           .       .     .     .   .    .      .     .

      Q6: "List the fares for Eastern's flight nine fifteen."
2. Often a change of cities in a later query will cause the annotators and adjudicators to drop all other constraints from context, but not always. Here is a case where another constraint should NOT be dropped:
     ig0036: "In flight meal, Pittsburgh to Atlanta, Wednesday"
     ig0046: "Atlanta to Oakland, Thursday"
It was decided that one valid interpretation of the second query included the constraint that the flights serve a meal.