File pofi.doc Version 2.1 7/16/92


1. General Principles:

1.1. Only reasonable interpretations will be used.

An annotator or judge must decide if a linguistically possible interpretation is reasonable or not.

1.2. The context will be used in deciding if an interpretation is reasonable.

Rules below which call for certain ambiguities are always overridden by context; e.g., although "8 o'clock" may be ambiguous in general between A.M. and P.M., if the context makes it clear that only one of these interpretations is reasonable, only that one will be used, and the phrase will not be considered ambiguous.

1.3 Each interpretation must be expressible as one SQL statement, unless another form of answer is explicitly allowed (e.g., yes/no questions. See 2.11.). (In such a case, however, the query must also have at least one answer producible by a single SQL statement, to ensure that systems that answer with only database contents still have a way to respond to every answerable query. See 1.6.)

1.4 All interpretations meeting the above rules will be used by the annotators to generate possible reference answers.

A query is thus ambiguous iff it has at least two interpretations that are fairly represented by distinct SQL expressions. (For practical reasons, if a query has more than six interpretations, we consider it hopelessly vague rather than testably ambiguous.) The reference SQL expression stands as a semantic representation or logical form. If a query has two interpretations that result in the same SQL, it will not be considered ambiguous. The fact that the two distinct SQL expressions may yield the same answer given the database is immaterial.

The annotators must be aware of the usual sources of ambiguity, such as structural ambiguity, exemplified by cases like "What is the earliest flight leaving Boston and arriving in Atlanta on November eleventh?", in which the modifying phrase "on November eleventh" may be attached to modify either "arriving in Atlanta" or "leaving Boston and arriving in Atlanta". More generally, if structural ambiguities like this could result in different (SQL) interpretations, they must be treated as ambiguous.

1.5 An attribute X of a type of entity Y may be used in connection with another type of entity Z, with the interpretation "Z's bearing the relation R to Y's with attribute X," for some particular relation R. Uncertainty as to what relation R is intended will be treated as an ambiguity.

For example, modifiers such as "discount", "first class", or "round trip" always refers to fares; when they are used as a modifier of something else, like seats or flights, they must be interpreted as referring to a set of associated fares. Thus "discount (or first class or round trip) flights" would be interpreted as meaning "flights with "discount (or first class or round trip) fares."

1.6 Some queries are considered unanswerable. Such a query will be classed "X" in its categorization file to indicate this. Detailed rules for classing queries "X" may be found in the documentation of the classification system. In general, in order to be answerable, a query must have a valid SQL interpretation whose evaluation with respect to the data base produces a valid answer, and which contains only information that a system under test has valid access to. As an example of the latter case, a query is considered unanswerable if, in order to answer it, a system would have to know a certain incorrect date that had been given previously in the recording session only in an answer. But if the subject picks up on the date and repeats it in a query, that query is perfectly well answerable.

1.7 Interpretations may incorporate certain pieces of prior context. In constructing an interpretation, an annotator may include, in addition to information in the query itself, information that has either appeared in the dialog in prior questions or that could be derived from the correct minimal answers to prior questions plus the contents of the database. (Such an interpretation is called "context-dependent" and will be tagged in its categorization file with an abbreviated pointer to the nearest prior question or answer that supplies the information.) The notion of inclusion intended here is quite close to a cut-and-paste of words or phrases, with appropriate adjustments for morpho-syntactic correctness. Information that is only implied by or deducible from this prior context is not allowed.

Note the congruence of this principle with the prior one: an interpretation is invalid if it requires additional information from an incorrect prior answer or from some aspect of the display that is not included in the correct answers, such as column headings.

1.8 In making interpretations, the annotators should incorporate the minimal constraints from context needed to generate the correct minimal answer, with the exception noted below in principle 2.29. This general principle is justified by the fact that, given the mechanism for generating a maximal answer (documented separately in "min_max_eval.txt.), using more contextual information than is necessary to generate the correct minimal answer would result in that information getting into the maximal answer, and sites supplying answers could not predict which fields could be in the maximal answer and which couldn't. When exceptions to this generality are agreed on, they will be explicitly documented, as in principle 2.29, so that sites won't have this problem.

2. Specific Principles:

In this arena, certain English expressions have special meanings, particularly in terms of the database distributed by TI in the spring of 1990 and revised in November 1990 and May 1991. Here are the ones we have agreed on: (In the following, "A.B" refers to field B of table A.)

2.1. Requests for enumeration.

A large class of tables in the database have entries that can be taken as defining things that can be asked for in a query. In the answer, each of these things will be identified by giving a value of the primary key of its table. These tables are:

   Name of Table      English Term(s)        Primary Key
   aircraft           aircraft, equipment    aircraft_code
   airline            airline                airline_code
   airport            airport                airport_code
   city               city                   city_code
   fare_basis         fare code              fare_basis_code
   class_of_service   class, class code      booking_class
   fare               fare                   fare_id
   flight             flight                 flight_id
   food_service       meals                  meal_code,meal_number,
   ground_service     ground transportation  city_code,airport_code,
   month              months                 month_number
   restriction        restrictions           restriction_code
   state              names of states        state_code
   time_zone          time zones             time_zone_code
   dual_carrier       dual carrier service   main_airline,
   flight_stop        (intermediate) stops   flight_id, stop_number    
2.2 Flights.

2.2.1 A flight "between X and Y" means a flight "from X to Y".

2.2.2 The "itinerary" of a flight refers to the set of all non-stop legs of that flight. When an "itinerary" is asked for, each leg of the flight will be identified by the set of tuples for the flight from the flight_leg table, including all fields.

2.2.3 Stops A request for a flight's stops will be interpreted as asking for the intermediate stops only, from the flight_stop table. In answering other questions about stops, the flight.stops field alone will be used if at all possible; the flight_stop table will only be used if it must be, and in that case, the SQL will include its primary key fields. (This means that fields from the flight_stop table are allowed in the maximal answer only in the latter case.)

Examples of queries that can be answered using only the stops field of the flight table are: "Is that flight nonstop?", "Which of those flights have stops?", "How many stops does that flight have?", and the interpretation of the yes/no question "Does that flight have a stop?" that is "How many stops does that flight have?".

Examples of queries that must be answered using the flight_stop table are: "Does that flight have a stop in Atlanta?", "What is the stop for that flight?", "Does that flight have a stop that's longer than 30 minutes?", "Show me flights that stop in Denver.", and the interpretation of the yes/no question "Does that flight have a stop?" that is "What is the stop for that flight?". "Stopovers" will mean "stops" unless the context clearly indicates that the subject intended "stopover", as in "Can I make a two day stopover on that flight?". In that case the query is answered using the stopover column of the restrictions table.

2.2.4 Properties of multi-leg flights.

In deciding whether a multi-leg flight has a given property when some or all of its legs have the property, these rules apply:

A flight serves a particular meal if AT LEAST ONE of its legs serves the meal.

A flight is a dual-carrier flight if AT LEAST ONE of its legs is a dual-carrier flight.

A flight is on a given airline if ALL of its legs are on the airline.

A flight is on a particular kind of aircraft if ALL of its legs are on that kind of aircraft.

2.2.5 References to legs for connecting flights are basically ambiguous between referring to the leg or to the entire flight. (Generally only the first leg; see the interpretation guidelines.)

2.2.6 A "red-eye" flight is one that leaves between 9 P.M. and 3 A.M. and arrives between 5 A.M. and 12 noon.

2.2.7 An "overnight" flight is equivalent to a "red-eye" flight.

2.2.8 The location of a departure, stop, or arrival should always be taken to be an airport.

2.2.9 The "schedule" of a flight is the arrival and departure time for every non-stop leg of the flight.

2.2.10 Temporal modifiers of flights, with no explicit indication of departure or arrival, such as "5:00 flight", "morning flight", or "late flight", are to be interpreted as refering unambiguously to the departure time of the flight, unless it is clear from context that the speaker intended to refer to the arrival time of the flight. For example, if the speaker says "the 4:15 flight" and there's one in the previous answer that arrives at 4:15 but none that leaves at 4:15, then surely they meant arrival_time. (For precise definition of the time modifiers, see 2.4 "Time".)

2.2.11 "What are the stops?" is ambiguous between an "event" interpretation of "stop" and a "location" interpretation. The "event" interpretation is answerable by enumerating the stops, as per 2.1 above; the "location" interpretation is answerable by listing the airports, as per 2.2.8 above.

2.2.12 "The times for those/that flight(s)" may be ambiguous between departure and arrival time. If the subject is clearly asking for more than one time for a flight, as in "What are the times for that flight?", both departure and arrival times will be given (ala "schedule"). If just one time for a flight is asked for, as in "What is the time for that flight?", only departure time will be given. And if this distinction is unclear, as in "What are the times for those flights?", there is an ambiguity, and either type of answer may be given.

2.2.13 Queries asking for a flight that returns on the same day as another flight whose date is not specified, such as "I'd like to fly a round trip from A to B and return on the same day.", are currently considered to be uninterpretable.

2.3 Fares.

2.3.1 A "one-way" fare is a fare for which round_trip_required = "NO".

2.3.2 A "round-trip" fare is a fare with a non-null value for fare.round_trip_cost.

2.3.3 The "cheapest fare" means the lowest one-direction fare.

2.3.4 A "coach" fare is one whose fare_basis.class_type = "COACH". Similarly, the fare modifiers "first class", "business class", and "thrift class" refer to values of the fare_basis.class_type field.

2.3.5 A reference to ranking of fares, e.g. "fares that are Y class or better", will be interpreted as a reference to the rank of the associated booking class (class_of_service.rank).

2.3.6 A "discounted fare" is one whose fare_basis.discounted = "YES".

2.3.7 Both "excursion fare" and "special fare" mean fares with non-null restriction codes (fare.restriction_code).

2.3.8 A "cheap" fare is an economy fare that is not class Y; that is, with economy = "YES" and booking_class NOT = "Y" in the fare_basis table.

2.3.9 For requests for airline fares, it will always be permissible to also include flights having those fares in the answer.

2.3.10 Restrictions on fares are not to be applied implicitly in answering queries. For example, a request for fares for a flight on a date near the session date will not omit those fares whose advance purchase restriction cannot be met.

2.4 Times.

2.4.1 The normal answer to otherwise unmodified "when" queries will be a time of day, not a date or a duration.

2.4.2 The answer to queries like "On what days does flight X fly" will be a list of days.day_name fields.

2.4.3 Time expressions (other than the "eight hundred hours" style) with a number from 1 to 12 in the hour position which don't specify "a.m." or "p.m.", such as "8:50", are ambiguous and may be interpreted as either a.m. or p.m.

2.4.4 Periods of the day.

The following table gives precise interpretations for some vague terms referring to time periods. The time intervals given include the end points. Items flagged with "*" are in the current (rdb3.3) database interval table. Each period is a contiguous block of time, so when an END TIME is less than a corresponding BEGIN TIME, it refers to the time on the next day.

            PERIOD       BEGIN TIME   END TIME
           morning*          0000       1200
           afternoon*        1200       1800
           evening*          1800       2200
           day*               600       1800
           night*            1800        600
           early morning*    0000        800
           mid-morning*       800       1000
           late morning*     1000       1200
           early afternoon*  1200       1400
           mid-afternoon*    1400       1600
           late afternoon*   1600       1800
           midday*           1100       1500
           early [flight]    0000       1000
           late [flight]     2000       0300
           breakfast time    0600       0900
           lunch time        1100       1400
           dinner time       1700       2000
           late night        2200       0300
           A.M.              0000       1159
           P.M.              1200       2359

2.4.5 "Earliest", when the relevant period of time is not further specified, will be taken to mean "earliest on any day (after midnight)", as opposed to earliest after the time of utterance or earliest after the beginning of a day (that is, 6 a.m.) as defined in the table above.

2.4.6 Interval endpoints.

Here is a list of a number of terms for time intervals, showing whether or not the end points of the intervals are to be including in the interval. The "periods of the day" referred to here are shown explicitly in 2.4.4.

                TERM        ENDPOINTS?
             before T1          N
             after T1           N
            between T1 and T2   Y
            arriving by T1      Y
            departing by T1     Y
            periods of the day  Y
2.4.7 "Noon" means strictly 1200 hours.

2.4.8 Flights "arriving on X": if a flight's arrival time is greater than its departure time, it arrives on the same day it leaves; otherwise, it arrives the next day.

2.4.9 When a day or date modifier is used with a period of the day that overlaps a day/date boundary, as in "Wednesday night", the period referred to is the period beginning on that day/date, not ending on it.

2.4.10 A date without a specified year will be interpreted as being in the period from and including the date of utterance up to but not including a year from the date of utterance. For instance, "March third", when uttered on a March third, must mean the date of the session that includes the utterance, not a year later. (Note that the year may be specified indirectly via allowable prior context or certain time phrases like "last March third" or "next March third".)

2.5 Units.

All units will be the same as those implicit in the database (e.g. feet for aircraft.wing_span, but miles for aircraft.range_miles, durations in minutes).

2.6 Meals.

2.6.1 For purposes of determining flights "with meals/meal service", snacks will count as a meal.

2.6.2 "list the types of meal" should produce one tuple per meal, not a single meal_code string.

2.7 "With" clauses.

"with"-modification clauses: "show me all the flights from X to Y with their fares" will require the identification of both flights and their fares (so if there are 2 flights, each with three fares, the answer will have 6 tuples, each with at least the flight_code and fare_code). In general, queries asking for information from two or more separate tables in the database will require the join of fields that would identify each table entry separately.

2.8 "class".

"Class" refers to the contents of the "booking_class" field in the "class_of_service" table.

2.9 Meaning requests.

2.9.1 With the particular exceptions noted below, requests for the "meaning" of something will only be interpretable if that thing is a code with a canned decoding definition in the database. In case the code field is not the key field of the table, information should be returned for all tuples that match on the code field. Here are the things so defined, with the fields containing their decoding:

         Table          Code Field       Decoding Field
       aircraft         aircraft_code    aircraft_description 
       airline          airline_code     airline_name
       airport          airport_code     airport_name
       city             city_code        city_name
       class_of_service booking_class    class_description
       code_description code             description
       days             days_code        day_name
     equipment_sequence aircraft_code_sequence
       fare_basis       fare_basis_code  booking_class, class_type,
                                           premium, economy, discounted,
                                           night, season, basis_days
       food_service     meal_code        meal_description
       interval         period           begin_time,
       month            month_number     month_name
       state            state_code       state_name
       time_zone        time_zone_code   time_zone_name
       restriction      restriction_code advance_purchase, stopovers, 
                                           minimum_stay, maximum_stay,
                                           application, no_discounts 
* - The aircraft_description field that answers this type of query is found in the aircraft table, joined with the equipment_sequence table on the aircraft_code fields.

2.9.2 "Define X" and "explain X" should be interpreted the same as "What does X mean."

2.9.3 Requests for "more information about X" or "everything about X" will be answered with ALL columns from the database table for X, if X refers to a type of entity that corresponds to a table; otherwise the request will be classified as unanswerable.

2.9.4 "What kind of X is Y" queries, where Y is clearly a kind of X, will be interpreted as equivalent to "what does Y mean?", where Y is a code field value for the table referred to by X.

2.10 Meaning/enumeration ambiguity. Vague queries such as "What is/are X?", "Describe X.", or "Give me information about X." may be treated as meaning questions ("What does 'X' mean?"), or as requests for enumeration ("list X"), or ambiguous between the two, according to the judgement of the annotators as to what the speaker meant in a particular case.

2.11 Yes/no questions.

2.11.1 Queries that are literally yes-or-no questions are considered to be ambiguous between interpretation as a yes-or-no question and interpretation as the corresponding wh-question. For example, "Are there flights from Boston to Philly?" may be answered by either a boolean value ("YES/TRUE/NO/FALSE") or a table of flights from Boston to Philadelphia. Annotators creating reference answers will provide what they feel, as native English speakers, are the entities being asked about.

2.11.2 A yes-or-no question about the existance of a difference, such as "Is there a difference in advance purchase requirements between AP/80 and AP/57?" has a corresponding wh-question whose treatment is described below in 2.20.

2.11.3 Presupposition failures (excluding the number presupposition specifically allowed in 2.25) on yes/no questions make the yes/no interpretation unreasonable, so that only the wh-question interpretation, giving a tabular answer, exists.

2.11.4 A yes-or-no question about a feature of individual members of a set, phrased as a question about the set as a whole, as in "Are those flights one-way?", can be answered by a table of only the positive ("yes") instances, e.g. the subset of those flights that are one-way. In other words, "Are those Xs Y?" has the corresponding wh-question "Which Xs are Y?". Note that in questions like this, there is a presupposition that all the members of the set have the same value for the feature, and if this is not true, the yes/no question interpretation is unreasonable and will be ignored, according to principle 2.11.3 above, although the wh-question interpretation is still valid.

2.12 A city and an airport will be considered "near" (or "nearby") each other iff the city is served by the airport, and two cities will be considered "near" (or "nearby") each other iff there is an airport that serves them both.

2.13 When it is clear that an airline is being referred to, the term "American" by itself will be taken as unambiguously referring to American Airlines.

2.14 References to "the city" or "downtown" are ambiguous, and may be interpreted as referring to any city that seems reasonable.

2.15 When a query refers to an aircraft type by giving an identifying code, such as "737", the answer consists of all the aircraft that have that code in either the aircraft.code field or the aircraft.basic_type field or both. If a manufacturer (such as "BOEING") is also given, it must match the value of aircraft.manufacturer.

2.16 Utterances whose answers require arithmetic computation are not now considered to be interpretable; this does not apply to arithmetic comparisons, including computing the maximum or minimum value of a field, or counting elements of a set of tuples.

2.17 Tolerances.

When a query is hedged by use of "around X" or "about X", where X is numeric, as in "Show me the flights that leave around 4 o'clock", these plus-or-minus tolerances should be used on the different types of quantities:

       Times ............. 30 minutes
       Dates ............. zero
       Days of Week ...... zero
       Other Quantities .. 10%
       (such as fares)
If none of these rules apply, then the tolerance is zero. Time periods defined by these tolerances include the end points.

2.18 Constraints on Flights and Fares.

When a constraint such as date, day, or class is given in a query that asks for flights and fares, as in

"Flights available Saturdays along with their fares"
"Flights and their fares on 7/23"
"Flights on 7/23", "What are the fares?"
the query must be interpreted with the constraint applying to both flights and fares. (Given the current structure of our database, if the constraint is taken to apply only to flights, fares will be returned that are not available on the date/day mentioned in the question.)

2.19 Prices and costs will be treated as asking for fares rather than as asking for dollar amounts.

2.20 Queries asking for a "difference" may on some occasions be interpreted as requesting a general comparison between attributes of entities, and in these cases, they may be answered by listing the entities in question along with the values for the attribute in question. For example, the query "What is the difference in advance purchase requirements between AP/80 and AP/57?" may be answered by listing the two fare classes along with their advance purchase requirements.

On other occasions, "difference" specifically means arithmetic subtraction, as in "What is the time difference between Dallas and Denver?", and the query will be counted as an unanswerable arithmetic query (see 2.16 above).

If an instance of "difference" seems ambiguous between the two interpretations, only the nonarithmetic, general comparison interpretation will be considered. (Note that this is an exception to the general rule that the existance of a valid but unanswerable interpretation make the whole query unanswerable.)

2.21 With certain Holiday exceptions given immediately below, questions about the calendar, such as "What is the date of next Thursday?", are not now considered to be interpretable.

2.22 References to these holidays are to be interpreted as referring to the dates shown:

         Holiday     Date
        Christmas    12/25
        New Year's   01/01
        4th of July  07/04
2.23 In the answer to a request for a set of counts, such as "How many flights on each airline have business class?", inclusion of the tuples for which the count is 0 is optional.

2.24 The measure of the size of an airplane is its seating capacity.

2.25 All presuppositions about the number of answers (either existence or uniqueness) will be ignored.

2.26 "Ticket", in addition to its literal meaning, is basically ambiguous between "fare" and "flight". For example, in "Show me tickets from Dallas to Boston serving lunch.", it would have to be taken to mean "flight", while in "How much is a ticket from Dallas to Boston?", it must refer to fares.

2.27 The terms "Dallas Fort Worth" and "Baltimore Washington" are basically ambiguous and may refer to either the cities named or the airports DFW and BWI.

2.28 The set of possible interpretations for a code is limited to the fields in the database that the code actually occurs in, but it may be additionally restricted by context.

Note that this implies that if there are just two code fields in the database having the value "Y", then the query "What is code Y?" with no contextual restriction is just two ways ambiguous and perfectly well answerable.

2.29 In response to a request for "departures" or "arrivals", the minimal answer should contain the identifications of the flights composing the answer, and the maximal answer may contain, in addition, the departure or arrival times and the origin or destination airports. This is not to be taken as defining what "departure" and "arrival" mean.

File pofi-guidelines.doc Version 1.0 6/19/92

Guidelines for the Principles of Interpretation

The principles of interpretation as stated allow some leeway for annotators and adjudicators to use their own best judgement in formulating interpretations of questions. For example, there is a general rule that context may make one interpretation of an ambiguous query unreasonable, and therefore ignorable; but deciding in particular cases that context makes an interpretation unreasonable is largely left up to human judgement. This document contains individual cases that exemplify judgements made by annotators and adjudicators, along with some general rules-of-thumb, to serve as guidelines for system builders who want to build the same kind of judgement into their systems. When possible, cases will be keyed to the principle that seems most relevant; at the bottom of this document, the section "Miscellaneous Cases" contains some isolated cases that haven't yet yielded a general rule-of-thumb.

1.1. Only reasonable interpretations will be used.

Guideline: if there are several possible interpretations of an utterance, and the subject should have known that one of them would produce a null answer, that one is probably unreasonable.

More generally, if one interpretation violates a pragmatic principle or maxim, such as "Be relevant", and another interpretation is available that has no such violations, then the first is probably unreasonable and may therefore be ignored.

Case #1: (hypothetical)

       A1: [a list of flights]
       Q2: "Which ones serve breakfast?"
       A2: [none of them]
       Q3: "Which ones have stops?"
"Ones" in Q3 is basically ambiguous between referring to a.) the list of flights in A1; and b.) the subset of them that serve breakfast. If there had been some flights which served breakfast, both interpretations would have been reasonable; but because the b.) interpretation would yield a null answer, and the speaker must have known this, it was rejected as unreasonable.

2.2.5 References to legs for connecting flights are basically ambiguous between referring to the leg or to the entire flight.

Guideline: references to second or later legs of a flight are almost certainly not reasonably interpreted as referring to the entire flight.

Because some annotators believe that a case might conceivably arise in which reference to a second leg would reasonably be interpreted as referring to the whole flight, this rule has not been made absolute. In fact, no such case has yet (5/15/92) occurred in real data. Here is their hypothetical example of such a case, in which Q4 might be taken as referring to the whole flight:

      Q1: "Show me flights from Boston to San Francisco."
      A1: [a table of flights]
      Q2: "What United Airlines flights leave in the afternoon?"
           --------- ---------- ---- --- ----- ------ --------------------
           106376    DAILY      BOS  SFO  1745   2127 UA93
           393553    DAILY      BOS  OAK  1720   2205 UA281/UA673
           420911    DAILY      BOS  SFO  1720   2214 UA281/UA297

      Q3: "What is the fare for the flight leaving at 5:20?"
      Q4: "What is the fare for United Airlines, flight 297?"

2.11.1 Queries that are literally yes-or-no questions are considered to be ambiguous between interpretation as a yes-or-no question and interpretation as the corresponding wh-question.

Guideline: if it seems clear that the subject was requesting information FROM the database, and not information about the system's ABILITY to provide that information, then a yes/no interpretation is unreasonable.

Some hypothetical cases where the yes-or-no interpretation is unreasonable are:

"Can you show me the morning flights?"
"Can you tell me if there are morning flights?"


1. The expression "... flight 915 ..." can't reasonably be interpreted as the flight that leaves or arrives at 9:15, as in utterance qx0061ss:

          FARE   ONE  ROUND
          ---    ----  ----  ---- --- ----- ------ --------------------
           F     $428  $856   ATL BWI  9:15  10:55 EA202
           Y     $286  $572   ATL BWI  9:15  10:55 EA202
           F     $428  $856   ATL BWI 10:02  11:45 DL1204
           Y     $286  $572   ATL BWI 10:02  11:45 DL1204
           .       .     .     .   .    .      .     .
           .       .     .     .   .    .      .     .
           .       .     .     .   .    .      .     .

      Q6: "List the fares for Eastern's flight nine fifteen."