Corpus Title: KAIROS Phase 1 Quizlet LDC Catalog-ID: LDC2025T11 Authors: Song Chen, Ann Bies, Justin Mott, Christopher Caruso, Jennifer Tracey, Stephanie Strassel 1.0 Introduction The KAIROS Phase 1 Quizlet Corpus contains the English and Spanish source data (including text, video and images) and annotations used for pre-evaluation research and system development during Phase 1 of the DARPA KAIROS Program. The Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS) Program aims to develop technology capable of understanding and reasoning about complex real-world events in order to provide actionable insights to end users. KAIROS systems utilize formal event representations in the form of schema libraries that specify the steps, preconditions and constraints for an open set of complex events; schemas are then used in combination with event extraction to characterize and make predictions about real-world events in a large multilingual, multimedia corpus. KAIROS quizlets are a series of narrowly defined tasks designed to explore specific evaluation objectives, enabling KAIROS system developers to exercise individual system components on a small data set prior to the full program evaluation. This release contains the complete set of quizlet data used in Phase 1 of KAIROS. There were four Phase 1 quizlets, with each quizlet building upon the data and results from the previous quizlets. The quizlets focus on two real-world Complex Events (CEs) within the Improvised Explosive Device (IED) bombing scenario: CE1001 (2018 Caracas drone attack) and CE1002 (Utah High School backpack bombing). Quizlets include both CE-relevant documents and off-topic distractor documents in English and Spanish, covering text, image and video sources. Trained annotators create gold standard reference annotations by labeling the scenario-relevant events and relations for each document, resulting in a structured representation of the temporally-ordered events, relations and arguments necessary to fully express the focus CEs. Quizlets may also include a reference knowledge graph, known as Graph G, which contains a subset of the manually labeled events and relations and constitutes the ground truth reference CE for evaluation. Some manually labeled events and relations are intentionally omitted from Graph G in order to evaluate schema instantiation and event prediction capabilities. Quizlet 1 was intended to introduce performers to the KAIROS evaluation tasks and did not require any data or annotation, so there is no Quizlet 1 data in this release. Quizlet 2 focused on schema generation and instantiation using the KAIROS program ontology. Data for Quizlet 2 includes source documents in the IED scenario, including some documents on IED bombing incidents. Quizlet 2 did not include manual annotation. Quizlet 3 focused on graph G generation using manual annotation. Data for Quizlet 3 includes source documents and preliminary manual annotation, along with the graph G that was generated for each CE. The manual annotation for this quizlet includes the labeled events and relations (plus their arguments) needed to understand the CE as a whole. Labeled events are also temporally ordered in start order. Quizlet 4 served as a dry run for the full Phase 1 evaluation, exercising the entire evaluation pipeline. Data for Quizlet 4 starts with the data from Quizlet 3, and includes additional source documents and additional manual annotation, along with an updated graph G. New manual annotation for this quizlet includes the annotation of additional events, relations, and entities present in the new source documents, the temporal ordering of all labeled events, as well as revisions and improvements to the annotation from Quizlet 3. 2.0 Directory Structure and Content Summary 2.1 Directory Structure The directory structure and contents of the package are summarized below – paths shown are relative to the base (root) directory of the package: ./data/source -- source data in subdirectories by data type ./data/annotation/quizlet_3 -- contains annotation data for Quizlet 3 ./data/annotation/quizlet_4 -- contains annotation data for Quizlet 4 ./data/graphG/quizlet_3 -- contains graph G for Quizlet 3 ./data/graphG/quizlet_4 -- contains graph G for Quizlet 4 ./docs -- documentation for source data and annotations ./tools/ -- software for LTF data manipulation The "./data/source" directory has a separate subdirectory for each of the following data types, and each directory contains one or more data files of the given type; the list below shows the directory and file-extension strings used for the data files of each type: gif/ -- contains "*.gif.ldcc" (image data) jpg/ -- contains "*.jpg.ldcc" (image data) mp4/ -- contains "*.mp4.ldcc" (video data) png/ -- contains "*.png.ldcc" (image data) svg/ -- contains "*.svg.ldcc" (image data) ltf/ -- contains "*.ltf.xml" (segmented/tokenized text data) psm/ -- contains "*.psm.xml" (companion to ltf.xml) Data types in the first group (image and video data) consist of original source materials presented in "ldcc wrapper" file format (see section 4.2 below). The latter group (ltf and psm) are created by LDC from source HTML data, by way of an intermediate XML reduction of the original HTML content for "root" web pages (see section 4.1 for a description of the process, and section 5 for details on the LTF and PSM file formats). The data files use 9-character file-IDs. For example: svg/JC002YBYQ.svg.ldcc (The "ldcc" file format is explained in more detail in section 4.2 below.) 2.2 Source Data A total of 30 root web pages were collected and processed, yielding 29 text data files, 216 image files, and 5 video files present in the corpus. 2.3 Annotation The table below summarizes the total number of documents and amount of annotation included in the corpus: total_ce - total complex events subject to data collection and annotation total_doc_src - root documents collected and processed total_evt - number of events annotated total_rel - number of relations annotated total_arg - number of arguments annotated quizlet | total_ce | total_doc_src | total_evt | total_rel | total_arg | 2 | NA | 10 | NA | NA | NA | 3 | 2 | 16 | 64 | 45 | 318 | 4 | 2 | 20 | 68 | 58 | 386 | 2.4 Graph G Graph Gs for Quizlets 3-4 are included in this release in the ./data/graphG/quizlet_{3,4} directories. Quizlet 2 did not include a graph G, and so does not have a corresponding graphG directory. Graph G is in JSON format. However, for convenience only, graph G information is also included in human-readable Excel (.xlsx) files. Note that the presentation of the graph G is slightly different between the .xlsx files for the two quizlets: For Quizlet 3, the content of the Excel files includes only the events and arguments that are in the graph G for each CE in Quizlet 3. For Quizlet 4, the graph G information is present in the highlighting in the Excel files, and the content of the Excel files is otherwise identical to the content in the annotation tables in ./data/annotation/quizlet_4. The events and relations that appear in the JSON version of the graph G are highlighted in yellow in the Excel files for each CE in Quizlet 4. 2.5 Software Tools Software tools in this release can be found in ./tools/ltf2txt/ A data file in ltf.xml format (for source data in each quizlet) can be conditioned to recreate exactly the "raw source data" text stream (the rsd.txt file) from which the LTF was created. The tools described here can be used to apply that conditioning, either to a directory or to a zip archive file containing ltf.xml data. In either case, the scripts validate each output rsd.txt stream by comparing its MD5 checksum against the reference MD5 checksum of the original rsd.txt file from which the LTF was created. (This reference checksum is stored as an attribute of the "DOC" element in the ltf.xml structure; there is also an attribute that stores the character count of the original rsd.txt file.) Each script contains user documentation as part of the script content; you can run "perldoc" to view the documentation as a typical unix man page, or you can simply view the script content directly by whatever means to read the documentation. Also, running either script without any command-line arguments will cause it to display a one-line synopsis of its usage, and then exit. ltf2rsd.perl -- convert ltf.xml files to rsd.txt (raw-source-data) 3.0 Annotation Approach The Phase 1 quizlets focus on two real-world incidents or Complex Events (CEs) in the Improvised Explosive Device (IED) bombing scenario: CE1001 (2018 Caracas drone attack) and CE1002 (Utah High School backpack bombing). Each CE has a separate source data set, where on-topic documents in the data set are relevant to the CE and suitable for both human annotation and system evaluation. The data sets are multilingual (English and Spanish) and multimedia (text, video, image). Source data format and usage are described in sections 4 and 5 below. The goal of annotation for the quizlet data was to develop a temporally ordered list of the unique events and relations (plus their arguments) necessary to fully account for the information conveyed about the CE in the data, to serve as ground truth for system evaluation. The data set of source documents is manually labeled for the scenario-relevant events the data set contains, resulting in a structured representation of the temporally-ordered events, relations and arguments necessary to fully express the CE. KAIROS evaluation utilizes the concept of Event Primitives, which defines the minimum level of event granularity that KAIROS is concerned with. Event Primitives (EPs) can be simply understood as manually labeled or system-extracted events within the KAIROS program. To support holistic CE understanding across a multilingual and multimedia data set, we do not label every mention of every individual event, relation and entity discussed in the document set for each CE; instead, we create exactly one annotation frame for each unique event or relation present in the data. This annotation of all the relevant information for a given event or relation, is not a cluster of individual event/relation mentions. Instead, it is an abstract structured representation of the event or relation as a whole, without any individual mentions or links to specific spans of document provenance. Timestamp temporal information is included as part of the annotation for each event. Inference and logical reasoning are permitted for arguments of events and for temporal information, but events and relations must be explicitly mentioned in the data to be labeled in the reference annotation. Events and relations (including their arguments) that comprise each CE are annotated on the CE’s data set as a whole, and thus do not include document provenance such as text offsets or video timestamps. Events and relations that are mentioned in multiple documents or in multiple media types (e.g., text and video) are listed only once, with their arguments and attributes reflecting the information available in the data set as a whole. Temporal ordering of all labeled events within each CE is provided by specifying the start order for each event, relative to other events for the CE. The event start may be specified as “exactly,” “before” or “after” some numbered order, or may be specified as unknown. Before and after may be combined. More than one event can have the same start order relative to other events. Argument entities with different annotation tags (for example, FAC and ORG) that referred to the same entity (for example, University of Pennsylvania) were coreferenced, so arguments that refer to the same real-world entity have the same entity_id within each CE. Detailed annotation guidelines are included in ./docs/KAIROS_Phase1_Eval_AnnotationGuidelines_v1.11.pdf. The annotation tag set is included in ./docs/KAIROS_Annotation_Tagset_Phase_1_V3.0.xlsx. See section 6.4 for a description of the tabs and fields in the tag set Excel file. For further information about annotation approaches in the KAIROS program, refer to Bies et al. (2024). 4.0 Source Data Preparation Trained KAIROS annotators searched the web for suitable quizlet documents; these documents were first harvested from various sources using an automated system developed by LDC, and then processed to produce a standardized format for use in downstream tasks. 4.1 Treatment of original HTML text content All harvested HTML content was initially converted from its original form into a relatively uniform XML format; this stage of conversion eliminated irrelevant content (menus, ads, headers, footers, etc.), and placed the content of interest into a simplified, consistent markup structure. The "homogenized" XML format then served as input for the creation of a reference "raw source data" (rsd) plain text form of the web page content; at this stage, the text was also conditioned to normalize white-space characters, and to apply transliteration and/or other character normalization, as appropriate to the given language. This processing creates the ltf.xml and psm.xml files for each harvested "root" web page; these file formats are described in more detail in section 5 below. 4.2 Treatment of non-HTML data types: "ldcc" file format To the fullest extent possible, all discrete resources referenced by a given "root" HTML page (style sheets, javascript, images, video, audio and other media files, etc.) are stored as separate files of the given data type, and assigned separate 9-character file-IDs (the same form of ID as is used for the "root" HTML page). In order to present these attached resources in a stable and consistent way, we developed a "wrapper" or "container" file format, which presents the original data as-is, together with a specialized header block prepended to the data. The header block provides metadata about the file contents, including the MD5 checksum (for self-validation), the data type and byte count, url, and citations of source-ID and parent (HTML) file-ID. The LDCC header block always begins with a 16-byte ASCII signature, as shown between double-quotes on the following line (where "\n" represents the ASCII "newline" character 0x0A): "LDCc \n1024 \n" Note that the "1024" on the second line of the signature represents the exact byte count of the LDCC header block. (If/when this header design needs to accommodate larger quantities of metadata, the header byte count can be expanded as needed in increments of 1024 bytes. Such expansion does not arise in the present release.) Immediately after the 16-byte signature, a YAML string presents a data structure comprising the file-specific header content, expressed as a set of "key: value" pairings in UTF-8 encoding. The YAML string is padded at the end with space characters, such that when the following 8-byte string is appended, the full header block size is exactly 1024 bytes (or whatever size is stated in the initial signature): "endLDCc\n" In order to process the content of an LDCC header: - read the initial block of 1024 bytes from the *.ldcc data file - check that it begins with "LDCc \n1024 \n" and ends with "endLDCc\n" - strip off those 16- and 8-byte portions - pass the remainder of the block to a YAML parser. In order to access the original content of the data file, simply skip or remove the initial 1024 bytes. 5.0 Overview of XML Data Structures 5.1 PSM.xml -- Primary Source Markup Data The "homogenized" XML format described above preserves the minimum set of tags needed to represent the structure of the relevant text as seen by the human web-page reader. When the text content of the XML file is extracted to create the "rsd" format (which contains no markup at all), the markup structure is preserved in a separate "primary source markup" (psm.xml) file, which enumerates the structural tags in a uniform way, and indicates, by means of character offsets into the rsd.txt file, the spans of text contained within each structural markup element. For example, in a discussion forum or weblog page, there would be a division of content into the discrete "posts" that make up the given thread, along with "quote" regions and paragraph breaks within each post. After the HTML has been reduced to uniform XML, and the tags and text of the latter format have been separated, information about each structural tag is kept in a psm.xml file, preserving the type of each relevant structural element, along with its essential attributes ("post_author", "date_time", etc.), and the character offsets of the text span comprising its content in the corresponding rsd.txt file. 5.2 LTF.xml -- Logical Text Format Data The "ltf.xml" data format is derived from rsd.txt, and contains a fully segmented and tokenized version of the text content for a given web page. Segments (sentences) and the tokens (words) are marked off by XML tags (SEG and TOKEN), with "id" attributes (which are only unique within a given XML file) and character offset attributes relative to the corresponding rsd.txt file; TOKEN tags have additional attributes to describe the nature of the given word token. The segmentation is intended to partition each text file at sentence boundaries, to the extent that these boundaries are marked explicitly by suitable punctuation in the original source data. To the extent that sentence boundaries cannot be accurately detected (due to variability or ambiguity in the source data), the segmentation process will tend to err more often on the side of missing actual sentence boundaries, and (we hope) less often on the side of asserting false sentence breaks. The tokenization is intended to separate punctuation content from word content, and to segregate special categories of "words" that play particular roles in web-based text (e.g. URLs, email addresses and hashtags). To the extent that word boundaries are not explicitly marked in the source text, the LTF tokenization is intended to divide the raw-text character stream into units that correspond to "words" in the linguistic sense (i.e. basic units of lexical meaning). 6.0 Documentation The ./docs folder contains documentation relevant to the data included in this release, along with DTD files for the "ltf" and "psm" xml file formats. 6.1 Root Pages and Child Assets The relationship between child assets and root HTML pages (parent_uid) is described in ./docs/parent_children.tab. In the following, the term "asset" refers to any single "primary" data file of any given type. Each asset has a distinct 9-character identifier. If two or more files appear with the same 9-character file-ID, this means that they represent different forms or derivations created from the same, single primary data file (e.g., this is how we mark corresponding LTF.xml and PSM.xml file pairs). Data scouting, annotation and related metadata are all managed with regard to a set of "root" HTML pages (harvested by the LDC for a specified set of events); therefore the ./docs/parent_children.tab and ./docs/document_profile.tab tables make reference to the asset-IDs assigned to those root pages. However, the present release does not include the original HTML text streams, or any derived form of data corresponding to the full HTML content. As a result, the "root" asset-IDs cited in tables are not to be found among the inventory of data files presented in the "./data/source" directory. Each root asset is associated with one or more "child" assets (including images, media files, style sheets, text data presented as ltf.xml, etc.); each child asset gets it own distinct 9-character ID. The root-child relations are provided in "parent_children.tab" table, and as part of the LDCC header content in the various "wrapped" data file formats (as listed in section 2.1). Each parent UID is represented by the combination of child_uid and child_asset_type (columns 2 and 3). The columns are tab-delimited and the initial line of the file provides the column labels as shown below: Col.# Content 1. parent_uid (the parent UID associated with the doc URL) 2. child_uid 3. child_asset_type (e.g. ".jpg.ldcc") 4. url 5. unwrapped_md5 (md5 checksum of the asset file without the ldcc wrapper) 6. wrapped_md5 (md5 checksum of the .ldcc-wrapped asset file) 7. download_date (download date of asset) 8. content_date (creation date of asset, or n/a) Notes: - Because ltf and psm files have the same "child" uid and differ only in the file extension (.ltf.xml or .psm.xml), only the ltf files are listed in the parent_children.tab document. - The URL provided for each .ltf.xml entry in the table is the "full-page" URL for root document associated with the "parent_uid" value. (For other types of child data -- images and media -- the "url" field contains the specific url for that specific piece of content.) - Some child_uids (for images or videos) may appear multiple times in the table, if they were found to occur identically in multiple root web pages. - The content_date is obtained for the parent document from the process that extracts the text (ltf) child asset. This date therefore appears only for ltf rows in the table, but can be considered to apply to the full parent document. 6.2 Document Profile Information about the source data in the package is provided in ./docs/document_profile.tab, including the source UID, the CE ID the document was scouted for, the language of the document, whether the document is CE relevant or a distractor document, and which quizlet(s) the document is used for. Col.# Content 1. parent_uid - parent_uid identifier for the document 2. ce_id - the ID of the CE (e.g. "ce2013") 3. ce_name - the name of the CE 4. language - language of the document 5. status - whether the document is on topic for the CE or a distractor document 6. quizlet_2 - whether the document is used in Quizlet 2: ta1 indicates the source document is used for the ta1 task, while ta2 indicates the source document is used for the ta2 task. EMPTY_NA indicates the document is not used for the quizlet. 7. quizlet_3 - whether the document is used in Quizlet 3: Yes indicates the source document is used for the quizlet. EMPTY_NA indicates the document is not used for the quizlet. 8. quizlet_4 - whether the document is used in Quizlet 3: Yes indicates the source document is used for the quizlet. EMPTY_NA indicates the document is not used for the quizlet. 6.3 Annotation Guidelines The ./docs directory also includes guidelines for annotation. ./docs/KAIROS_Phase1_Eval_AnnotationGuidelines_v1.11.pdf - guidelines for quizlet and eval data annotation 6.4 Annotation Tag Set The ./docs directory also includes documentation about the annotation tagset used in the annotation of the quizlet data, which was also the official tagset for KAIROS Phase 1: ./docs/KAIROS_Annotation_Tagset_Phase_1_V3.0.xlsx - annotation tag set (ontology) for KAIROS Phase 1 This annotation tag set (also known as the annotation ontology) was used for the annotation of events, relations, and their argument entities. The tag set includes type, subtype, sub-subtype, attribute, and temporal start/end timestamp specifications. Please refer to section 3 of this README and the annotation guidelines (./docs/KAIROS_Phase1_Eval_AnnotationGuidelines_v1.11.pdf) for additional information about the annotation procedure. The tag set is included as an Excel file, with the following five tabs: - events (the labels, output values, definitions, templates, arg labels, and arg constraints for events) - entities (the labels, output values, and definitions for entities) - relations (the labels, output values, definitions, templates, arg labels, and arg constraints for relations) - attributes (the labels, definitions, and output values for attributes of events, relations, and arguments) - temporal startend (the start and end type labels, output values, and definitions for temporal timestamp annotation of events and relations, along with the output format for times) The initial line of each tab provides the column labels as shown below. 6.4.1 events Col.# Content 1. AnnotIndexID (a unique ID for the tag in the format LDC_KAIROS_evt_NNN) 2. Type (the human-readable type label) 3. Output Value for Type (the output value for the type as it appears in the annotation tables) 4. Subtype (the human-readable subtype label) 5. Output Value for Subtype (the output value for the subtype as it appears in the annotation tables) 6. Sub-subtype (the human-readable sub-subtype label) 7. Output Value for Sub-subtype (the output value for the sub-subtype as it appears in the annotation tables) 8. Definition (natural language definition of the full tag) 9. Template (human-readable templatic representation of the event and its arguments) 10. arg1 label (human-readable role label for argument 1) 11. Output value for arg1 (the output value for the arg1 role label as it appears in the annotation tables) 12. arg1 type constraints (the list of entity types that may fill the arg1 role, including whether any event or relation could fill the role) 13. arg2 label (human-readable role label for argument 2) 14. Output value for arg2 (the output value for the arg2 role label as it appears in the annotation tables) 15. arg2 type constraints (the list of entity types that may fill the arg2 role, including whether any event or relation could fill the role) 16. arg3 label (human-readable role label for argument 3) 17. Output value for arg3 (the output value for the arg3 role label as it appears in the annotation tables) 18. arg3 type constraints (the list of entity types that may fill the arg3 role, including whether any event or relation could fill the role) 19. arg4 label (human-readable role label for argument 4) 20. Output value for arg4 (the output value for the arg4 role label as it appears in the annotation tables) 21. arg4 type constraints (the list of entity types that may fill the arg4 role, including whether any event or relation could fill the role) 22. arg5 label (human-readable role label for argument 5) 23. Output value for arg5 (the output value for the arg5 role label as it appears in the annotation tables) 24. arg5 type constraints (the list of entity types that may fill the arg5 role, including whether any event or relation could fill the role) 25. arg6 label (human-readable role label for argument 6) 26. Output value for arg6 (the output value for the arg6 role label as it appears in the annotation tables) 27. arg6 type constraints (the list of entity types that may fill the arg6 role, including whether any event or relation could fill the role) Notes: - The annotation for events and relations used a three-level annotation tag, which included a high-level type, a more specific subtype under each type, and a finer-grained sub-subtype under each subtype. The three levels together comprise the annotation tag for the event or relation. - A sub-subtype of "unspecified" indicates that none of the fine-grained sub-subtypes under the subtype is appropriate for the annotated event or relation in the context of the document. This may be either because the document context does not support a finer-grained reading (so the higher level subtype is the most specific reading in the document context), or it may be because the available fine-grained sub-subtypes are not applicable (in which case, the higher level subtype is the most specific tag available). - Each event has a defined set of argument roles, and only the defined roles are available for annotation. The maximum number of roles for an event in this tag set is six. - Argument constraints for each argument role list the entity types that may fill the argument role, using the entity type output values, along with whether an event (any event type) or relation (any relation type) may fill the argument role. 6.4.2 entities Col.# Content 1. AnnotIndexID (a unique ID for the tag in the format LDC_KAIROS_ent_NNN) 2. Type (the human-readable type label) 3. Output Value for Type (the output value for the type as it appears in the annotation tables) 4. Definition (natural language definition of the tag) Notes: - The annotation for entities used only a single type label for each entity. 6.4.3 relations Col.# Content 1. AnnotIndexID (a unique ID for the tag in the format LDC_KAIROS_rel_NNN) 2. Type (the human-readable type label) 3. Output Value for Type (the output value for the type as it appears in the annotation tables) 4. Subtype (the human-readable subtype label) 5. Output Value for Subtype (the output value for the subtype as it appears in the annotation tables) 6. Sub-subtype (the human-readable sub-subtype label) 7. Output Value for Sub-subtype (the output value for the sub-subtype as it appears in the annotation tables) 8. Definition (natural language definition of the full tag) 9. Template (human-readable templatic representation of the relation and its arguments) 10. arg1 label (human-readable role label for argument 1) 11. Output value for arg1 (the output value for the arg1 role label as it appears in the annotation tables) 12. arg1 type constraints (the list of entity types that may fill the arg1 role, including whether any event or relation could fill the role) 13. arg2 label (human-readable role label for argument 2) 14. Output value for arg2 (the output value for the arg2 role label as it appears in the annotation tables) 15. arg2 type constraints (the list of entity types that may fill the arg2 role, including whether any event or relation could fill the role) Notes: - The annotation for events and relations used a three-level annotation tag, which included a high-level type, a more specific subtype under each type, and a finer-grained sub-subtype under each subtype. The three levels together comprise the annotation tag for the event or relation. - A sub-subtype of "unspecified" indicates that none of the fine-grained sub-subtypes under the subtype is appropriate for the annotated event or relation in the context of the document. This may be either because the document context does not support a finer-grained reading (so the higher level subtype is the most specific reading in the document context), or it may be because the available fine-grained sub-subtypes are not applicable (in which case, the higher level subtype is the most specific tag available). - Each relation has a set of two defined argument roles, and only the defined roles are available for annotation. - Argument constraints for each argument role list the entities types that may fill the argument role, using the entity type output values, along with whether an event (any event type) or relation (any relation type) may fill the argument role. 6.4.4 attributes Col.# Content 1. Attribute Label (the human-readable attribute label) 2. Definition (natural language definition of the attribute) 3. Output Value for Attribute (the output value for the attribute as it appears in the annotation tables) Notes: - The rows in this tab are divided into sections for Event Attributes, Relation Attributes, and Argument Attributes for Arguments of Events, under the Attribute Label column. 6.4.5 temporal startend Col.# Content 1. Start/End Type Label (the human-readable type label) 2. Output Value for Start/End Type (the output value for the type as it appears in the annotation tables) 3. Definition (natural language definition of the temporal type) Notes: - The rows in this tab include an additional section showing the Output Format for Start/End Times under the Start/End Type Label column. 7.0 Annotation Output Tables Below are detailed descriptions for each annotation output table. The annotation tables can be found in the following directories in this release: ./data/annotation/quizlet_{3,4}/{ce1001,ce1002}/ 7.1 Event Primitives table: _events.tab This table has the following columns for event primitives: Col.# Content 1. eventprimitive_id - unique identifier assigned to each event primitive 2. description - natural language description of the event primitive 3. type - LDC annotation labels 4. subtype - LDC annotation labels 5. subsubtype - LDC annotation labels 6. attribute - Event attribute; comma separated if multiple 7. significance - critical or optional 8. start_datetime_type - one of starton, startbefore, startafter, startunk 9. start_datetime - format is YYYY-MM-DDTHH:MM:SS. Unknown value is represented as x 10. end_datetime_type - one of endon, endbefore, endafter, endunk 11. end_datetime - format is YYYY-MM-DDTHH:MM:SS. Unknown value is represented as x 7.2 Relations table: _relations.tab This table has the following columns for relations: Col.# Content 1. relation_id - unique identifier assigned to each relation 2. description - natural language description of the relation 3. type - LDC annotation labels 4. subtype - LDC annotation labels 5. subsubtype - LDC annotation labels 6. attribute - relation attribute; comma separated if multiple 7. significance - critical or optional 7.3 Arguments table: _arguments.tab This table has the following columns for the arguments of both event primitives and relations: Col.# Content 1. eventprimitive_id - unique identifier for the event primitive in which the argument fills a slot, EMPTY_NA if the argument slot is for a relation 2. relation_id - unique identifier for the relation in which the argument fills a slot, EMPTY_NA if the argument slot is for an event 3. description - natural language description of the argument 4. arg_id - unique identifier for each entity argument; if argument is an event, this matches the event primitive id 5. entity_id - Entities having the same id indicate entity coreference 6. arg_num - arg1, arg2 etc. associated with the arg role from the annotation tagset 7. slot_type - argument role from the annotation tagset, e.g. recipient, instrument 8. type - LDC annotation labels 9. subtype - LDC annotation labels 10. subsubtype - LDC annotation labels 11. attribute - argument attribute; comma separated if multiple 12. arg_inference - e.g. "notinferred" Note that annotation of whether an argument is inferred or not is added during Quizlet 4, so argument table doesn't have arg_inference field. 7.4 Temporal ordering annotation: _temporal.tab This table includes the temporal ordering of annotated event primitives. Relations are not subject to temporal ordering. The table has the following columns: Col.# Content 1. eventprimitive_id - unique identifier for the event primitive 2. order1_type - one of exactly, before, after, unknown 3. order1_value - value is a single whole number, indicating the order 4. order2_type - one of exactly, before, after, unknown. May be EMPTY_OPT if no value is provided 5. order2_value - value is a single whole number, indicating the order. May be EMPTY_OPT if no value is provided 8.0 Known Issues The annotation of Quizlet 3 is representative of the annotation planned for the KAIROS Phase 1 evaluation at the time of Quizlet 3, but some differences from the final evaluation annotation approach may exist because requirements for the evaluation were not fully known at the time that the Quizlet 3 data was prepared. Specifically: - some events may be annotated at a coarser-grained level than was used in the final Phase 1 annotation - some inferable event and argument relations may not be labeled - some inferable temporal ordering information may not be labeled - argument coreference has not been annotated (the entity_id field is empty in the arguments files) - type constraints have not been validated, so that illegal entity types may populate an argument role for some events or relations - the arg_role field in the arguments.tab files uses the annotation label for argument roles, rather than the official output label - the arg_role field in the arguments.tab files contains some invalid value in some rows: "destination place" should be "destination" and "sourceplace" should be "origin" - the arg_inference field in the arguments.tab files is not present - quality review was limited These issues were addressed and corrected in the quality control review for the annotation in Quizlet 4. 9.0 References DARPA. 2018. Knowledge-directed Artificial Intelligence Reasoning Over Schemas (KAIROS). Defense Advanced Research Projects Agency, DARPA BAA HR001119S0014. Ann Bies, Jennifer Tracey, Ann O'Brien, Song Chen, Stephanie Strassel. 2024. Spanless Event Annotation for Corpus-Wide Complex Event Understanding. LREC- COLING 2024: The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. Turin, May 20-24. 10.0 Copyright © 2019 The Atlantic Monthly Group, © 2019 The New York Times Company, © 2019 Boston Globe Media, © 2019 Hearst Communications, Inc., © 2019 ROHM Co., Lt, © 2020 The Local, © 2020 Copyright France 24, © 2020 Elcomercio.pe, © 2020 Europa Press, © 2020 Exportadora Cinematográfica Española SLU, © 2020 Frandsen Digital Media, © 2020 Guardian News & Media Limited or its affiliated companies, © 2020 La Republica EC, © 2020 Malecón Media Group SL, © 2020 NBCUNIVERSAL MEDIA, LLC, © 2020 Scripps Media, Inc., © 2020 Sinclair, Inc., © 2020 The Associated Press, © 2020 Vox Media, LLC, © 2020 Yahoo, © 2020, Nexstar Media Group, Inc., © Copyright Gannett 2020, ©2020 Google, ©2019 Critical Threats Project, © 2020 Trustees of the University of Pennsylvania 11.0 Contacts Christopher Caruso - KAIROS Tech Lead Song Chen - KAIROS Project Manager Dana Delgado - KAIROS Project Manager Ann Bies - KAIROS Project Coordinator ------ README created by Song Chen on October 28, 2024 updated by Ann Bies on October 30, 2024 updated by Song Chen on November 4, 2024 updated by Stephanie Strassel on November 19, 2024