README FILE FOR: RATS Speech Activity Detection (SAD) Corpus LDC Catalog-ID: LDC2015S02 Authors: David Graff , Xiaoyi Ma , Stephanie Strassel Kevin Walker , 0.0 Overview of README contents 1.0 Introduction 2.0 Corpus Structure 2.1 Organization of Directories 2.2 File Name Patterns 3.0 Structure of Documentation Tables 3.1 Source audio summary: source_file_info.tab 3.2 Retransmitted audio set summary: retrans_fileset_info.tab 4.0 Description of Data Files 4.1 FLAC-compressed Audio 4.2 Tab-Delimited Annotations 5.0 Description of Audio Collection Process 5.1 Transceiver Channel Specifications 5.2 Recording, Signal Processing, Quality Control and Alignment 6.0 Description of SAD Annotation Process 1.0 Introduction The RATS SAD corpus comprises audio and annotation data created by the LDC to provide training, development and initial test sets for the Speech Activity Detection (SAD) task in the DARPA RATS program. The goal of the RATS program was to develop Human Language Technology (HLT) systems capable of performing speech detection, language identification, speaker identification and keyword spotting on the severely degraded audio signals that are typical of various radio communication channels, especially those employing various types of handheld portable transceiver systems. To support that goal, the LDC assembled a specialized system for transmission, reception and digital capture of audio data, such that a single source audio signal could be distributed and recorded over eight distinct transceiver configurations simultaneously. The relatively clear source audio data was annotated manually to provide the labels needed for a given HLT task - e.g. time boundaries for the onset and offset of speech activity - and these annotations were then projected onto the corresponding eight channels of audio that were recorded from the radio receivers. Further details are provided in later sections of the README file. The source audio used to create the SAD corpus consisted entirely of conversational telephone speech (CTS) recordings, taken either from previous LDC CTS corpora, or from CTS data collected specifically for the RATS program. Previous LDC corpora used in the RATS collection included Fisher English (LDC2004S13, LDC2005S13) and Fisher Levantine Arabic (LDC2007S02). The CTS data that was newly collected for use in the SAD corpus included speakers in three languages: Levantine Arabic, Pashto and Urdu. Corpus creation for RATS SAD involved five languages, listed below together with their 3-letter abbreviations and the amount of source audio present for each: #SOURCE #SOURCE LNG FILES HOURS Full Name ------------------------------- alv 701 141.4 Levantine Arabic eng 692 116.3 English fas 36 9.1 Farsi pus 137 34.3 Pashto urd 170 41.9 Urdu -------------------- Total 1736 343.0 All files are single-channel, and hours reported above are based on file duration; each file is one side of a two-party conversation, so the number of hours of actual speech is roughly half the amount shown above. Between 5 and 8 transmission channels were recorded for each source file, so the total amount of audio in the corpus, counting both source and retransmission data, is 2957.8 hours in 14724 files. Acknowledgments: This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D10PC20016. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. We would like to express special thanks to Dan Ellis at Columbia University, and John Hansen at the University of Texas at Dallas, for their substantial technical assistance during the creation of the RATS corpus. 2.0 Corpus Structure 2.1 Organization of Directories The corpus is presented on a single hard disk; the overall directory structure is as follows: index.html docs/ README.txt -- this file all_flac_info.tab -- paths, durations, file sizes, sample checksums source_file_info.tab -- list/attributes of source audio retrans_fileset_info.tab -- list/attributes of degraded audio sets RATS_SAD_1PGuidelines_V1.pdf -- 1st-pass annotation instructions RATS_SAD_2PGuidelines_V2.pdf -- 2nd-pass annotation instructions data/ dev-1/ audio/ {A,B,C,D,E,F,G,H,src}/ sad/ {A,B,C,D,E,F,G,H,src}/ dev-2/ audio/ {A,B,C,D,E,F,G,H,src}/ sad/ {A,B,C,D,E,F,G,H,src}/ train/ audio/ alv/ {A,B,C,D,E,F,G,H,src}/ eng/ {A,B,C,D,E,F,G,H,src}/ fas/ {A,B,C,D,E,F,G,H,src}/ pus/ {A,B,C,D,E,F,G,H,src}/ urd/ {A,B,C,D,E,F,G,H,src}/ sad/ alv/ {A,B,C,D,E,F,G,H,src}/ eng/ {A,B,C,D,E,F,G,H,src}/ fas/ {A,B,C,D,E,F,G,H,src}/ pus/ {A,B,C,D,E,F,G,H,src}/ urd/ {A,B,C,D,E,F,G,H,src}/ To summarize the organization of the data: - The primary division marks the inventories designated for use as training, initial development set (dev-1), and initial evaluation set (dev-2). These partitions were based on sampling recommendations provided by a team at SAIC, whose task was to administer and score the evaluations of the HLT systems developed for RATS. - Each partition is subdivided into separate paths for audio and SAD annotation files. - The training partition is further subdivided according to the languages used in the source CTS recordings. In the dev-1 and dev-2 sets, files from all languages are grouped together. (Information on the language and source corpus for all data files is provided by the two tables in the "docs" directory, as described in the next section.) - The lowest directory level divides the data according to the channel condition, i.e. source audio (src) or one of the eight transceiver channels, labeled "A" - "H". 2.2 File Name Patterns All data file names consist of a numeric ID, the language used in the recording, and a channel identifier, as follows: source audio files: {iiiii}_{lng}_src.{type} transceiver files: {iiiii}_{jjjjj}_{lng}_{c}.{type} where: {iiiii} is a 5-digit source audio identifier {jjjjj} is a 5-digit transmission session identifier {lng} is one of: alv: Levantine Arabic eng: American English fas: Farsi (Persian) pus: Pashto urd: Urdu {c} is one of: A B C D E F G H {type} is one of: "flac", "tab" So, relating a given source file to all the transmitted files derived from it, involves matching the first five digits of the file names; all the files recorded simultaneously from transceivers in a given transmission session will have identical file names, except for the single letter representing the particular channel. Over the course of the RATS corpus creation project, a given source audio file could be used more than once for retransmission, so that the 5-digit source ID ({iiiii} above) could be associated with two or more 5-digit transmission session IDS ({jjjjj} above); this re-use of a given source file does not occur in the data used for SAD. 3.0 Structure of Documentation Tables 3.1 FLAC file information for all audio: all_flac_info.tab This table has 6 tab-delimited columns, as follows: Col# Content 1 FILEPATH -- starting at "data/" 2 SECONDS -- audio duration in seconds (floating point) 3 FLC_KB -- flac-compressed file size in kilobytes 4 UNC_KB -- uncompressed audio size in kilobytes 5 C_RATIO -- compression ratio (floating point) 6 SAMP_MD5 -- checksum of uncompressed sample data Note that the "SAMP_MD5" value, which comes from the FLAC header of each file, is computed over just the uncompressed sample data, excluding all file-header content; it therefor differs from the MD5 checksum of the flac file taken as a whole. 3.2 Source audio summary: source_file_info.tab This table has six tab-delimited columns, as follows: 1 SRC_ID -- 5-digit numeric ID assigned to the source audio 2 CORPUS -- corpus of origin: "fisher" or "rats" 3 LNG -- "alv", "eng", "fas", "pus", "urd" (cf. 2.2 above) 4 SRC_FILE -- full name of source file in corpus of origin 5 MINUTES -- source recording duration (floating-point) 6 PART -- partition: "train", "dev-1" or "dev-2" The SRC_ID field is used as the file-ID portion of the corresponding file names in the "src" subdirectories under "audio" and "sad" (e.g. "audio/LNG/src/12345_LNG_src.flac", "sad/LNG/src/12345_LNG_src.tab"). 3.3 Retransmitted audio set summary: retrans_fileset_info.tab This table has five tab-delimited columns, as follows: 1 RETRANS_ID -- 5-digit_5-digit source_retrans ID 2 RETRANS_BGN_AT -- transmit start time (EST): YYYYMMDD_hhmmss 3 N_PTT -- number of "push-to-talk" regions (integer) 4 CHANNELS -- indicator of presence/absence of channels 5 PART -- partition: "train", "dev-1", "dev-2" The RETRANS_ID field matches the initial numeric ID fields of the corresponding file names. N_PTT is an _approximate_ count of the number of push-to-talk regions in the transmission session, based on a log file created by the voice-activation relay in the transmission system. The actual number of button-on/button-off transitions may vary within the session from one channel to another, due to mechanical limitations or other factors in the various transceiver configurations. (Note that channel G was not mediated by any PTT device - its transceiver was always engaged over the entire duration of every session.) CHANNELS is an eight-character string in which a given position will either be the letter for the given channel (ABCDEFGH), or an underscore character, indicating that the particular channel failed during that session. In most sessions, all channels worked as intended, but there were many sessions in which one or more channels failed to produce usable recordings, for various reasons. Here's a summary of the distinct patterns in the CHANNELS field: 979 ABCDEFGH 270 ABCDEFG_ 65 AB_DEFGH 349 _BCDEFGH 1 _B_DEFGH 2 __CDEFGH 70 ___DEFGH ---- 1736 sessions total This shows that four channels (D, E, F, G) worked as intended in all sessions; channel H failed in 270 sessions, but all other channels worked in those cases; channel A failed a total of 422 times, and channels B and/or C also failed in some of those sessions; B failed a total of 72 times, and C failed 136 times (65 + 1 + 70). Note that the term "worked as intended" is a generalization; there may be portions of audio on a given channel in a given session where a typical human user would say that the transmission failed, because one or more utterances were unintelligible. The metrics used to determine whether a given channel "succeeded" were based on aggregate signal processing measures over the entire session for the given channel (described in more detail in section 5 below), and not on the intelligibility of utterances contained in the session. 4.0 Description of Data Files 4.1 FLAC-compressed Audio All audio files are presented here as single-channel, 16-bit PCM, 16000 samples per second; lossless FLAC compression is used on all files; when uncompressed, the files have typical "MS-WAV" (RIFF) file headers. All the source CTS audio data, whether previously published in Fisher corpora or newly collected for RATS, was originally captured as 8-bit mu-law, 8000 samples per second. That original format has been converted to 16-bit, 16-KHz for consistency and ease of use in combination with the original capture format of the retransmission audio channels. 4.2 Tab-Delimited Annotations All RATS annotation files are presented as tab-delimited tables with 12 columns per row, as follows: 1 data partition (train, dev-1, dev-2) 2 file_ID ("{iiiii}_{lng}_src" or "{iiiii}_{jjjjj}_{lng}_{c}") 3 start time (floating-point seconds) 4 end time (floating-point seconds) 5 Speech Activity Detection (SAD) segment label 6 SAD provenance 7 speaker ID 8 SID provenance 9 language ID 10 LID provenance 11 transcript 12 transcript provenance This layout was designed to provide a consistent tabular format for use across all RATS tasks; as such, only a subset of fields are relevant to SAD annotation: fields 1-6, and fields 9 and 10; the others (speaker ID, SID provenance, transcript, transcript provenance) are not relevant, and are left empty in the SAD annotation files. The "language ID" field shows the expected language of the source audio file (alv, eng, fas, pus or urd); the "LID provenance" in all cases is shown as "original", to indicate that, with respect to the SAD annotations, no additional effort was made to confirm the language being used in any given utterance in the file. Field 5 (SAD segment label) can have one of the following values: S : speech segment NS : non-speech segment NT : "button-off" segment (not transmitted according to PTT log) RX : "button-off" segment (not transmitted according to RMS scan) Field 6 (SAD provenance) can be either "manual" or "automatic", indicating that the particular time-stamp boundaries for the segment were set by manual annotation, or by integration of manual labels with subsequent automatic signal processing. The "src" channel segment labels are all "manual", and comprise only "S" and "NS" labels. The other labels, and "automatic" provenance, apply only to transmitted audio channels. The annotation procedures are described in more detail in section 6 below. 5.0 Description of Audio Collection Process 5.1 Transceiver Channel Specifications The layout of hardware on the eight transceiver channels is as follows: :: Transmitter :: Receiver :: RF Band / CHN :: Make Model :: Make Model :: Modulation --------------------------------------------------------------- A :: Motorola HT1250 :: AOR AR5001/D :: UHF/NFM B :: Midland GXT1050* :: AOR AR5001/D :: UHF/NFM C :: Midland GXT1050* :: TenTec RX400 :: UHF/NFM D :: Galaxy DX2547 :: Icom IC-R75 :: HF/SSB E :: Icom IC-F70D :: Icom IC-R8500 :: VHF/NFM F :: Trisquare TSX300 :: Trisquare TSX300 :: UHF/FHSS G :: Vostek LX-3000 :: Vostek VRX-24LTS :: UHF/WFM H :: Magnum 1012 HT :: TenTec RX340 :: HF/AM Explanation of "RF Band / Modulation" acronyms: HF: High Frequency VHF: Very High Frequency UHF: Ultra High Frequency AM: Amplitude Modulation FHSS: Frequency Hopping Spread Spectrum NFM: Narrow-band Frequency Modulation SSB: Single-Side-Band WFM: Wide-band Frequency Modulation 5.2 Recording, Signal Processing, Alignment and Quality Control The transmission and capture of eight simultaneous channels from a single source audio file ran as a continuous, automated process. It was managed via a database, in which source audio files were assigned a numeric "src_id" as they were queued up for transmission, and each resulting set of 8-channel received audio files was assigned a common "sig_id" when the transmission system created the recordings. These "src_id" and "sig_id" numbers were used in the names various audio files. As the recordings were uploaded from the capture system to network storage, additional automated processes were applied to establish the time offsets for "button-on" regions in the PTT channels, measure signal levels at 20 msec intervals on each file, and do cross- correlation based alignment between the source audio data and channel G (which was generally the least degraded, and was always engaged, rather than being controlled by PTT button events). Frame-based RMS signal level measures were used to determine the "peak-to-valley" dynamic range for each transceiver audio file; if the peak energy or the difference between highest and lowest frame energy never reached given thresholds (established heuristically for each channel), the particular file was flagged as unusable in the database. "Button-on" regions were identified on the basis of time-stamp log data from the voice-activated relay (VAR) system that triggered button events for transmission on channels A-E and H. For channel F, the PTT button transitions were identified using a simple signal analysis process for finding peak-clipped transients in the audio. For channels A, B, C, E and H, the frame-based RMS data was also used in combination with the log-based "button-on" regions, to identify portions where the radio carrier signal dropped out prematurely (i.e. where transmission on a given channel stopped before the VAR turned the transmit button off - this could happen on extended utterances, if the utterance duration exceeded operating parameters for the transceiver, or the device overheated, etc). Based on the results of these processes involving PTT events in each session, the original "speech / non-speech" (S/NS) annotations on the source audio data have been modified by further subdividing the "S" regions, and assigning new labels to certain portions: "NT": "button-off" state indicated by VAR log or channel F analysis "RX": loss of carrier detected by frame-based RMS measures In effect, both NT and RX segments in annotation files indicate that (some portion of) a speech region was not transmitted. The difference between NT and RX has to do with how they are asserted, and how they are distributed among the various channels: - NT regions are based on the VAR time-stamp log, and will be found consistently across all the PTT-mediated channels (A-F,H) for a given instance within a given transmission session; there may be slight differences in NT offsets for channel F relative to the others, because button events on channel F are determined directly from signal analysis, rather than from the VAR log. - RX regions are based on per-frame RMS levels in a given channel, and is typically unrelated to transmission behavior on other channels. Regarding time alignment, initial study of the 8-channel capture data showed small but noticeable and consistent differences in time offsets among the 8 transceivers. Relative to channel G, the other seven channels showed time lags between 0.004 and 0.031 seconds. Meanwhile, the start-up of the 8-channel capture was physically isolated from the start-up for playing the source signal into the transmitters - the transmission and reception/capture systems were only approximately coordinated in time. The "S/NS" annotations, which were done manually on the clean source audio file, were adapted to each of the 8 transceiver audio files as follows: First, an alignment tool called "skewview", created and made available by Dan Ellis of Columbia University during the early stages of the RATS program, was used to establish an alignment offset between the source audio file and channel G. This tool (which can be found at http://labrosa.ee.columbia.edu/projects/skewview/) uses normalized cross-correlation to compute the optimal time offset between two related signals on a frame-by-frame basis. Its output can reveal both gradual drifts in offsets (due to marginally different sampling rates) and abrupt discontinuities (e.g. due to sampling dropouts caused by buffer overruns, etc); total absence of coherent alignment over consecutive frames indicates that the two signals being compared have nothing in common. For the sessions that yielded a strong and steady correlation between source audio and channel G, the median time offset for the session as a whole was used as a baseline reference, and relative offsets for the other 7 channels were added to this value. Then the time stamps and labels of the source annotations were transposed to the adjusted alignment offset for each transceiver channel in the given session, and this version of the annotation was used in combination with the button-on timing data to produce the eight transceiver annotations. A final step of QC was to arbitrarily select a speech ("S") segment up to 5 seconds in duration from the session (typically from the middle or latter part of the source audio file), use the computed alignment offsets for each channel to extract the corresponding transceiver segment (given that these were also labeled "S" - i.e. had presumably been transmitted as expected), and then use skewview again, comparing the source audio extract to each channel extract. If this run yielded less than 1 msec offset difference between the source and each transceiver extract, the session data and alignments were flagged as fully successful. 6.0 Description of SAD Annotation Process For SAD annotation, speech vs. non-speech was defined as follows: Speech includes: - One person talking with no other sounds or voices in the background - One person talking with non-speech noises (e.g. traffic or music) in the background - Laughter or other vocalizations from that same person like coughing, sneezing (excluding singing) Non-speech includes: - Silence, or regions with no noise other than crackle or other noises from the recording - Background noise including sound effects, TV or radio, traffic, other noises - Background noise including faint background speech, laughter or vocalizations from someone else - Singing from any speaker - Non-vocal music - Computer voices (not human voices) Annotators were instructed to exclude recordings with multiple speakers on the main channel (i.e., report them as unsuitable for annotation and remove them from the pipeline). Annotation files were produced using a three-step process. Step 1: Automatic SAD Labeling Following echo-cancellation of the 8-bit, single-channel wav files, LDC's in-house automatic speech activity detector 'segmenter' was run against the audio to produce a speech segmentation for each file. The segmenter provided a .1 second padding on each side of each segment. Preliminary annotation files were then generated from these segmentations and sent for manual annotation. Step 2: Manual First Pass SAD Annotation First pass annotation was performed as a quick correction pass over the automatic SAD output. Wherever possible files were assigned to native speakers of the language in question. First pass annotators are generally less experienced junior annotators. During first pass SAD annotation, annotators were instructed to do the following: a) listen to all segments marked as speech by the auto-segmenter - delete non-speech segments - flag mixed speech/non-speech segments for editing in a 2nd pass - do nothing with speech segments b) listen to all segments marked as non-speech by the auto-segmenter. For any region containing speech, create a speech segment for that region. Step 3: Manual Second Pass SAD Annotation Second pass SAD annotation was performed as a careful review of first pass annotators' output. Wherever possible files were assigned to native speakers of the language in question. Second pass annotators are generally highly experienced senior annotators. During second pass SAD annotation, annotators were instructed to do the following: a) listen to each speech segment and adjust the endpoints as needed to ensure accuracy. Merge or split segments as needed to ensure that lengthy periods of non-speech are excluded from speech segments (using a threshold of .5 seconds or less). b) listen to each non-speech segment and adjust the endpoints as needed to ensure accuracy c) listen to the entire recording again from start to finish to ensure that segmentation is complete and accurate.