Dialogs Re-Enacted Across Languages

Nigel G. Ward, Jonathan E. Avila, Emilia Rivas, Divette Marco

Dialogs Re-enacted Across Languages (DRAL) is a collection of 3816
pairs of matching Spanish and English utterances.  Each pair
consists of a short fragment extracted from a spontaneous conversation
and a close re-enactment in the other language by the original
speaker.


Expected Uses

The DRAL corpus was designed for the evaluation of future
speech-to-speech translation systems on their handling of
conversational utterances, for training cross-language prosody models
for such systems, and for the comparative study of dialog behaviors
across languages.


Collection Procedure 

There were 129 unique speakers, all bilingual speakers of General
American English and of Mexico-Texas Border Spanish.  In various
pairings they came to the lab to be recorded with close-talking
microphones.  Each pair first had a 10-minute conversation in one
language, sometimes following topic ideas or engaging in simple
activities that we suggested in order to increase the diversity of
pragmatic functions.  From these conversations, various fragments
of the conversations were chosen for re-enactment, and the original
speakers duly produced equivalents in the other languages.  Each
re-enactment was vetted for fidelity to the original and naturalness
in the target language by three people: the speaker, the interlocutor,
and the producer.  Subsequently the re-enactments were split into
individual matching utterances. 

In total there were 128 conversations, recorded in 2022 through 2023
in El Paso, Texas.  The paired utterances averaged 2.7 seconds,
comprising 172 minutes in total, in each language.

Directory Structure and File Formats

data/speech/utterances/ is the main data set.  Each file is 16-bit,
16KHz mono audio in flac format.  Filenames are in the format
EN_050_S_19.flac, where 50 is the conversation ID, S indicates that
this is a short fragment, and 19 identifies the utterance.  The paired
utterance has the same name, but with ES instead of EN.

data/speech/reenactments/ contains the fragments chosen for
reenactment and their re-creations in the other language.  While for
most purposes the pairs in utterances/ will be more useful, these
longer pairs could support the study of pause patterns and the timing
of turn-taking.  The filenames are in the same format, but with L
instead of S.  These are stereo files, although not all include audio
on both tracks.


docs/ contains

README.txt, this file

an Interspeech 2023 paper that situates and overviews this data
collection: Towards Cross-Language Prosody Transfer for
Dialog. Jonathan E. Avila and Nigel G. Ward.
      
a Technical Report with full detail: UTEP-CS-23-27, Dialogs
Re-enacted Across Languages, Version 2. Avila, Ward, Rivas and
Marco.

metadata/ contains 4 csv files, designed to serve the
following purposes.  1) Training and testing according to our standard
split, 76:24, by referring to the last field of utterances.csv.  2)
Supporting conditioning on or normalization with respect to overall
speaker behavior, by referring to the participant_id_unique field of
utterances.csv. 3) Conditioning on the translation direction,
referring to the original_or_reenacted field of utterances.csv. 4)
Conditioning on speaker dialects, by looking up the participant_id
from utterances.csv in the participants.csv table.  5) Similarly
conditioning on language_strength, with values {1 English stronger, 2
English slightly stronger, 3 equal, 4 Spanish slightly stronger, 5
Spanish stronger}.


Languages: ENG and SPA, equally