BOLT Arabic Discussion Forums
| Item Name: | BOLT Arabic Discussion Forums |
| Author(s): | Jennifer Tracey, Haejoong Lee, Stephanie Strassel, Safa Ismael |
| LDC Catalog No.: | LDC2018T10 |
| ISBN: | 1-58563-839-0 |
| ISLRN: | 663-919-074-680-5 |
| DOI: | https://doi.org/10.35111/9fjp-2a75 |
| Release Date: | March 15, 2018 |
| Member Year(s): | 2018 |
| DCMI Type(s): | Text |
| Data Source(s): | discussion forum |
| Project(s): | BOLT |
| Application(s): | machine translation |
| Language(s): | Egyptian Arabic |
| Language ID(s): | arz |
| License(s): |
LDC User Agreement for Non-Members |
| Online Documentation: | LDC2018T10 Documents |
| Licensing Instructions: | Subscription & Standard Members, and Non-Members |
| Citation: | Tracey, Jennifer, et al. BOLT Arabic Discussion Forums LDC2018T10. Web Download. Philadelphia: Linguistic Data Consortium, 2018. |
| Related Works: | View |
Introduction
BOLT Arabic Discussion Forums was developed by the Linguistic Data Consortium (LDC) and consists of 813,080 discussion forum threads in Egyptian Arabic harvested from the Internet using a combination of manual and automatic processes.
The DARPA BOLT (Broad Operational Language Translation) program developed machine translation and information retrieval for less formal genres, focusing particularly on user-generated content. LDC supported the BOLT program by collecting informal data sources -- discussion forums, text messaging and chat -- in Chinese, Egyptian Arabic and English. The collected data was translated and annotated for various tasks including word alignment, treebanking, propbanking and co-reference. The material in this release represents the unannotated Arabic source data in the discussion forum genre.
Data
Collection was seeded based on the results of manual data scouting by native speaker annotators. Scouts were instructed to seek content in Egyptian Arabic that was original, interactive and informal. Upon locating an appropriate thread, scouts submitted the URL and some simple judgments about it to a database, via a web browser plug-in. When multiple threads from a forum were submitted, the entire forum was automatically harvested and added to the collection. The scale of the collection precluded manual review of all data. Only a small portion of the threads included in this release were manually reviewed, and it is expected that there may be some offensive or otherwise undesired content as well as some threads that contain a large amount of non-Arabic content. Language identification was performed on all threads in this corpus (using CLD2), and threads for which the results indicate a high probability of largely non-Arabic content are listed in arz_suspect_LID.txt in the docs directory of this package. It should also be noted that many threads may contain a mixture of Egyptian and other varieties of Arabic, even among the threads that are primarily Arabic.
The corpus is comprised of zipped HTML and XML files. The HTML files are a raw HTML file downloaded from the discussion thread. If the thread spanned multiple URLs, it was stored as a concatenation of the downloaded HTML files. The XML files were converted from the raw HTML.
Acknowledgement
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR0011-11-C-0145. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
Samples
Please view this html sample and xml sample.
Updates
None at this time.