LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1

Item Name: LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1
Author(s): Mohamed Maamouri, David Graff, Basma Bouziri, Sondos Krouna, Ann Bies, Seth Kulick
LDC Catalog No.: LDC2010L01
ISBN: 1-58563-555-3
ISLRN: 898-935-705-624-6
DOI: https://doi.org/10.35111/wgjk-zy44
Release Date: July 19, 2010
Member Year(s): 2010
DCMI Type(s): Text
Application(s): natural language processing, machine translation, information retrieval
Language(s): Standard Arabic, Arabic
Language ID(s): arb, ara
License(s): LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 Agreement
Online Documentation: LDC2010L01 Documents
Licensing Instructions: Subscription & Standard Members, and Non-Members
Citation: Maamouri, Mohamed, et al. LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 LDC2010L01. Web Download. Philadelphia: Linguistic Data Consortium, 2010.
Related Works: View

Introduction

The LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1 was developed by researchers at LDC. SAMA 3.1 is based on, and updates, Buckwalter Arabic Morphological Analyzer (BAMA) 2.0 (LDC2004L02), which was developed by Tim Buckwalter. Since this is the first public release of SAMA, it has been numbered continuously to reflect the continuity between this release and previous BAMA releases.

SAMA 3.1 is a software tool for the morphological analysis of Standard Arabic. SAMA 3.1 considers each Arabic word token in all possible prefix-stem-suffix segmentations, and lists all known/possible annotation solutions, with assignment of all diacritic marks, morpheme boundaries (separating clitics and inflectional morphemes from stems), and all Part-of-Speech (POS) labels and glosses for each morpheme segment. The generated output may then be reviewed by users, and the most appropriate annotation selected from among several choices.

The software layer of SAMA 3.1 relies on a data layer that consists primarily of three Arabic-English lexicon files: prefixes (1328 entries), suffixes (945 entries), and stems (79318 entries representing 40654 lemmas). The lexicons are supplemented by three morphological compatibility tables used for controlling prefix-stem combinations (2497 entries), stem-suffix combinations (1632 entries), and prefix-suffix combinations (1180 entries).

Differences since BAMA 2.0

The input format, output format, and data layer of SAMA 3.1 were designed to be backward compatible with BAMA. Incremental changes to the data layer in SAMA have resulted in:

  • increased lexicon coverage in the dictionary files
  • important changes and additions to the inventory of POS tags
  • more possible solutions generated for numerous word forms

Data-layer changes are summarized in more detail in the table_updates*.txt documentation files included in the corpus documentation.

The software implementation has been updated to allow more input/output options, installation and configuration options, and smoother incorporation in other Perl tools/services.

The structure of the dictionary and morphotactic tables has remained the same (the tables provided with SAMA 3.1 differ from the BAMA 2.0 tables only in size and content, not in format). Logical separation between the software layer and data layer allows the new software tools to be used with previous versions of the tables (instructions are provided with software documentation).

The basic logic that implements the segmentation and analysis look-up for Arabic words is essentially unchanged since BAMA 2.0. The perldoc documentation for the SAMA.pm Perl module gives a full account of the tokenization logic.

The data layer is now accessed through Berkeley DB, with result-caching enabled by default, leading to improved performance. Various utility scripts have also been added to the software package to facilitate more flexible interaction with tools and data.

UTF-8 is now the default input/output and internal character encoding, with automatic conversion of different input encodings (cp1256, iso-8859-6, and Buckwalter transliteration are also accepted). With this change, the use of UTF-8 as input is now fully supported, eliminating a range of problems that would result from having to convert to cp1256 for analysis. Full details about input/output options are provided in the SAMA.pm documentation.

Further details on changes in software options and implementation may be found in the perldoc software tool documentation, and in the Changes*.txt documentation files.

Dependencies

There are two dependencies for installing and using SAMA 3.1: the DB_File.pm module (available from CPAN), and Encode::Buckwalter (included with the SAMA 3.1 distribution). The DB_File module in turn requires that the Berkeley DB libraries be present.

Samples

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Updates

There are no updates available at this time.

Additional Licensing Instructions

This 'members-only' corpus is available to current members who can request the data at the listed reduced-license fee. Contact ldc@ldc.upenn.edu for information about becoming a member.

Available Media

View Fees





Login for the applicable fee