Departmental Papers (CIS)

A New Approach to Lexical Disambiguation of Arabic Text

Lyle Ungar, University of Pennsylvania
Rushin Shah, Carnegie Mellon University
Dean Foster, University of Pennsylvania
Mark Y. Liberman, University of Pennsylvania
Mohamed Maamouri, University of Pennsylvania

Document Type Conference Paper

Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 725–735, MIT, Massachusetts, USA, 9-11 October 2010. c 2010 Association for Computational Linguistics

Abstract

We describe a model for the lexical analysis of Arabic text, using the lists of alternatives supplied by a broad-coverage morphological analyzer, SAMA, which include stable lemma IDs that correspond to combinations of broad word sense categories and POS tags. We break down each of the hundreds of thousands of possible lexical labels into its constituent elements, including lemma ID and part-of-speech. Features are computed for each lexical token based on its local and document-level context and used in a novel, simple, and highly efficient two-stage supervised machine learning algorithm that overcomes the extreme sparsity of label distribution in the training data. The resulting system achieves accuracy of 90.6% for its first choice, and 96.2% for its top two choices, in selecting among the alternatives provided by the SAMA lexical analyzer. We have successfully used this system in applications such as an online reading helper for intermediate learners of the Arabic language, and a tool for improving the productivity of Arabic Treebank annotators.

 

Date Posted: 25 July 2012