Diacritic-Aware Arabic Word Matching

Words in Arabic consist of letters and short vowel symbols called diacritics that are typically inscribed atop regular letters. Changing some diacritics may change both the syntax and semantics of a word; turning a word into another. These results in difficulties when matching two or more words solely based on basic string matching techniques. Typically, Arabic NLP applications resort to morphological analysis to battle ambiguity originating from this and other challenges. In this paper, we introduce the implication relationship algorithm (IRA) which takes two words with the same non-diacritic letters and decides whether they are the same or not. It compares the words and computes a distance metric between diacritics. Second, we introduce the morphology subsume algorithm (MSA) which computes a metric that measures how much one word is a morphological replacement of another word with the same non-diacritic letters. Both algorithms are sound. When each makes a full decision, its decision is always correct. However, MSA is incomplete as it cannot make a decision in a number of cases; which could be the case for expert human readers as they might require context to decide as well. Nevertheless, our experiments show that after several refinement iterations for IRA rules, IRA provides an answer for 100% of the word pairs given, and MSA provides an answer for about 95% of the words given. Both IRA and MSA distance metrics agree on 93% of the intersection. The high agreement value is evidence that Arabic NLP applications that do not directly need the morphological features may use the computationally-lighter IRA algorithm for disambiguation. We demonstrate this result with a lemma disambiguation case study.