The HAIRCUT System at TREC-9

he token stream. Thetext was lowercased, punctuation was removed, anddiacritical marks were retained. Tokens containingdigits were preserved; however only the first two of asequence of digits were retained (e.g., 1920 became19##). The result is a stream of blank-separatedwords.When using n-grams we construct indexing termsfrom the same sequence of words. These n-gramsmay span word boundaries; an attempt is made todiscover sentence boundaries so that n-gramsspanning sentence...