New Tools for Web-Scale N-grams

While the web provides a fantastic linguistic resource, col lecting and processing data at web-scale is beyond the reach of most academic laboratories. Previous research has relied on search engin es to collect online information, but this is hopelessly ine fficient for building large-scale linguistic resources, such as lists of named-e ntity types or clusters of distributionally-similar words . An alternative to pro- cessing web-scale text directly is to use the information pr ovided in an N-gram corpus. An N-gram corpus is an efficient co mpression of large amounts of text. An N-gram corpus states how often each sequence of words (up to length N) occurs. We propose tools for working with enhanced web-scale N-gram corpora that include richer levels of source annotation, such as part-of-speech tags. We describe a new set of search tools that make use of these tags, and collectively lower the barrier for lexical learning and ambiguity resolution at web-scale. The tools will allow novel sources of information to be applied to long-standing natural language challenges.