A field evaluation of the Italian “Automated reverse directory assistance” service

The paper describes a field evaluation of the automated ‘reverse directory assistance’ service presently in use in Italy in which information about names and addresses is provided by a TTS system. A simulation of the service using a natural voice was also run to get comparative data. Both services were accessed from an office room and a call-box on the street. Different evaluation metrics, such as intelligibility, task completion, task correctness, transaction success, and user's reactions were used. The aim of the work was to evaluate TTS synthesis in real world use and to make a comparison between laboratory data and data on system performance in a real application. Such a comparison suggested that in laboratory tests more attention should be dedicated to simulate more closely the conditions that can be predicted in real world use, by including important aspects that are generally not taken into consideration in laboratory tests and that are likely to have a large influence on TTS system performance such as environmental noise, prosody, and task complexity. The results also underline the importance of field evaluations to get an overall view of the usability of a service in real applications and with users who are as similar as possible to actual users.