Semantic Automation from Screen Capture

© Semantic Automation from Screen Capture Omer Barkol, Ruth Bergman, Ayelet Pnueli, Sagi Schein HP Laboratories HPL-2009-161 Automation technology, Image analysis, Visual parsing IT organizations want efficiency and are constantly attempting to reduce or eliminate manual work. Automation technology facilitates the transition from manual to automatic. However, until now, automation technology has been applied in limited settings in IT organizations. For example, automated software testing uses automation technology, but only about 5-10% of software testing is automatic. One of the reasons automation technology is not more widely adopted, is that there is a very high barrier to using it. The objects and constructs used for today's automation originate from the underlying application or computing environment, so that using automation technology is as complex as programming. We propose a new automation technology for applications with Graphical User Interfaces (GUI). Unlike existing automation technology, which uses the objects defined by the technology that was used to build the GUI, the technology we propose uses screen capture. The system described in this paper infers GUI semantics from screen images. The system is modelled after the notion of compilers for programming languages. It has a structural analysis module, which corresponds to the lexical analyzer (Lex), and a semantic analysis module, which corresponds to the parser (Yacc). Structural analysis uses image analysis techniques including segmentation and object recognition, and semantic analysis uses graph grammars and visual parsing. External Posting Date: July 21, 2009 [Fulltext] Approved for External Publication Internal Posting Date: July 21, 2009 [Fulltext] Copyright 2009 Hewlett-Packard Development Company, L.P. Semantic Automation from Screen Capture Omer Barkol HP Labs Haifa, Israel omer.barkol@hp.com Ruth Bergman HP Labs Haifa, Israel ruth.bergman@hp.com Ayelet Pnueli HP Labs Haifa, Israel ayelet.pnueli@hp.com Sagi Schein HP Labs Haifa, Israel sagi.schein@hp.com