Searching and Classifying Mobile Application Screenshots

Abstract

Searching for a particular application layout image is a challenging task. No search provider gives an adequate method to filter the query results to the look of a mobile application. Searching for a particular style of application requires lots of manual time sifting through the results returned. Search engines such as Google are too broad and return too many unrelated results without providing sufficient filters on things like category or layout type. This paper proposes a technique that would allow the searching and classifying of mobile application screenshots based on the layout of the content, the category of the application, and the text in the image. It was originally conceived to support REMAUI (Reverse Engineering Mobile Application User Interfaces), an active research project headed up by Dr. Csallner. REMAUI has the ability to automatically reverse engineer the User Interface layer of an application by being given input Images. The long term goal of this work is to create a full search framework for any UI image. In this paper, we introduced the first steps to this framework by focusing on mobile UI screenshots. We discuss 3 techniques to classify the layout of the image, Block Analysis, Interval Encoding, and Bag of Visual Words. We continue on to discuss a method to classify the category of the application based on the text in the image. Finally, we put all the information together in a single REST API. The API can search input images by the image content and filter by type and layout. The results are ranked by Solr for relevance and returned as json by the API.