Using Property-Based Testing, Weighted Grammar-Based Generators, and a Consensus Oracle to Test Browser Rendering Engines and to Reproduce Minimized Versions of Existing Test Cases
View/ Open
Date
2019-12-10Author
Martin, Joel David
0000-0001-8734-2019
Metadata
Show full item recordAbstract
Verifying that a web browser rendering engine correctly renders all valid web pages is challenging due to the size of the input space (valid web pages), the difficulty of determining correct rendering for any given web page (the test oracle problem), and the degree to which normal variation in browser rendering behavior can obscure other differences (fonts, bor- ders, input controls, etc). These challenges lead to manual human involvement during the testing process. We propose a new Property-Based Testing (PBT) approach that addresses these challenges in order to enable automated web browser render testing. Our approach is composed of the following modules: a system for translating HyperText Markup Lan- guage (HTML) and Cascading Style Sheets (CSS) specification data into grammar-based generators; a grammar weighting system that controls test case generation along with mul- tiple methods for automatically adjusting those weights; and a consensus oracle of multiple rendering engines to identify failing test cases.
Our approach increases the practicality of real-world testing. It has the ability to re- produce existing test cases from external sources, is able to shrink test cases to assist with root-cause analysis, and supports generic test specifications. We validated and characterized the effectiveness of our approach with a constrained markup grammar developed for this purpose. Applying our approach while testing Mozilla Firefox and Google Chrome revealed 1695 unique test cases that yield rendering differences. Testing of Mozilla’s new Servo web browser revealed multiple bugs including eleven crashes and resource leaks. We reported these issues to Mozilla developers and are working together to develop and verify solutions.