Adaptive Load Balancing And Change Visualization For WebVigil

Abstract

There is a need for selective monitoring, the contents of web pages. Periodical visit for understanding changes to a web page is both inefficient and time consuming. WebVigiL is a system developed for automating the change detection and timely notification of HTML/XML pages based on user specified changes of interest. User interest, specified as a sentinel/profile, is automatically monitored by the system using a combination of learning-based and event-driven techniques.

The first prototype concentrated on the functionality of the WebVigiL system. This thesis extends the WebVigiL system in a number of ways. The primary focus of this thesis is the improvements to the performance and scalability aspects of the prototype. A load balancer is proposed based on the analysis and estimation of the load factors imposed by a sentinel on the system. An adaptive load balancer has been developed for the WebVigiL, that uses various sentinel properties to distribute sentinels among servers using a number of strategies. A web-based DASHBOARD has been developed to enable users to manage and visualize changes to their sentinels. As part of the DASHBOARD, a Dual-Frame presentation for displaying the detected changes is provided (both for HTML and XML). Other contributions of the thesis include integration of the above modules into the current system, making the WebVigiL system stable and robust (by fixing various bugs) and testing the system for various test cases.