screen-scraper 4.0

screen-scraper is a tool used to extract data from web sites, and consists of two main parts. The first is a proxy server that can be run locally in order to view in plain text the contents of both HTTP and HTTPS requests and responses as they pass between a web browser and server. The second is an engine that can be easily configured to extract information from web sites, handling such tasks as authentication to a site, following redirects, and automatic handling of cookies. Special patterns, which can optionally make use of regular expressions, are used to identify and extract data. An embedded scripting engine is included so that extracted data can be manipulated, written out to a file, inserted into a database, etc. screen-scraper is written in Java, and has been tested on Mac OS X and various flavors of Linux and Windows.

screen-scraper also has the ability to be run as a server, which allows for it to be invoked from external applications written in a variety of languages. The current supported languages are Java, PHP, and any COM-friendly language such as Visual Basic or ASP.