Free worlds largest spider software for windows

Gyxis Image Spider is designed to work as a spider on web pages, looking for images and downloads them intelligently.
Enter url to search, choose scope (below web folder, domain, anything), choose where and how to save the found images. Designed for long runs.
You can save your progress in an xml file and continue spidering at a later time.

Burp spider is a tool for enumerating web-enabled applications enumBurp spider is a tool for enumerating web-enabled applications that uses various intelligent techniques to generate a comprehensive inventory of an applications functionality and content.
Burp spider enables the user to obtain a detailed understanding of how a web application works, avoiding the time-consuming and unreliable task of manually following links, submitting forms and scouring HTML source code.
Potentially vulnerable application functions can be quickly identified, allowing the user to check for specific vulnerabilities such as SQL injection and directory traversal.
Main features:
- Accurate HTML and JavaScript parsers to effectively enumerate the applications content and functionality.
- Presentation of findings in tree and table formats, with detailed information about all results.
- Handling of HTML forms, with automatic or user-guided form submission.
- Full integration with other burp suite plugins.
- Authentication to protected areas of the application using supplied credentials.
- Processing of cookies.
- Detection of custom "not found" responses.
- Fine-grained scope control.
- SSL support.
- Identification of dynamic "application" pages which use data parameters or are session-dependent.
- IDS evasion techniques.
- Support for downstream proxy server.
- Authentication to downstream proxy and web servers, using basic, NTLM or digest authentication types.
- Optimised memory and disk usage to allow efficient spidering of very large sites.
System requirements:
- Java Runtime Environment version 1.4 or later

Niche Spider grabs data from thousands of sites, and then presents the data collected in neat pages without needing any database Niche Spider is a simple, small and powerful spider, a piece of software which grabs specific webpage data. It can grab data from tens of thousands of sites, and then present the data collected in nice, neat pages without the need of any database. Yes, NO database programming required!
When visiting a website, Niche Spider also leaves its calling card in the sites visitor logs (the Referer and User-Agent, which you can set), just like Googlebot or any other spider or web crawler. That, in itself, is a great way to announce yourself to the world, and to generate some curiosity/traffic at the same time.
However , Niche Spider is NOT Googlebot, or Yahoos Slurp, or MSNs MSNBot. Google, Yahoo and MSN have access to computers, servers and databases us lesser mortals can only dream about.
And the spiders they use crawl the web, looking for and following links to pages, which are then added to their respective databases. Such is their power they can traverse the entire web in a month or two, or less, adding and updating billions of pages in the process.
Niche Spider does NOT crawl the web, but targets a small piece of the web instead (a niche area). You decide what that niche is. It may, or may not be related to your websites theme. Thats completely up to you.
And before we get started, theres a couple of other points. Niche Spider is FREE. However, in return, the odd link is added to the bottom of one or two of the pages produced (not every page). Of course, you can delete any such links if you wish. But if you keep any on, however, it would be very much appreciated.
It is also possible to make some cash by adding Google AdSense code to each template. Or you can include whatever affiliate products you may be aiming to promote. Its up to you. But theyll then appear on every page produced by Niche Spider. Enough said!

Chilkat Spider ActiveX is a web crawler component. It will help you create programs to spider a site. robots.txt compliant.