robots.txt
Sponsored Links
Sponsored Links
Secleted [ 0 ] software to compare
Results 1 - 15 of about 17
Robots-txt-generator 1.01
The spiders that visit your site consumes much of your bandwidth? If you want to control the robots or spiders that visit your site, robots-txt generator is the solution. more>>
robots-txt generator
The spiders that visit your site consumes much of your bandwidth?
If you want to control the robots or spiders that visit your site, robots-txt generator is the solution.
It is very common that the webmaster of each site spends a lot of money paying for bandwidth and it is unfair that a significant portion of that bandwidth is consumed spiders.
Robots txt generator, you can stop or control the bots.
Some robots or spiders saturate your server making countless queries per minute?
With Robots txt generator, you can limit how many requests can make any robot or spider per hour or per minute.
In addition, the program has a database of more than 300 bots, allowing you to search on the database the source and owner of any bot you have found in your logs.
You can even specify different restrictions for different bots. Per example, you can allow google bot, but disallow google-image-bot. You can specify a different restriction level for each one of the 300 bots in the database or can specify some restrition to the "general" bots, plus some specific for some specifi bots. Like giving a ban to some specific bot.
Best of all, is that the software is free!
System Requirements: none
<<less Download (51.50KB)
Added: 2009-09-15 License: Freeware Price: $0
downloads
RoboGen LE 1.52 R2
a visual editor for Robot Exclusion Files more>> RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public. RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable<<less
Download (1.5M)
Added: 2008-11-03 License: Freeware Price:
356 downloads
WebSEOTool 1.0
Web SEO Tool toolbar for Internet Explorer.Use our free SEO tools for your comprehensive search engine optimization, analyis and research. more>>
Web SEO Tool toolbar for Internet Explorer.Use our free SEO tools for your comprehensive search engine optimization, analyis and research. Find out how many total backlinks there are for any one website. Analyze incoming and outgoing links. Extract all links from the specified domain. Check the number of links pointing to the website of your choice.View the number of backlinks found in search engines for multiple websites. Find domains containing specific keywords or phrases. Check keyword results and get suggestions. Check multiple websites search engine position with a single keyword. Check how many pages Google has indexed for one website. Check for and view the contents of the robots.txt file for any website. View the headers of the site of your choice. Extract meta-tags information from a web page. Look up the hostname for any IP address. Validate your cascading stylesheets with the W3C CSS validator. Find out the WHOIS details for your domain. Find links on Twitter to one website. Check multiple sites for links bookmarked in Delicious. Keyword Rank Checker. List Cleaner, and Turnkey PageRank Checker.
System Requirements: Internet Explorer
<<less Download (1.48MB)
Added: 2009-10-13 License: Freeware Price: $0.00
downloads
Chilkat Spider ActiveX 1.1.0
Chilkat Spider ActiveX is a web crawler component more>>
Chilkat Spider ActiveX is a web crawler component. It will help you create programs to spider a site. robots.txt compliant.
<<less Download (564KB)
Added: 2006-10-20 License: Freeware Price:
1147 downloads
Other version of Chilkat Spider ActiveX
Chilkat Software, Inc. - Advanced features include caching, "avoid" patterns, and robots.txt compliance. Chilkat Spider ActiveX. Crawl a website with this freePrice: $0
License:Freeware
License:Freeware
AblyFTP 2.9.9
Free FTP Uploader and Image disk builder. Bilingual ( English, French ) For c... more>> Free FTP Uploader and Image disk builder. Bilingual ( English, French ) For complex related sites or disk images: Import/export files status; General and specific parameters; Shortcuts to share Files and folders between projects; Variables subtitution for ascii files ; deny info for robot.txt; Pre and post build commands; Local emulator for tutoring, testing or disk image building; Right-click selections. Auto-Install, Menu Uninstall, Multi-documents, preferences and contextual html help. Includes examples<<less
Download (625KB)
Added: 2006-08-10 License: Freeware Price: Free
1170 downloads
Other version of AblyFTP
files to manage a site stylesystem; deny info for robot.txt; Pre and post build commands; Local emulator for tutoring, testing or disk image building; Right-click selections. Auto-InstallLicense:Freeware
RTSoftwares Sitemap Generator 1.0
RTSoftwares Sitemap Generator is one of the most marvelous web design tool that generate HTML/TXT sitemap. Sitemap generated by sitemap generator is robots.txt compatible. more>>
RTSoftwares Sitemap Generator 1.0 is one of the most marvelous web design tool that generate HTML/TXT sitemap. Sitemap generated by sitemap generator is robots.txt compatible. Easy-to-use sitemap generator for Web masters. One click sitemap generation with many configurable options. Download your copy of free sitemap generator and create rich html sitemap for your Web site with a simple button click. Currently Sitemap generator does not creates XML sitemap but can only create HTML/TXT sitemap. Register for the updates, send your feedback / suggestions and we will try the best to implement the suggestions to make sitemap generator a better software for webmasters.
Download (638.9KB)
Added: 2009-05-10 License: Freeware Price:
downloads
Web Bot Programming Library 1.0.0
Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance. more>>
Crawl the web with this free ActiveX web bot component. Advanced features include caching, "avoid" patterns, and robots.txt compliance.
Whats new in this version: None
<<less Download (0.52MB)
Added: 2008-09-07 License: Freeware Price: $0
432 downloads
SEOpen 0.8
SEOpen is an extension I made for Firefox help with search engine optimization. more>> SEOpen is an extension I made for Firefox to help with search engine optimization. It provides many helpful tools at the click of a mouse.
Current SEOpen features are all accessed through right-clicking an open area on a web page or by using the new toolbar; they include:
* Yahoo Backlinks
* Pages in Yahoo Index
* Google Backlinks
* Google Cache
* Pages in Google index
* Google Translate to English
* Google Related
* PageRank Check
* MSN Backlinks
* Pages in MSN Index
* Alexa Overview
* Alexa Traffic
* Alexa Related
* Alexa Backlinks
* "Mass Check" multiple sources at once
* Check DMOZ Inclusion
* Keyword Density
* Page Size Checker
* HTML Validator
* Server Header Viewer
* Wayback Machine
* Check robots.txt
* Whois Info
CrapWare Free! Planned for future versions:
* Yahoo & MSN Cache
* Spider Simulator and/or Text Browser view
* IP-based SEO tools
* Toolbar Option<<less
Download (7KB)
Added: 2008-10-18 License: Freeware Price:
371 downloads
SpidEye 1.0.0
SpidEye showing what a Web Crawler would see while navigating the web. more>> SpidEye is designed to be like a spiders eye, showing what a Web Crawler would see while navigating the web.
-Validation Connects to Markup Validation service provided by W3C, for both local and remote files.
-Code Shows sites source code in advanced multi PHP, (X)HTML, WML, CSS, JS syntax highlighter .
-User Agent: Spideye
-Full HTML Navigation
-Displays section of a compound document.
-List of Favorites
-Configurable Homepage
-Customizable Proxy/Port
-HTML Tags Organizes relevant bots tags: Meta, Anchors and Links.
-Text Generates a unique word list with occurrence count and calculates Text to Code ratio.
-FavIcon Displays image type besides all sizes and resolutions of the sites Favorite Icon.
-Robots.txt Analyzes robots.txt in a syntax-highlighted viewer.
-File Info Give file information and calculates file hash. If requested, can monitor text modifications per site, detecting possible unauthorized tampering.
-Documentation For each item or property displayed, a detailed information web-page is available under the Help menu.
-No installation necessary.
-Can run on USB -Flashdrive / Stick / Pen.
-Make no entries to the Windows registry.
-Leave no data on the guest computer.
-Sends no information, unless a page verification is required.<<less
Download (1.02MB)
Added: 2008-06-20 License: Freeware Price: Free
556 downloads
Other version of SpidEye
Calculates file's hash, displays all FavIcon images, analyzes robots.txt. Uses no DLLs nor external dependencies. Enhancements: Includes unspecified updatesGet Going Server 1.00.1175
Get Going Server - create and design a website with an easy to use browser based tick and with http server included more>>
Get Going Server - create and design a website with an easy to use browser based tick and with http server included Get Going Server will allow you to create and design a website with an easy to use browser based tick, type and save editing system without requiring html programming.
Free, fully functioning software program, with no installation or setup.
Your existing computer libraries and files remain unchanged and unmodified. Blaiz Enterprises prides itself on this respect function.
Main features:
- Browser driven editing/viewing environment
- Simple 3 color manager
- 15 built-in color schemes
- 18 customisible font/color/border/background/foreground styles - change text fonts, background images, foreground images (logo) etc with ease
- Image manager for easy image uploading
- Realtime fully automatic image format optimisation/conversion for smart seamless image handling
- 21 images - 8 system (e.g. favicon.ico, logo, background etc)/13 user images
- Automatic "favicon.ico" creation
- Online Contact Form (subject, name, email address and large message box fields) - send data to specified url
- Tick and set "robots.txt" creation with custom code support
- Raw html code input boxes - footer, header and column enhancement
- Save website to disk as static html pages - host website on 3rd party server
- View/save entire website settings, images and content as single encoded text block - easy backup, retrieval, transportation and setup of complete website
- Internet Color Manager support, for click and drag realtime code free color editing
- 20 customisible/renamable colors for use with raw html code
- Definable list of browser safe fonts
- Automatic text header creation
- Automatic hyperlinking of web links ( http:// )
- Easy embedding of images using insert codes [image-0]...[image-20]
- Extendable with raw html code [html]...raw code...[/html]
- System insert codes - [date], [year], [name] etc
- Keyword/description meta tag creation for search engine indexing
- Flexible editing options - font, color, images and content
- Built-in HTTP server for internet broadcast of website
- Advanced multi-mode, multi-type, multi-masking, flexible internet traffic filtration
- Supports basic security authentication (login)
- Raw activity logs
- Definable server port
- Definable network transfer rate (upload/download)
- Lightweight architecture (VUN) - minimal resource usage
- Intelligent behaviour
- Built-in help viewer with additional html/plain text external viewing
- Clone support for multiple independent instances
- Smart, built-in network management and display
- User Management (customise options/settings and appearance)
- Schemes
- Multiple-user support
- Detailed program information panels - see Help/Information tab
- Portable support for computer independent operation - no installation, no Start button placement, all files retained on source storage device (usb stick, usb disk etc)
- Integrated Help
- No installation
Version restrictions:
- a maximum 5Mb upload for each image to the server
This download is marked as adware because it displays advertisement banners or other type of commercials while running.
Note: The freeware version has an optional "Ad removal" fee, if required
<<lessFree, fully functioning software program, with no installation or setup.
Your existing computer libraries and files remain unchanged and unmodified. Blaiz Enterprises prides itself on this respect function.
Main features:
- Browser driven editing/viewing environment
- Simple 3 color manager
- 15 built-in color schemes
- 18 customisible font/color/border/background/foreground styles - change text fonts, background images, foreground images (logo) etc with ease
- Image manager for easy image uploading
- Realtime fully automatic image format optimisation/conversion for smart seamless image handling
- 21 images - 8 system (e.g. favicon.ico, logo, background etc)/13 user images
- Automatic "favicon.ico" creation
- Online Contact Form (subject, name, email address and large message box fields) - send data to specified url
- Tick and set "robots.txt" creation with custom code support
- Raw html code input boxes - footer, header and column enhancement
- Save website to disk as static html pages - host website on 3rd party server
- View/save entire website settings, images and content as single encoded text block - easy backup, retrieval, transportation and setup of complete website
- Internet Color Manager support, for click and drag realtime code free color editing
- 20 customisible/renamable colors for use with raw html code
- Definable list of browser safe fonts
- Automatic text header creation
- Automatic hyperlinking of web links ( http:// )
- Easy embedding of images using insert codes [image-0]...[image-20]
- Extendable with raw html code [html]...raw code...[/html]
- System insert codes - [date], [year], [name] etc
- Keyword/description meta tag creation for search engine indexing
- Flexible editing options - font, color, images and content
- Built-in HTTP server for internet broadcast of website
- Advanced multi-mode, multi-type, multi-masking, flexible internet traffic filtration
- Supports basic security authentication (login)
- Raw activity logs
- Definable server port
- Definable network transfer rate (upload/download)
- Lightweight architecture (VUN) - minimal resource usage
- Intelligent behaviour
- Built-in help viewer with additional html/plain text external viewing
- Clone support for multiple independent instances
- Smart, built-in network management and display
- User Management (customise options/settings and appearance)
- Schemes
- Multiple-user support
- Detailed program information panels - see Help/Information tab
- Portable support for computer independent operation - no installation, no Start button placement, all files retained on source storage device (usb stick, usb disk etc)
- Integrated Help
- No installation
Version restrictions:
- a maximum 5Mb upload for each image to the server
This download is marked as adware because it displays advertisement banners or other type of commercials while running. Note: The freeware version has an optional "Ad removal" fee, if required
Download (574KB)
Added: 2007-05-29 License: Adware Price:
878 downloads
SeoQuake 2.0.10
Firefox extension aimed primarily at helping web masters who deal with SEO and internet promotion of their web sites more>>
Firefox extension aimed primarily at helping web masters who deal with SEO and internet promotion of their web sites Seoquake is an extension aimed primarily at helping web masters who deal with search engine optimization and internet promotion of web sites. Seoquake allows to obtain and investigate many important search engine optimization parameters of the internet project under study on the fly, compare them with the results, save them for future work, obtained for other, competitive, projects.
Beginner web masters will find SeoQuake useful for estimating effectiveness and competitive ability of their SEO efforts. In the hands of professional SeoQuake will become a powerful and indispensable tool for analyzing optimization and promotion level of internet web-projects.
SeoQuake can be distinguished from similar projects by a wide range of parameters available for analysis, simplicity of use and also a tool for creation of custom parameters.
SeoQuake consists of two functional parts. The first one is SeoBar. The second part is the computation of the parameters for search results and its inclusion into search engine result pages ? SERPs ? for the most popular search engines.
List of parameters.
- Google PR
Google PageRank of current page
- Google Index
Number of indexed pages. Google version.
- Google link
Number of links, pointed to the current page. Google version (Except links from concerned domain).
- Google cachedate
Date of current Google cache.
- Yahoo Index
Number of indexed pages. Yahoo version.
- Yahoo link
Number of links, pointed to the current page. Yahoo version (Except links from concerned domain).
- Yahoo Linkdomain (LD)
Number of links, pointed to the current domain. Linkdomain (LD) operator finds pages linking to domain, as opposed to link which finds references to a single page. Yahoo version.
- Yahoo Linkdomain2 (LD2)
Number of links, pointed to the current domain (2-level domain). Linkdomain2 (LD2) operator finds pages linking to an entire domain, as opposed to linkdomain (LD) which finds references to a current domain (subdomain for example). Yahoo version.
- Yahoo Directory
Domain presence in the Yahoo Directory.
- MSN Index
Number of indexed pages . MSN version.
- MSN link
Number of links, pointed to the current page. MSN version (Except links from concerned domain).
- MSN Linkdomain (LD)
Number of links, pointed to the current domain. Linkdomain (LD) operator finds pages linking to domain, as opposed to link which finds references to a single page. MSN version.
- MSN Linkdomain2 (LD2)
Number of links, pointed to the current domain (2-level domain). Linkdomain2 (LD2) operator finds pages linking to an entire domain, as opposed to linkdomain (LD) which finds references to a current domain (subdomain for example). MSN version.
- Yandex CY
Yandex Rank of current page.
- Yandex index (russian SE)
Number of indexed pages. Yandex version.
- Yandex link
Number of links, pointed to the current page. Yandex version.
- Yandex catalogue
Domain presence in the Yandex Directory.
- Rambler index (russian SE)
Number of indexed pages. Rambler version.
- Rambler top100
Domain presence in the Rambler top100 catalogue.
- DMOZ Directory
Domain presence in the Dmoz Directory.
- Alexa Rank
Traffic estimation index
- WebArchive age
First date in Archive.org
- Del.icio.us index
Link to del.icio.us history for current url.
- Technorati index
Link to Technorati history for current url.
- Digg index
Link to Digg history for current url.
- IP lookup
Current domain IP.
- Whois link
Link to the Whois record of current domain.
- View source
Link to the source code of current web page.
- Check robots.txt
Check if robots.txt file exists and if it does show link to it.
- Page links
Display quantity of internal/external links with the link to the link list.
System requirements:
- Firefox: 2.0 ? 3.0a9
<<lessBeginner web masters will find SeoQuake useful for estimating effectiveness and competitive ability of their SEO efforts. In the hands of professional SeoQuake will become a powerful and indispensable tool for analyzing optimization and promotion level of internet web-projects.
SeoQuake can be distinguished from similar projects by a wide range of parameters available for analysis, simplicity of use and also a tool for creation of custom parameters.
SeoQuake consists of two functional parts. The first one is SeoBar. The second part is the computation of the parameters for search results and its inclusion into search engine result pages ? SERPs ? for the most popular search engines.
List of parameters.
- Google PR
Google PageRank of current page
- Google Index
Number of indexed pages. Google version.
- Google link
Number of links, pointed to the current page. Google version (Except links from concerned domain).
- Google cachedate
Date of current Google cache.
- Yahoo Index
Number of indexed pages. Yahoo version.
- Yahoo link
Number of links, pointed to the current page. Yahoo version (Except links from concerned domain).
- Yahoo Linkdomain (LD)
Number of links, pointed to the current domain. Linkdomain (LD) operator finds pages linking to domain, as opposed to link which finds references to a single page. Yahoo version.
- Yahoo Linkdomain2 (LD2)
Number of links, pointed to the current domain (2-level domain). Linkdomain2 (LD2) operator finds pages linking to an entire domain, as opposed to linkdomain (LD) which finds references to a current domain (subdomain for example). Yahoo version.
- Yahoo Directory
Domain presence in the Yahoo Directory.
- MSN Index
Number of indexed pages . MSN version.
- MSN link
Number of links, pointed to the current page. MSN version (Except links from concerned domain).
- MSN Linkdomain (LD)
Number of links, pointed to the current domain. Linkdomain (LD) operator finds pages linking to domain, as opposed to link which finds references to a single page. MSN version.
- MSN Linkdomain2 (LD2)
Number of links, pointed to the current domain (2-level domain). Linkdomain2 (LD2) operator finds pages linking to an entire domain, as opposed to linkdomain (LD) which finds references to a current domain (subdomain for example). MSN version.
- Yandex CY
Yandex Rank of current page.
- Yandex index (russian SE)
Number of indexed pages. Yandex version.
- Yandex link
Number of links, pointed to the current page. Yandex version.
- Yandex catalogue
Domain presence in the Yandex Directory.
- Rambler index (russian SE)
Number of indexed pages. Rambler version.
- Rambler top100
Domain presence in the Rambler top100 catalogue.
- DMOZ Directory
Domain presence in the Dmoz Directory.
- Alexa Rank
Traffic estimation index
- WebArchive age
First date in Archive.org
- Del.icio.us index
Link to del.icio.us history for current url.
- Technorati index
Link to Technorati history for current url.
- Digg index
Link to Digg history for current url.
- IP lookup
Current domain IP.
- Whois link
Link to the Whois record of current domain.
- View source
Link to the source code of current web page.
- Check robots.txt
Check if robots.txt file exists and if it does show link to it.
- Page links
Display quantity of internal/external links with the link to the link list.
System requirements:
- Firefox: 2.0 ? 3.0a9
Download (117KB)
Added: 2007-10-15 License: Freeware Price:
777 downloads
Site Spider 1.0.0
Spiders a site and creates output files containing (1) a list of URLs in the site, (2) a Google sitemap, (3) an XML summary file with title, description, and keywords for each page, and (4) links to U more>>
Spiders a site and creates output files containing (1) a list of URLs in the site, (2) a Google sitemap, (3) an XML summary file with title, description, and keywords for each page, and (4) links to URLs external to the spidered domain. Robots.txt compliant.
Whats new in this version: None
<<less Download (468.75KB)
Added: 2008-09-07 License: Freeware Price: $0
414 downloads
Keyword Crawler 1.1
Online website optimization tools: see your site like webengines more>> Online website optimization tools: explore and see your site like webengines. Count and display broken and internals links. Get Google pagerank for each file and generate a SiteMaps file in XML format. Keyword Crawler could also calculate word density in pages and display the metas and observes the directives of the robots.txt and robot meta tags.<<less
Download (410KB)
Added: 2005-08-29 License: Freeware Price: $0.00
1704 downloads
IntelliMapper 2.5
IntelliMapper is intelligent sitemapping software that makes it super easy to create perfect XML sitemaps quickly & easily that can be submitted to Google, Yahoo!, MSN and Ask. IntelliMapper uses a si more>>
IntelliMapper is intelligent sitemapping software that makes it super easy to create perfect XML sitemaps quickly & easily that can be submitted to Google, Yahoo!, MSN and Ask.
* Easy-to-use wizard system helps you create a site map in minutes.
* Only 3 simple steps: Enter your website address, choose a couple of settings and press Start. Thats it!
* Creates one sitemap that can be submitted to Google, Yahoo!, MSN and Ask at the same time...saving you time and effort.
* Comes with easy-to-follow manual loaded with screen shots walks you through using the software step-by-step... but you probably wont even need it :-)
* Has built-in FTP software so you can upload your finished map to your web site without leaving the IntelliMapper.
* Indexes your site quickly and shows you the indexing progress.
* Gives you a handy list of broken links on your site so you can fix them. Broken links are automatically excluded from the sitemap.
* Automatically honors robots.txt files
<<less* Easy-to-use wizard system helps you create a site map in minutes.
* Only 3 simple steps: Enter your website address, choose a couple of settings and press Start. Thats it!
* Creates one sitemap that can be submitted to Google, Yahoo!, MSN and Ask at the same time...saving you time and effort.
* Comes with easy-to-follow manual loaded with screen shots walks you through using the software step-by-step... but you probably wont even need it :-)
* Has built-in FTP software so you can upload your finished map to your web site without leaving the IntelliMapper.
* Indexes your site quickly and shows you the indexing progress.
* Gives you a handy list of broken links on your site so you can fix them. Broken links are automatically excluded from the sitemap.
* Automatically honors robots.txt files
Download (2372K)
Added: 2007-10-26 License: Freeware Price: $0.00
735 downloads
Linking The Web 2.0
Free utility to monitor backlinks to your site and email you about any problems more>>
"Linking The Web" is a free utility that regularly monitors all backlinks to your sites. If some of the links dont exist, contain wrong anchor, are covered by a "nofollow" flag or hidden by robots.txt then you will be immediately notified by email. Also the program can keep track of your purchased links and notify you about their expiration dates. The software is free of charge and works under all versions of Microsoft Windows.
<<less Download (780K)
Added: 2008-07-03 License: Freeware Price: $0
504 downloads
Other version of Linking The Web
don't exist, contain wrong anchor, are covered by a "nofollow" flag or hidden by robots.txt then you will be immediately notified by email. Also the program can keep track of your purchased linksPrice: $0.00
License:Freeware
License:Freeware
Secleted [ 0 ] software to compare
- Page: 1 of 2
- 1
- 2
Copyright Notice:
Software piracy is theft, Using crack, password, serial numbers, registration codes, key generators is illegal and prevent future software development. The above robots.txt search only lists software in full, demo and trial versions for free download. Download links are directly from our mirror sites or publisher sites, torrent files or links from rapidshare.com, yousendit.com or megaupload.com are not allowed
