Home > Features >

Spidering Control
You can specify exactly which pages to spider, as well as which ones not to.  You can specify which pages to include with your Primary URL and Secondary URLs settings.  Just as easily, you can list pages that you don't want to be included within Exclusion URLs/Masks.
You can also determine whether or not you want the siteLevel spider to obey your site's robots.txt file.
With siteLevel, you can also spider password protected areas of your site.  You simply need to provide a username and password for the given area(s) within the Password Authenication setting of your account management area.
NOTE: If you do choose to spider a password protected area of your site, visitors to your site will still be required to have a valid username and password to enter the protected area.  If a search result for a protected area comes up from your site search box and a user clicks on the result, they will be prompted, as before, for their username and password.

Sign Up

It just takes a few easy minutes to get your own site search engine.

Simply fill in the information below to get started:

Member Login

Having trouble logging in?

© Copyright 2024, whatUseek, Corp.  Terms  Privacy Policy  Feedback