Block search engine spiders

Many people are scared of spiders but when it comes to the kind sent by search engines, webmasters are generally happy to let them crawl all over their site. Of course, it’s a good thing for your ranking to have these little pieces of code looking at your pages regularly to check for updates. Recently though, site administrators have been getting a little ticked off with Yahoo!’s new spidering tactics. By using a different IP address for each page the search giant checks, many people are now finding their bandwidth being choked.

It looks as if Yahoo! isn’t in a rush to change its new spidering techniques, but thankfully, if you’re being blighted by a plague of spiders there are measures you can take to block them. One measure is to use the following meta tag to talk to the spiders: “” You will need this META tag on every page in your Web site that you don’t want indexed. Obviously if you have a lot of pages on your site this may take a while, so grab yourself a decent HTML editor like Arachnophilia, which allows you to perform a global search and replace and add an HTML header tag to every Web page that you open in the program.

An alternative to this method is to download one of the spider-blocking scripts from TechPatterns to help you cut down on the amount of traffic search engines like Yahoo! are sending to your site.