Thread
:
Yahoo Slurp Crawler.... not playing by the rules?
View Single Post
03-09-2006, 09:23 AM
#
6
jms
Name:
jms
Title:
______
Status:
Offline
Join Date:
Jul 2005
Location:
Pittsburgh,PA
Rate My Car:
105 / 340
Your Ride:
99 328I Convertible
yep, that would do it. although some crawlers do ignore the robots.txt file, though 99% do, every once in a while there's one that does. you might want to add some referrer check to it to prevent a loop.
jms
View Public Profile
Send a private message to jms
Visit jms's homepage!
Find More Posts by jms