United Bimmer Community - BMW Forum - View Single Post - Yahoo Slurp Crawler.... not playing by the rules?

**jms** · 03-09-2006, 09:23 AM

yep, that would do it. although some crawlers do ignore the robots.txt file, though 99% do, every once in a while there's one that does. you might want to add some referrer check to it to prevent a loop.

03-09-2006, 09:23 AM	#6
jms Name: jms Title: ______ Status: Offline Join Date: Jul 2005 Location: Pittsburgh,PA Rate My Car: 105 / 340 Your Ride: 99 328I Convertible	yep, that would do it. although some crawlers do ignore the robots.txt file, though 99% do, every once in a while there's one that does. you might want to add some referrer check to it to prevent a loop.