You are currently viewing our boards as a guest which gives you limited access to view most discussions and access our other features. By joining our free community you will have access to post topics, communicate privately with other members (PM), respond to polls, upload content and access many other special features. Registration is fast, simple and absolutely free so please, join our community today!
If you have any problems with the registration process or your account login, please contact contact us.
Geek ChatAsk computer questions or just engage in general geek talk on here
Whoa... I just caught Slurp in somewhere it shouldn't be...
Now, United Bimmer's robots.txt file has this in it first of all:
[code]Disallow: /forums/printhread.php[/code]
And second of all, the ONLY link to that page is from the actual thread itself, and that link has a rel="nofollow" tag in it, so even if I didn't have the robots.txt entry, it still should follow it.
yep, that would do it. although some crawlers do ignore the robots.txt file, though 99% do, every once in a while there's one that does. you might want to add some referrer check to it to prevent a loop.
Yeah, I watch all the bots that crawl us very closely (mostly to ensure performance and efficiency), as right now there's only 12, so it's easy to keep tabs on them.
If I ever see one violating robots.txt, I'd probably ban it... but we get 65% of all our search engine traffic from Yahoo, so I'd rather keep it happy.
03-08-2006, 11:28 PM
Similar Threads
