When looking at our who's online link, it's usually filled with "crawlers" snooping around our site. Here are a couple of links to show how this snooping is being put to use.

http://www.webcrawler.com/webcrawler303/ws/results/Web/movies+john+wayne+died+in/1/417/TopNavigation/Relevance/iq=true/zoom=off/_iceUrlFlag=7?_IceUrl=true
(http://www.webcrawler.com/webcrawler303/ws/results/Web/movies+john+wayne+died+in/1/417/TopNavigation/Relevance/iq=true/zoom=off/_iceUrlFlag=7?_IceUrl=true)
http://www.ask.com/web?qsrc=999&l=dis&siteid=15141&o=15141&q=movies+john+wayne+died+in&sq=1&ifr=1

Once these crawlers find bits of new information , they put it on their site as a link, at the same time charging their advertisers for this service. I haven't quite figured out if this is good for a site like ours, or not.

Any ideas?

Chester :newyear:

I call them pirates. You can block them with a robots.txt file uploaded to your web site folder. PM me if you need more info.

Bob

I think they ought to be shot at sunrise. As far as im concerned, this is only thievery on their part.

Crawlers are okay as long as they don't slow down the server. I see them as a necessary evil. We want to be indexed everywhere.

There are two types, the ones you want and the ones you don't want. Search engine spiders such as Google, Yahoo and the such are good, as they index your site so folk can find you when they do a web search with their favourite search engine.

Then there are the ones such as BoardTracker and BoardReader. They index your site and show your messages on their site and they get advertising revenue from their sponsors. These are the ones I call pirates. If you have set things up properly you don't need these, the search engine spiders will do all the work for you.

I have one of the top 3 Volvo car club forums in the world and have blocked these 'pirates'. We get the revenue from our advertisers, not them. Parasites.

But, it is every forum admin's choice.

Bob