rejetto forum

spiders

Gorlas · 1 · 2444

0 Members and 1 Guest are viewing this topic.

Offline Gorlas

  • Occasional poster
  • *
    • Posts: 8
    • View Profile
    • Ancient World Archives
I have done a bit of research on how search engines spider sites.
I discovered that they make rapid connections unless you tell them not to.
I use a robots.txt file on my site which can be used to slow those down or tell the bots not to index your site at all.
I used to just block all the ip's that did that until I ran a whois lookup on some of the addresses and found out they were search engines. I unblocked the search engines ( I may regret that over time as my traffic increases ) so they could spider the site.

Hope this helps clear up the whys of some of your rapid connection attempts.
« Last Edit: May 12, 2009, 11:07:52 AM by Gorlas »