Show Posts
1
HFS ~ HTTP File Server / spiders
« on: May 12, 2009, 10:56:08 AM »
I have done a bit of research on how search engines spider sites.
I discovered that they make rapid connections unless you tell them not to.
I use a robots.txt file on my site which can be used to slow those down or tell the bots not to index your site at all.
I used to just block all the ip's that did that until I ran a whois lookup on some of the addresses and found out they were search engines. I unblocked the search engines ( I may regret that over time as my traffic increases ) so they could spider the site.
Hope this helps clear up the whys of some of your rapid connection attempts.
I discovered that they make rapid connections unless you tell them not to.
I use a robots.txt file on my site which can be used to slow those down or tell the bots not to index your site at all.
I used to just block all the ip's that did that until I ran a whois lookup on some of the addresses and found out they were search engines. I unblocked the search engines ( I may regret that over time as my traffic increases ) so they could spider the site.
Hope this helps clear up the whys of some of your rapid connection attempts.
