rejetto forum

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Gorlas

Pages: 1
HFS ~ HTTP File Server / spiders
« on: May 12, 2009, 10:56:08 AM »
I have done a bit of research on how search engines spider sites.
I discovered that they make rapid connections unless you tell them not to.
I use a robots.txt file on my site which can be used to slow those down or tell the bots not to index your site at all.
I used to just block all the ip's that did that until I ran a whois lookup on some of the addresses and found out they were search engines. I unblocked the search engines ( I may regret that over time as my traffic increases ) so they could spider the site.

Hope this helps clear up the whys of some of your rapid connection attempts.

Everything else / How about a new section for Site Ads
« on: May 01, 2009, 06:39:45 AM »
Just curious, I haven't seen an area for site ops that would like to have others see and leave feed back on their sites.
Let me start with mine.

I'd like to know what you think.


Pages: 1