rejetto forum

Feature request: stop search bots

CBB · 20 · 7037

0 Members and 1 Guest are viewing this topic.

Offline CBB

  • Occasional poster
  • *
    • Posts: 21
    • View Profile
I'd like to prevent search bots from reading the context of my site, or at least from following "folder archive" references.
The most bothering is Google bot, it follows "folder archive" reference, downloads about 15 megabytes, interrupts downloading and then repeats this operation several times each day.
Now my solution is to ban google bots IPs, but as far as I remember there is a possibility to add something like "metaname=robot, nofollow" to references.


Offline maverick

  • Tireless poster
  • ****
    • Posts: 1052
  • Computer Solutions
    • View Profile
Add this to the head section of your template ....

<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

Add attached file to the root of your vfs and hide.

That's it!
« Last Edit: December 02, 2007, 05:17:11 PM by maverick »
maverick


Offline Tuskenraider

  • Occasional poster
  • *
    • Posts: 74
    • View Profile
you can also place a robots.txt file in the root directory... like mine is www.aguyincookeville.com/robots.txt  and that will stop em as well!! i also have issue with robots.. darn goggle, yahoo and ask.com robots.. but this text file has pretty well stopped them.  enjoy!

tuskenraider
pfffssshhh i dont need a signature...


Offline maverick

  • Tireless poster
  • ****
    • Posts: 1052
  • Computer Solutions
    • View Profile
Tuskenraider

Doesn't look like your read my reply above ???
maverick


Offline CBB

  • Occasional poster
  • *
    • Posts: 21
    • View Profile
Thank you, I've fulfilled your recomendations.
But I also think that preventing bots to follow "folder archive" references should be done in the default template.


Offline Tuskenraider

  • Occasional poster
  • *
    • Posts: 74
    • View Profile
Tuskenraider
Doesn't look like your read my reply above ???

doh... sorry man.... ive not had my morning coffee.. many apologies!

Tusken-where the hecks my coffee cup-Raider
pfffssshhh i dont need a signature...


Offline maverick

  • Tireless poster
  • ****
    • Posts: 1052
  • Computer Solutions
    • View Profile
I also think that preventing bots to follow "folder archive" references should be done in the default template.

I disagree.  It's a matter of personal preference based on the contents of the site.  There might be some admins that want their sites spidered.
maverick


Offline rejetto

  • Administrator
  • Tireless poster
  • *****
    • Posts: 13310
    • View Profile
Next beta will include a "stop spiders" option.
It just will serve this standard robots.txt file (but only if there's no such file in the file system).

I'm gonna make this option ON by default, and hidden while in "easy mode". Any opinion is welcome.


Offline CBB

  • Occasional poster
  • *
    • Posts: 21
    • View Profile
I also think that preventing bots to follow "folder archive" references should be done in the default template.

I disagree.  It's a matter of personal preference based on the contents of the site.  There might be some admins that want their sites spidered.
Please take into attention that here I mean exclusively "folder archive" references, not other references or directories, so my proposal will not lead to obstacles of site spidering, but only to traffic diminishing.


Offline CBB

  • Occasional poster
  • *
    • Posts: 21
    • View Profile
Next beta will include a "stop spiders" option.
It just will serve this standard robots.txt file (but only if there's no such file in the file system).

I'm gonna make this option ON by default, and hidden while in "easy mode". Any opinion is welcome.

I support this decision, it seems to be very reasonable.


Offline MarkV

  • Tireless poster
  • ****
    • Posts: 763
    • View Profile
I also think that preventing bots to follow "folder archive" references should be done in the default template.

I disagree.  It's a matter of personal preference based on the contents of the site.  There might be some admins that want their sites spidered.
Please take into attention that here I mean exclusively "folder archive" references, not other references or directories, so my proposal will not lead to obstacles of site spidering, but only to traffic diminishing.

+1
http://worldipv6launch.org - The world is different now.


Offline rejetto

  • Administrator
  • Tireless poster
  • *****
    • Posts: 13310
    • View Profile

Offline Foggy

  • Tireless poster
  • ****
    • Posts: 806
    • View Profile

Offline rejetto

  • Administrator
  • Tireless poster
  • *****
    • Posts: 13310
    • View Profile
i asked what's exactly referring to, not the meaning of +1 itself.


Offline MarkV

  • Tireless poster
  • ****
    • Posts: 763
    • View Profile
i asked what's exactly referring to, not the meaning of +1 itself.

Stopping spiders from following 'folder archive' links and downloading data, thus wasting precious resources. Your 'stop spiders' option would stop them completely.
http://worldipv6launch.org - The world is different now.