rejetto forum

Feature request: stop search bots

CBB · 20 · 7036

0 Members and 1 Guest are viewing this topic.

Offline rejetto

  • Administrator
  • Tireless poster
  • *****
    • Posts: 13310
    • View Profile
i asked what's exactly referring to, not the meaning of +1 itself.

Stopping spiders from following 'folder archive' links and downloading data, thus wasting precious resources. Your 'stop spiders' option would stop them completely.

does anyone know how the robots.txt should be to get this job done?


Offline Foggy

  • Tireless poster
  • ****
    • Posts: 806
    • View Profile
maybe something like /*/~folder.tar under the not allow section?


Offline CBB

  • Occasional poster
  • *
    • Posts: 21
    • View Profile
i asked what's exactly referring to, not the meaning of +1 itself.

Stopping spiders from following 'folder archive' links and downloading data, thus wasting precious resources. Your 'stop spiders' option would stop them completely.

does anyone know how the robots.txt should be to get this job done?

I meant here to change attributes of references in the default template, not to add robots.txt. It would be usefull to add rel='nofollow" to "folder archive" references, see http://googleblog.blogspot.com/2005/01/preventing-comment-spam.html or http://en.wikipedia.org/wiki/Nofollow .


Offline rejetto

  • Administrator
  • Tireless poster
  • *****
    • Posts: 13310
    • View Profile
AFAIK nofollow doesn't stop spiders.
stops the rank propagation.
i think it has truly no meaning on folder archives.


Offline day

  • Occasional poster
  • *
    • Posts: 65
    • View Profile