I'd like to prevent search bots from reading the context of my site, or at least from following "folder archive" references.
The most bothering is Google bot, it follows "folder archive" reference, downloads about 15 megabytes, interrupts downloading and then repeats this operation several times each day.
Now my solution is to ban google bots IPs, but as far as I remember there is a possibility to add something like "metaname=robot, nofollow" to references.