0 Members and 1 Guest are viewing this topic.
Quote from: rejetto on December 03, 2007, 01:52:15 AMi asked what's exactly referring to, not the meaning of +1 itself.Stopping spiders from following 'folder archive' links and downloading data, thus wasting precious resources. Your 'stop spiders' option would stop them completely.
i asked what's exactly referring to, not the meaning of +1 itself.
Quote from: MarkV on December 03, 2007, 05:23:38 AMQuote from: rejetto on December 03, 2007, 01:52:15 AMi asked what's exactly referring to, not the meaning of +1 itself.Stopping spiders from following 'folder archive' links and downloading data, thus wasting precious resources. Your 'stop spiders' option would stop them completely.does anyone know how the robots.txt should be to get this job done?