# robots.txt file # Version 2.1 - 3/27/06 # BASIC robots.txt file to reduce crawler load. # Must be named "robots.txt" and be in the root folder. # Only Google, MSN, Yahoo, and Google-Image are allowed. # Certain files excluded. # Yahoo-Slurp held back to 1 download every 45 seconds # Throttle Yahoo Slurp User-agent: Slurp Crawl-delay: 45 # Googlebot Yahoo MSN User-agent: Googlebot User-agent: Slurp User-agent: msnbot Disallow: Disallow: /blank Disallow: /BLANK Disallow: /print Disallow: /PRINT Disallow: /secure Disallow: /images # Exclude all other robots User-agent: * Disallow: / # Google Images User-Agent: Googlebot-Image Allow: /