# Robots.txt blocks various web-crawlers from accessing certain directories # Most modern spiders (yahoo, ask, google) will obey the directives below # Sample syntax # #User-agent: SPIDER_NAME #Disallow: DIRECTORY/FILE LIST # User-agent: * Disallow: /gallery