# robot exclusion for site
User-agent: Slurp
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: *
Disallow: /dev
User-agent: *
Crawl-delay: 15
User-Agent: Charlotte
Disallow: /
User-agent: Speedy
Crawl-Delay: 60