User-agent: Amazonbot Disallow: /do-not-crawl/ User-agent: SemrushBot Disallow: /