User-agent: AhrefsBot User-agent: MJ12bot user-agent: rogerbot User-agent: BacklinkCrawler User-agent: SemrushBot User-agent: Seomoz User-agent: rogerbot User-agent: dotbot User-agent: Semrush User-agent: Majestic12 User-agent: Majestic-SEO User-agent: Screaming Frog SEO Spider Disallow: / User-agent: * Disallow: /admin/ Allow: /*.js$ Allow: /*.css$