# Robots.txt file from https://www.sci99.com User-agent: SemrushBot User-agent: YandexBot User-agent: AhrefsBot User-agent: DotBot User-agent: BLEXBot User-Agent: Linguee User-Agent: trendictionbot User-agent: MJ12bot User-agent: FISPIDER Disallow: / User-agent: * Crawl-delay: 10 Disallow: /info/ Disallow: /mobile/info/