Specifications

Summary

• Robots.txt
  • Google's Robots.txt Specifications
  • The Web Robots Pages
  • W3C Recommendation
  • Yandex
• HTML tags
  • Google's Robots meta tag and X-Robots-Tag HTTP header specifications
  • The Web Robots Pages
  • W3C Recommendation
  • Yandex
• HTTP header
  • Google's Robots meta tag and X-Robots-Tag HTTP header specifications

Details

• Google's Robots.txt Specifications

  Supported

  • Robots.txt

• Google's Robots meta tag and X-Robots-Tag HTTP header specifications

  Unsupported

  • HTML tags
    • Unsupported: <META> tags (Link)
  • HTTP header
    • Unsupported: X-Robots-Tag HTTP header (Link)

• The Web Robots Pages

  Partial support

  • Robots.txt
  • HTML tags
    • Unsupported: <META> tags (Link)

• W3C Recommendation

  Partial support

  • Robots.txt
  • HTML tags
    • Unsupported: <META> tags (Link)

• Yandex

  Partial support

  • Robots.txt
  • HTML tags
    • Unsupported: <META> tags (Link)
    • Unsupported: <noindex> tags (Link)
    • Unsupported: Attribute rel="nofollow" of the <a> tag (Link)