📡 Breaking news
0/0
Analyzing latest trends...
AI Text-to-Speech.

Google Microsoft OpenAI and Anthropic Agree on Unified AI Training Opt-Out Standard.

-
Google Microsoft OpenAI and Anthropic Agree on Unified AI Training Opt-Out Standard.
Top AI Giants Google, Microsoft, OpenAI, and Anthropic Unite on Standardized

In a major joint initiative aimed at respecting content creator consent and standardizing web-scraping rules, leading American artificial intelligence developers Google, Microsoft, OpenAI, and Anthropic have jointly pledged to honor a unified robots.txt instruction. Under the new agreement, all four platforms pledge to cease web crawling and training data extraction from websites that explicitly opt out via standardized site directives.

Technical Specification, Developer Consensus, and Deployment Schedule

  • Standardizing Web Crawling Directives via IETF Framework:

    • Eliminating Webmaster Confusion: Previously, AI companies utilized fragmented user-agent tags and custom directives (such as GPTBot, Google-Extended, or ClaudeBot), causing widespread confusion for webmasters attempt to block training bots.

    • IETF Draft Specification Standard: The four tech giants agreed to implement a standardized directive defined under an upcoming Internet Engineering Task Force (IETF) draft specification.

    • Implementation Syntax Example: Web publishers can configure their root robots.txt file using the unified User-Agent: __usage_ai-training parameter to control access:

HTTP
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

  • Rollout Timelines Across Partner Platforms:

    • OpenAI: Confirmed support for the standardized tag within the coming months.

    • Microsoft (Bing/Copilot): Announced full integration starting February 1, 2027.

    • Google: Committed to complete rollout across its crawling infrastructure within 2027.

 

Source: IETF 

💬 AI Content Assistant

Ask me anything about this article. No data is stored for your question.

Comments

Post a Comment