← All news

Artificial Intelligence

Google and Microsoft among those signing up to OpenAI’s plan for a new website crawling standard — including opting-out of internet scraping

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won’t stop their bots from crawling it.

Read the full story on TechRadar.

Read at source (opens in a new tab)

Latest from TechRadar

Latest digest

Week 44 EN

Open the digest

Sources

69 active sources

24 Turkish · 45 English

All sources