By The escalating debate over the use of news content to train AI models has prompted many news publishers to block ChatGPT from scanning their sites. A survey by journalist Ben Welsh revealed that almost half of the 1,148 news publishers surveyed, including Google AI and Common Crawl, have implemented such restrictions. Criticisms have arisen regarding the effectiveness of blocking options, with new features like ‘Google-Extended’ not fully preventing content usage for generative AI.
High-profile news organizations in the UK and US, such as the BBC, New York Times, CNN, and Reuters, block OpenAI, while blocking is less widespread in other parts of the world, as per data from SEO consultant Gary Kirwan. This study aimed to analyze how news sites’ blocking of the OpenAI crawler bot impacts ChatGPT’s responses.
Results varied across 24 news brands, with 11 providing full responses, 11 offering limited or no responses, and two exhibiting responses subject to change. In Brazil, ChatGPT accessed headlines from unblocked sites like Globo News but was restricted from UOL and Record News. In South Africa, it successfully accessed headlines from some sources but faced challenges with BBC News. In the UK, the BBC and The Guardian presented varied responses, while MailOnline prompted alternative suggestions. Unexpected responses, including varied explanations for restricted access, were observed, and responses differed even for sites allowing crawling by OpenAI’s bots
