Artificial intelligence-powered chatbots have become the talk of the digital space in recent times. However, their incredible capabilities have not exempted them from controversy. It is well-known that ChatGPT, for instance, took whatever it wanted from the internet without barely asking: books, websites, files, texts of all kinds… thanks to this data collection, ChatGPT is capable of interacting with humans as if it were one.
As expected, this massive data “theft” has not been well-received by major platforms. In response, a vast majority of them have decided to completely block access for AIs to their websites.
Each company has its own tools and methods to extract the necessary data from the internet. One of the tools employed by OpenAI is GPTBot, a web crawler that “could be used to improve future models” of the company. In addition to GPTBot, there are also other open-source tools such as the Common Crawl Bot.
To shed light on the matter, Originality.ai, a website specialized in detecting AI-generated content, analyzed the top 1,000 websites in the world to see which sites had started to completely block these tracking elements. In their latest discovery, it was found that some of the pages that had prohibited GPTBot’s access included Amazon, Quora, New York Times, and Wikihow, among others.
Media outlets are the most active in blocking web crawling activities. The Guardian, Bloomberg, and CNN, for instance, do not allow GPTBot to access their content. The increasing importance of data and its usage has also led to a rise in blocking Common Crawl Bot, with a 13.9% restriction rate. It’s worth noting that before the introduction of GPTBot, Common Crawl Bot was only restricted on about 5% of websites.
It’s no surprise that the online world is “protecting” itself amidst the AI frenzy. The lack of legal regulation continues to be one of the major challenges faced by this technology, which has to rely on the evolution of the industry and its needs.
