Perplexity, OpenAI, and Anthropic under fire for ignoring robots.txt

The world of artificial intelligence (AI) is booming, with new companies and applications emerging at an astonishing rate. But behind the scenes of this exciting progress, a troubling trend is taking root. Several AI companies, including Perplexity, have been accused of scraping content from websites – essentially copying and pasting information – even when those websites explicitly tell them not to.

This blatant disregard for boundaries is raising concerns about ethics and ownership in the digital age.

The crux of the issue lies in a protocol called robots.txt. Established in 1994, robots.txt acts as a set of instructions for web crawlers, the automated programs that websites use to index content. Websites can use robots.txt to tell crawlers which pages are off-limits for scraping. While compliance with robots.txt is voluntary, it’s a well-respected norm within the web development community.

Here’s where things get messy. Perplexity, a company offering a free AI search engine, has been accused of scraping content from Forbes, Wired, and The Shortcut, even though these websites clearly indicated “no scraping” zones in their robots.txt files. This raises a big question: why would Perplexity, or any AI company for that matter, risk damaging their reputation by blatantly ignoring these protocols?

The answer lies in the data itself. Websites are treasure troves of information, and for AI companies, this information is the fuel that drives their technology. Text and data scraped from websites are used to train AI models, making them better at tasks like generating text, translating languages, or answering questions.

However, scraping copyrighted content without permission is not only unethical, but it can also have legal ramifications. In the case of Perplexity, their AI tool was caught generating content that closely resembled scraped articles, with minimal attribution and sometimes even factual inaccuracies. This raises serious concerns about the quality and reliability of AI-generated information.

The plot thickens further with the revelation from TollBit, a startup that connects publishers with AI firms. According to TollBit, big names in the AI industry, like OpenAI (creators of ChatGPT) and Anthropic (creators of Claude), have also been bypassing robots.txt restrictions. These companies previously claimed to respect “do not crawl” instructions, making their actions even more hypocritical.

Perplexity’s CEO, Aravind Srinivas, attempted to defend his company’s actions by downplaying the importance of robots.txt. He argued that it’s not a legal framework and suggested a need for a “new kind of relationship” between publishers and AI companies. This line of reasoning is concerning, as it suggests a disregard for established norms and a desire to operate in a grey area.

The larger concern here is the potential erosion of trust between content creators and AI companies. If AI companies can’t be held accountable for respecting basic boundaries like robots.txt, it creates an environment where content creators are constantly at risk of having their work stolen and repurposed without proper credit or compensation.

The future of AI is undoubtedly bright, but it needs to be built on a foundation of ethical practices and respect for intellectual property. As AI technology continues to evolve, it’s crucial to establish clear guidelines and regulations that ensure responsible data collection and utilization. This will not only protect the rights of content creators but also foster a more sustainable and trustworthy environment for AI development.

Related /