Cloudflare's new technology takes attack to AI 'crawlers'

Dubai: The attack is now being taken to the AI ‘crawlers’.

The US company Cloudflare recently confirmed its solutions are blocking AI crawlers from accessing content without permission – or compensation. This way, the company says it is the first Internet infra provider to do so.

Website owners can now get to choose if they want AI crawlers to access their content, and decide how AI companies can use it.

“AI companies can also now clearly state their purpose - if their crawlers are used for training, inference, or search - to help website owners decide which crawlers to allow,” said a statement.

The Internet has always operated by way of search engines indexing content and directing users back to the original websites. “This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information,” said Cloudflare.

“That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”