IAB Tech Lab Launches AI Working Group to Combat Content Scraping
The Interactive Advertising Bureau Technology Laboratory (IAB Tech Lab) has launched a new working group to tackle the challenges posed by artificial intelligence (AI) in the digital publishing landscape. The Content Monetization Protocols (CoMP) for AI Working Group aims to help content creators and businesses navigate the evolving digital ecosystem and capitalize on new engagement and monetization opportunities.
The initiative responds to the growing threat of AI technologies, which have been causing dramatic traffic reductions and uncompensated content scraping. According to TollBit's Q4 2024 report, simultaneous bot traffic has surged by 117% quarter-over-quarter, with sites scraped an average of 5.05 million times. Publishers are estimated to face $2 billion in annual ad revenue losses due to AI systems providing answers directly, bypassing source websites.
The working group, formed on August 20, 2025, will focus on standardizing various aspects to enable publishers to exercise control and enforce terms of use at scale. This includes standardizing bot and agent access controls, content discovery mechanisms, Cost per Crawl monetization APIs, comprehensive LLM Ingest APIs, and NLWeb integration protocols. The framework establishes three core mechanisms for publisher content monetization: content access controls, LLM-friendly discovery, and monetization options including Cost per Crawl (CPCr) pricing and LLM Ingest APIs. The group was established following a workshop that drew 80 executives from publishers, edge cloud providers, and AI monetization startups.
The IAB Tech Lab's CoMP for AI Working Group is a significant step towards helping publishers adapt to the AI-driven digital ecosystem. By standardizing various protocols and mechanisms, the group aims to help publishers capitalize on new engagement and monetization opportunities, ensuring the economic sustainability of digital publishing.
