Should Publishers Allow LLMs to Crawl Content?
In this post, we’ll explore how trade publishers can balance paywalls and AI visibility, and how to maintain traffic while making informed choices.
Introduction: The Impact of LLMs on Publishing
The rapid rise of Large Language Models (LLMs) is reshaping the publishing world, especially when it comes to search and content discovery. With tools like Google’s AI Overviews and other advanced language models, users are now getting direct answers in search results, often without needing to click through to a publisher’s website. This shift has led to a surge in zero-click searches, challenging traditional SEO strategies and making it harder for publishers to drive traffic the old-fashioned way. In this new environment, high-quality content is more important than ever for maintaining visibility in search results. Publishers must rethink their strategies, focusing on how LLMs interpret and surface content, to ensure their work remains discoverable and valuable to users. Understanding the evolving role of LLMs in search is now essential for any publisher looking to stay relevant and competitive.
The Role of Google in Search and Content Discovery
Google continues to be the primary gateway for online discovery, and its introduction of AI Overviews has fundamentally changed how search results are delivered to users. With AI Overviews, Google’s search engine now provides users with instant answers, often summarizing information from multiple sources without requiring a click to the original website. For publishers, this means that high-quality, well-structured content is crucial—not just for users, but for LLMs that power these overviews. While this approach can reduce direct traffic, it also highlights the need for publishers to create content that stands out in a crowded ecosystem. By ensuring their content is authoritative, relevant, and easily understood by both users and LLMs, publishers can maintain visibility and continue to reach their audience, even as the search landscape evolves.
Understanding the Four Types of Publisher Content
Let’s break down a typical trade publisher’s content into four categories, considering how content is organized into different pages and distributed across various platforms:
- Premium Editorial Content: This is the high-quality journalism behind the paywall, typically accessed through dedicated pages on the publisher’s website. It’s the core value that subscribers pay for.
- News Content: Often a mix of free and gated articles. These articles may appear on different pages or even across multiple sites, and publishers can choose how much of this they want to expose.
- Other Content (Directories, Events, Newsletters): This can be used flexibly to draw in audiences and is often distributed across various platforms and websites to maximize reach.
- Web Exclusives or Free Content: A top-of-funnel category that can be fully open to attract new visitors to the site. These are often featured on landing pages or shared across platforms to increase visibility.
How to Balance Paywalls and LLM Accessibility
We recommend that publishers control access to their premium content by keeping it behind a paywall. But for that premium tier, consider providing excerpts or summaries that LLMs can use for training and to generate answers. Making these excerpts crawlable helps LLMs get the information they need to answer user queries. By controlling what LLMs can access and use, publishers can get more users to visit their site while still protecting premium content.
Using Structured Data to Ensure Source Attribution
While there’s no absolute guarantee, using structured data and schema markup helps ensure that links to your site are included in AI-generated summaries. Google’s engines and other AI-driven engines rely on structured data to identify sources and provide proper attribution. There are several ways to increase the likelihood of attribution, including optimizing links and structured data.
Exploring Alternative Revenue Streams
As AI Overviews and zero-click searches become more prevalent, publishers can no longer rely solely on traditional search referral traffic to sustain their business. To adapt, it’s essential to explore alternative revenue streams that go beyond the old models. Building a strong brand and fostering direct relationships with audiences through email newsletters or exclusive digital communities can help publishers maintain engagement and loyalty. Offering premium content or specialized journalism that delivers unique value can attract users willing to pay for high-quality information. By diversifying their strategies and focusing on high-value, niche content, publishers can create new sources of revenue and reduce their dependence on search-driven clicks. This approach not only supports quality journalism but also helps publishers thrive in a changing digital landscape.
Conclusion: The Future of Search and Publishing
Looking ahead, the future of search and publishing will be shaped by the continued evolution of LLMs and AI-driven platforms. Publishers who prioritize high-quality content, embrace strategic SEO, and innovate in how they engage audiences will be best positioned for success. Building trust and a strong brand will become even more important as traditional click-through rates decline and users increasingly rely on AI-generated answers. By focusing on user needs and adapting to the LLM-driven search ecosystem, publishers can ensure their relevance and growth. The integration of AI in search is not just a challenge—it’s an opportunity for publishers to redefine their role in the digital world, deliver value to their audiences, and build a sustainable future for quality content.
Making an Informed Choice
In the end, it’s all about balancing visibility with revenue. By structuring your content and using best practices, you can let LLMs help with discoverability without giving away the farm. It’s a strategic choice that each publisher can tailor to their needs.
Having a clear strategy is essential for publishers to thrive in the evolving digital landscape. Strong brands and media organizations that focus on real, high-quality content are better positioned for long-term success. The first priority should be to protect your core value, but publishers may still benefit from allowing limited access to LLMs. What works for one publisher may not work for another, so decisions should be based on each publisher’s unique situation. As one industry expert said, “Over time, adapting your approach and distributing content across multiple platforms is key.” Some traditional SEO tactics still work, but a better, more adaptive approach is needed for the long run.
I drive results for magazine publishers.
Experienced professional with a demonstrated history of working in the information technology and services industry. Skilled in internet engineering, marketing, business development and digital publishing.
