October 16, 2026

# The paywall paradox: why gating content loses readers, and what AI agents change

The paywall paradox is the trade publishers make when they put articles behind a paywall, a gate that requires payment for full access. Publishers collect revenue from the few readers who pay, yet they lose the audience and influence that made the journalism worth paying for. Most readers who hit a paywall look for the information elsewhere instead of paying.

Author: By Parallel

## What is the paywall paradox?

The paywall paradox is the trade publishers make when they put articles behind a _paywall_, a gate that requires payment for full access. Publishers collect revenue from the few readers who pay, yet they lose the audience and influence that made the journalism worth paying for. Most readers who hit a paywall look for the information elsewhere instead of paying.

The paradox now has a second layer, because AI agents read pages on behalf of people. Publishers gate content and block AI crawlers to stop free extraction. Researchers at Rutgers Business School and The Wharton School measured the traffic cost in their preprint, the Rutgers and Wharton study on blocking AI crawlers[Rutgers and Wharton study on blocking AI crawlers]. Large publishers lost about 7% of their weekly visits within six weeks of blocking, and the authors suggest blocking “may reduce a publisher’s presence in AI-generated responses and summaries.”

We see the paradox as structural. In The Web’s Second User[The Web’s Second User], our August 2025 manifesto, we argued that business models built on human attention can’t serve AIs. By default, we wrote, the web trends toward “paywalls, gated APIs, and private data silos.”

We build for AI consumers and publishers at once, and we think the way out starts with paying sources for what they contribute.

## What readers do when they hit a paywall

Paywalls are a routine part of reading news online, and payment at the gate is rare. Pew Research Center measured the gap in a March 2025 survey of US adults. The Reuters Institute tracks the same question across 20 countries, and both surveys rely on what respondents report.

### What do most Americans do when they encounter a paywall?

Pew polled 9,482 US adults in March 2025 for its Pew Research Center survey on paying for news[Pew Research Center survey on paying for news], published June 24, 2025. In that survey, 74% of adults say they encounter paywalls at least sometimes. And 38% say they hit them “often” or “extremely often.”

Pew then asked the people who encounter paywalls about their first reaction. Among them, 1% say paying is their typical first move, 53% look for the information elsewhere, and 32% give up. The answers are self-reported, and they describe the first thing respondents say they do.

For a publisher, the default result of a paywall is a lost reader or a reader you hand to a free competitor. The 53% who look elsewhere deserve the most attention. Their choice depends on substitution, and the same logic applies when an AI agent picks its sources.

### Why do Americans choose not to pay for online news?

The same survey found that 83% of US adults didn’t pay for news in the past year. The 17% who did include people who donated to a news outlet or joined as members, since Pew counted those forms of support as paying.

Pew asked the people who didn’t pay for their main reason. Of that group, 49% say plenty of free news is available and 32% say they aren’t interested enough. Another 10% say news costs too much, and 8% say it isn’t good enough.

Price ranks third on that list, far behind free alternatives. In our reading, substitution drives the paradox, because a paywall competes with the free pages that cover the same story.

### Which groups are most likely to pay for news?

Pew’s data show that paying for news concentrates in a narrow slice of the public, with wide gaps by group.

- **Education:** 27% of college graduates paid for news, triple the share of adults with a high school diploma or less.
- **Income:** 30% of adults with the highest incomes paid, compared with 8% of those with the lowest.
- **Age:** 25% of adults 65 and older paid, compared with 12% of adults ages 18–29.
- **Party:** 21% of Democrats and leaners paid, compared with 14% of Republicans and leaners.

The pattern holds outside the US. The Reuters Institute Digital News Report 2026[Reuters Institute Digital News Report 2026], published June 16, 2026, covers 20 richer countries. Across that group, 17% of people paid for online news, down from 18% in 2025, with Norway at 40% and Sweden at 32%.

The authors describe 10–20% as “a ceiling in most markets.” A paywall strategy that depends on this group serves an audience that skews older and wealthier than the public. The report frames subscription growth as stalled, so each new gate competes for the same small pool of payers.

## How paywall models trade reach for revenue

Publishers use five common paywall models. Each one sets a different point on the curve between reach and subscription revenue.

Model	How it works	What it trades
Hard paywall	The publisher gates all content	Gives up most reach in exchange for revenue from committed subscribers
Metered	Readers get a quota of free articles, then hit the gate	Keeps casual readers and asks frequent readers to subscribe
Freemium	The publisher leaves some articles free and gates premium ones	Opens commodity coverage and charges for exclusive work
Lead-in	Readers see the first part of each article before the gate	Offers a preview to readers and search engines and holds back the full text
Dynamic	The gate adapts to reader signals such as visit frequency	Raises conversion per reader at the cost of complexity for the publisher

Google describes two of these models in Google’s flexible sampling guidance[Google’s flexible sampling guidance], which Google Search Central updated on December 10, 2025. Google defines metering as giving users “a quota of articles to consume before requiring users to subscribe.” The guidance covers lead-in sampling as well, which shows part of an article before the gate.

For most daily news publishers, Google suggests 6–10 free articles per user per month. Google’s user studies found that satisfaction “starts to degrade significantly when paywalls are shown more than 10% of the time.” Google doesn’t publish the underlying data, so read the threshold as guidance.

Read together, Google’s numbers favor leaving most visits open. Each model in the table picks a spot on the reach and revenue curve, and none of them escapes the trade. A generous meter keeps reach, and a hard wall keeps revenue.

## The costs paywalls push onto the open web

A paywall shapes who can afford to read and how far a story travels. Publishers weigh both costs, and in some cases they decide the gate does more harm than good.

We track these costs because the open web sparked the AI revolution. Each new gate narrows the pool of sources that people and agents can read and cite.

### How do paywalls affect the digital divide?

Yonas Kassa makes the access argument in a short position paper posted to arXiv in 2024 (2409.00026). His Income-Paywall Expenditure Gap analysis[Income-Paywall Expenditure Gap analysis] compares income per capita with the cost of paying for access, a measure he calls _IPEG_. In a hypothetical web with a paywall on every site, Kassa estimates that 135 countries with 6.56 billion people couldn’t afford access.

Treat that figure as a thought experiment, since Kassa wrote the paper alone and models a scenario that doesn’t exist today. He concedes that his assumptions are crude. He argues for a side, too, making the case that advertising remains the scalable way to keep the web open.

The direction holds, though the magnitude is rough. Each gate you add shifts access toward readers with higher incomes, in your own market and across borders.

### Why do news organizations drop their paywalls?

Publishers have a long record of opening the gate. Mike Ananny and Leila Bighash of USC published a study of 69 paywall exceptions[study of 69 paywall exceptions] in the International Journal of Communication in 2016. They documented cases between 1999 and 2015 in which outlets dropped or reconfigured their paywalls.

In one historical case from 2004, the L.A. Times reversed course after traffic to its “Calendar Live” section fell 97% behind a paywall.

Nieman Lab reported the L.A. Times wildfire coverage results[L.A. Times wildfire coverage results] after the paper made its January 2025 fire coverage free. According to the Times, traffic rose 800% and new subscriptions rose over 259% compared with the previous 30 days.

The Times estimated a 45% lower subscriber conversion rate during the free period and didn’t share absolute numbers. One crisis and one publisher’s own figures can’t show that free access caused the growth.

In September 2024, Nieman Lab covered the Salt Lake Tribune’s paywall plans[Salt Lake Tribune’s paywall plans]. The nonprofit paper raised $340,000 to remove its paywall on 2024 election coverage. The Tribune calls its paywall “a necessary evil” and aims to remove it in the long run.

In March 2025, Nieman Lab reported that Wired un-paywalled public-data stories[Wired un-paywalled public-data stories]. In Nieman Lab’s summary, “On stories that are primarily sourced to public data, there will be no paywall.”

Across these cases, publishers open the gate when reach and public value matter more than conversion. They price content by its public value during emergencies, and compensation tied to AI use could make that pricing continuous.

## The AI-era paywall paradox

The classic paradox assumes a human reader deciding whether to pay. AI agents now read pages for people, and they add a second, sharper version of the same trade.

An _AI agent_ is software that reads the web and completes tasks for a person. At the launch of Index, we wrote that “AI agents don’t see ads or consume content the way humans do.” The agent reads the page, answers the question, and the person it serves may never visit the source.

### A crawler used to be a prelude to a visit

In our case for contribution-based compensation[case for contribution-based compensation], we wrote, “A crawler used to be a prelude to a visit. Increasingly, a crawler is the visit.” We noted that “a model can extract value from a document without generating a pageview, an impression, or a subscription event.”

Pew Research Center measured the click side of this shift. In its Pew analysis of Google AI summaries[Pew analysis of Google AI summaries], published July 22, 2025, Pew studied March 2025 browsing data from 900 US adults. Users clicked a traditional result in 8% of visits with a Google AI summary, compared with 15% of visits without one.

Those users clicked a link inside the summary in 1% of visits. Pew’s data are observational and cover Google alone, so they show an association with summaries without proving a cause.

The _crawl-to-referral ratio_ counts how many pages a company’s bots crawl for each human visitor they send back. David Belson of Cloudflare published Cloudflare crawl-to-referral data[Cloudflare crawl-to-referral data] for News and Publications sites in the first week of August 2025. Anthropic’s ratio was 2,500 crawls per referral, OpenAI’s was 152, and Perplexity’s was 32.7.

Cloudflare’s ratios swing by month, so read these as a snapshot of one week. The Reuters Institute’s 2026 report cites Chartbeat data on Google organic search traffic to more than 2,500 sites. That traffic fell 33% worldwide and 38% in the US between November 2024 and November 2025.

Publishers let crawlers in because crawling used to bring readers back, and on these numbers that bargain no longer clears.

### Publishers respond by blocking AI crawlers

Many publishers answer through _robots.txt_, a plain text file at a site’s root that tells crawlers which pages they may fetch. For the BuzzStream analysis of 100 news sites[BuzzStream analysis of 100 news sites], Vince Nero checked robots.txt files at top US and UK outlets for 11 AI crawlers. In the April 8, 2026 update, 79% block at least one AI training bot and 71% block at least one retrieval or live search bot.

BuzzStream, which first published the study on December 18, 2025, found that 14% block all the bots it checked and 18% block none. For a baseline, the Reuters Institute’s crawler-blocking study[Reuters Institute’s crawler-blocking study] covered top news sites in 10 countries. By the end of 2023, 48% of them blocked OpenAI’s crawlers, including 79% in the US.

Harry Clarkson-Bennett, director of search engine optimization (SEO) at The Telegraph, gave BuzzStream the publisher’s reasoning. “Publishers are blocking AI bots using the robots.txt because there’s almost no value exchange,” he said.

A robots.txt file states a request, and crawlers that ignore it face no technical barrier. Our explainer on how web crawlers work[how web crawlers work] covers that limit and the other tools sites use to manage bots. Blocking a retrieval bot has a second effect, because a bot that honors the rule can’t fetch your page for a live AI answer.

Crawlers that behave well identify themselves and honor these rules. Our crawler, ShapBot, builds Parallel’s index, and you can manage it with a `User-agent: ShapBot` directive, as Parallel’s crawler documentation[Parallel’s crawler documentation] describes. Blocking stays your choice.

Blocking works as the paywall of the crawler era. It guards against free extraction, and it takes the publisher out of reach for the bots that respect it.

### Visibility rises while value falls

Blocking leaks, according to researchers at the Tow Center for Digital Journalism. Klaudia Jaźwińska and Aisvarya Chandrasekar published the Tow Center study of AI search citations[Tow Center study of AI search citations] in Columbia Journalism Review on March 6, 2025. They tested 8 AI search tools on 1,600 queries from 20 publishers, and the tools as a group were wrong on more than 60%.

Some tools identified excerpts from publishers that had blocked their crawlers, and licensing deals didn’t guarantee accurate citation. The test measured whether a tool could name the source of an excerpt. It doesn’t show that chatbots read paywalled pages, and the authors note that other routes, such as secondary references, could explain the correct answers.

In our Visibility-Value Gap research[Visibility-Value Gap research], published July 31, 2026, Parallel’s research team ranked domains two ways. _Visibility_ measures how often a domain’s pages surface when AI systems look for answers.

Value uses the _Shapley value_, a method from game theory. It measures how much worse an outcome gets without a given player, averaged across every combination of players. In our study, each domain’s value is its share of the value created, based on its marginal contribution.

We found that the two lists “overlap less than you’d think.” The study sorts domains into four groups:

Group	Visibility	Value	Examples from the study
Expected winners	High	High	No examples cited here
Overexposed	High	Low	medium.com (value rank 108, gap of 57 places); scribd.com (value rank 122, gap of 60)
Underrated	Low	High	arxiv.org (value rank 37, gap of 98 places); fda.gov (value rank 64, gap of 51)
Long tail	Low	Low	No examples cited here

Uniqueness explains the split. In the study, we wrote that “marginal contribution collapses when information is substitutable.” A domain that repeats facts found on many other pages adds little to an answer, though its pages may surface across many AI results.

For paywalls, the key observation is that “heavily paywalled domains can see their Shapley values collapse even as they keep gaining visibility.” The sample covers one week of Index activity, the week of May 25, 2026, and it skews toward enterprise use cases.

The data are observational. We didn’t name the paywalled domains or publish the size of the drop, and the finding doesn’t show that a paywall caused it.

Our reading is that the agent version of the paradox mirrors the 53% of Pew respondents who look elsewhere. An agent that finds the same facts on an open page has little reason to depend on the gated one. One research direction we’d propose is to track paywalled domains’ visibility and value over several months, to test whether the gap widens as gating tightens.

## Paywall SEO in search and AI answers

Google gives publishers a way to gate content and stay in its index. In Google’s paywalled content markup[Google’s paywalled content markup] documentation, updated September 8, 2026, Google Search Central asks you to add JSON for Linking Data (JSON-LD) to gated pages. You set `isAccessibleForFree` to `false` and add a `hasPart` element whose `cssSelector` points to the gated section.

Google says the markup “helps Google differentiate paywalled content from the practice of cloaking.” _Cloaking_ means showing crawlers different content than readers see, which violates Google’s spam policies. Google accepts class selectors such as `.paywall` in this field, so wrap your gated text in an element with that class.

### Html

1
2
3
4
5
6
7
8
9
10
11
12
13
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "Your article headline",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywall"
  }
}
</script>``` <script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "Your article headline",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywall"
  }
}
</script>
```

The same documentation states that AI Overviews and AI Mode follow Search preview controls such as `nosnippet`. Pair the markup with the metering limits in Google’s sampling guidance, which links lower satisfaction to paywalls on more than 10% of visits.

Agents outside Google need their own signals. An llms.txt file[llms.txt file] at your site root can point agents to the open pages you want them to read.

These signals let you gate and stay visible with systems that honor them. They tell an agent where to look, and they say nothing about how much your page adds to an answer.

## Paths forward: pricing contribution instead of access

Publishers have three levers beyond a single gate, and each lever prices something different. A hybrid paywall prices access to unique work, pay per crawl prices crawling, and contribution-based compensation prices what a source adds to an answer.

We build for the third lever, because Pew’s readers and our Index data both point to uniqueness.

### Hybrid paywalls that gate what’s unique

A _hybrid paywall_ gates the coverage readers can’t get elsewhere and opens commodity and public interest coverage. Wired opens its public data stories, and the Salt Lake Tribune raised $340,000 to open its 2024 election coverage while keeping its meter.

Pew found that 49% of people who didn’t pay cite free alternatives as their main reason. Gating substitutable content buys you little revenue, because readers can find the same facts on a free page.

Our recommendation is to sort your coverage into two groups before you change any gate. Put original reporting and proprietary data in one group, and syndicated stories or stories that many outlets report in the other. Gate the first group and open the second.

### Pay per crawl and licensing deals

In a July 1, 2025, post on Cloudflare’s Content Independence Day[Cloudflare’s Content Independence Day], Cloudflare changed its default to block AI crawlers unless they pay. Cloudflare introduced _pay per crawl_ the same day, a system that lets a site charge AI crawlers for the pages they fetch.

Cloudflare CEO Matthew Prince wrote that “the web is being stripmined by AI crawlers with content creators seeing almost no traffic.” Cloudflare runs the pay per crawl product, so it has a commercial interest in that framing.

Licensing deals are the other route. In our view, today’s deals are flat fees that each publisher negotiates on its own. They reach owners with scale and leave smaller publishers out.

Both models price access. A crawler pays the same fee whether your page shaped the answer or added nothing. Neither model rewards the unique work that makes a page worth reading.

### Contribution-based compensation

Today you face a binary choice. You can allow AI crawlers and get little in return, or block them and drop out of reach for bots that respect the rule.

We built Index by Parallel to replace that choice with a price on contribution. With the launch of Index by Parallel[launch of Index by Parallel] on May 19, 2026, site owners can enter a domain and see which queries their content answers. You can see how often agents reference your content and how unique its contribution is.

We estimate each source’s Shapley value at the moment of inference, when an agent uses a page to produce an answer. For Index, the players are sources and the outcome is the work the agent performs. We base payouts on that estimate, so content that’s hard to replace earns more than content an agent could find on other pages.

Index by Parallel homepage: See what your content is worth to AI agents

Your content earns when it improves an answer, with no click required. You learn which work agents rely on before you decide what to gate.

Our launch partners include The Atlantic and Fortune among news publishers. Data partners include PR Newswire, ZoomInfo, and Tracxn, and independent creators include Azeem Azhar (Exponential View) and Packy McCormick (Not Boring).

Index is open to agents built outside Parallel. For agent builders, Index offers a compensated route to sources that are hard to reach, including paywalled data. That route reduces your reliance on scraping restricted content.

In Fortune’s coverage of Index[Fortune’s coverage of Index], our CEO Parag Agrawal said that “agents will use the web a lot more than humans.” Nicholas Thompson, CEO of The Atlantic, said in the same piece that “the economics of the web have not caught up with that reality.” Fortune and The Atlantic both participate in Index, and Fortune noted the risk that publishers grow dependent on another intermediary.

### Audit what makes your content irreplaceable

Our Visibility-Value Gap research points to a working rule for the _agentic web_, the part of the web that AI agents read and act on. From that study, we concluded that original reporting and proprietary data outperform syndicated content. You can run an audit to find which of your pages fall into each group:

List the datasets and original reporting your newsroom owns.
For each item, check how many other pages report the same facts.
Gate or license the unique material, and open the substitutable material.
Mark gated sections for search, and set crawler rules on purpose.
Recheck the list as coverage of your topics changes.

Step 2 takes the most work by hand. The Parallel Search API[Parallel Search API] takes an objective in natural language, such as “pages reporting this county’s 2026 eviction figures.”

Search returns ranked URLs with dense excerpts. The Extract API[Extract API] turns those URLs into clean markdown, including PDFs and pages heavy with JavaScript, so your team can compare them side by side. We offer this as a suggested workflow, and we haven’t published a customer outcome for it.

A publisher could extend the audit into its own substitutability index that tracks, topic by topic, how many pages repeat its coverage over time. We see that as a proposed research direction, and we haven’t run it.

## The path forward for the paywall paradox

The classic paradox begins with a reader who leaves the gate for a free site. The agent version begins with an agent that finds the same facts on an open page and routes around the gated one. In both versions, pages that are hard to replace keep their value, and gates on content found elsewhere cost the publisher reach.

We think staying open wins once markets reward contribution. Transparent attribution tells a publisher which work an agent used, and open markets based on value pay for that work without demanding a click. Publishers get a reason to keep pages readable, and agent builders get a fair way to reach the sources their users need.

The shift from pricing attention to pricing contribution has started, and we’re building the infrastructure for the web that follows it.

If you publish, join the Index[join the Index] to see what your content is worth to AI agents.

Ready to get started? Try Parallel[Try Parallel] and build with our APIs. You can also Contact sales[Contact sales] to plan a deployment with our team.

Are you an agent? Read this to onboard Parallel[Are you an agent? Read this to onboard Parallel]

## FAQs about the paywall paradox

### What does it mean when something is behind a paywall?

Content behind a paywall requires payment or a subscription before you can read it in full, and some sites ask for a free registration instead. The models vary in how much they gate. A hard paywall blocks every article, and a metered paywall lets you read a set number each month.

Paywalls differ in how they work as well. Some sites send the full article to your browser and hide it on screen until you subscribe. Others keep the full text on the server and send it to subscribers alone.

### Are paywall bypasses illegal?

This answer is general information about US law, and it isn’t legal advice. Ask a lawyer about your specific situation.

In the Copyright Office’s Section 1201 FAQ[Copyright Office’s Section 1201 FAQ], the U.S. Copyright Office says Section 1201 prohibits circumventing technological measures that control access to copyrighted works. Its example is bypassing a password control on a subscription video service, and the answer for any specific bypass depends on the facts and jurisdiction.

Publishers have pressed the issue, as the News/Media Alliance on 12ft.io[News/Media Alliance on 12ft.io] statement from July 2025 shows. The trade group said it secured the takedown of the site and called it “illegal circumvention technology.” That characterization is the Alliance’s own, and the sources we reviewed show a takedown by the site’s web host with no court ruling.

We won’t recommend a bypass tool. For free access to gated journalism, check your public library’s digital subscriptions, a site’s monthly free allowance, or a gift link from a subscriber. If your agents fetch pages at scale, our overview of the legal picture for web scraping[legal picture for web scraping] covers the questions to raise with counsel.

### Can ChatGPT read articles behind a paywall?

AI search tools draw on what their crawlers can fetch, what licensing partners supply, and what other pages restate about a story. The Tow Center noted OpenAI’s 16th and 17th news licensing deals in February 2025. It found that licensing didn’t guarantee accurate citation.

The Tow Center found that some tools identified excerpts from publishers that had blocked their crawlers. That result shows publishers can’t count on their blocking preferences being honored or visible. It doesn’t show that the tools read paywalled pages.

If you publish behind a paywall, assume your facts reach AI answers through other pages unless those facts are unique to you. Unique reporting is the material a gate or a license can protect.

By Parallel

October 12, 2026