stuffonscreen.
Page checkedhttps://www.theverge.com/

3 of 6 assistants can read this page.

Claude and Perplexity can’t read this page. Training crawlers from Anthropic, Google, Apple, Common Crawl, Meta and ByteDance are blocked in robots.txt. That doesn’t stop you appearing in answers.

Checked 28 Sept 2026, 11:30 am UTC
17 crawlers · 1 page · Fastly

By assistant

ChatGPT

OpenAI
Partly
robots.txt
Partly blocked
Firewall
200 OK
Page content
Readable

robots.txt blocks ChatGPT-User, though it may ignore that for requests people make.

OAI-SearchBot · ChatGPT-User

Claude

Anthropic
Blocked
robots.txt
Blocked
Firewall
403 Blocked
Page content
Not reached

robots.txt blocks Claude-SearchBot and Claude-User.

Claude-SearchBot · Claude-User

Perplexity

Perplexity
Blocked
robots.txt
Blocked
Firewall
403 Blocked
Page content
Not reached

robots.txt blocks PerplexityBot and Perplexity-User.

PerplexityBot · Perplexity-User

Google AI Overviews

Google
Can read
robots.txt
Allowed
Firewall
Not tested
Page content
Readable
Googlebot

Microsoft Copilot

Microsoft
Can read
robots.txt
Allowed
Firewall
Not tested
Page content
Readable
Bingbot

Siri & Spotlight

Apple
Can read
robots.txt
Allowed
Firewall
Not tested
Page content
Readable
Applebot
Fix this firstAffects ChatGPT, Claude, Perplexity and Gemini

robots.txt blocks ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User

These crawlers put pages into AI answers and search results. While they’re blocked, those services won’t show or link this page.

  1. In robots.txt, delete the Disallow lines under “User-agent: ChatGPT-User”, “User-agent: Claude-SearchBot”, “User-agent: Claude-User”, “User-agent: PerplexityBot” and “User-agent: Perplexity-User”. If you'd rather be explicit, replace those groups with:
    User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /
  2. Run this check again.

The page itself

Text without JavaScript1,915 words in the HTML
Readable
noindexMeta tag and HTTP header
Not set
Snippet limitsnosnippet, max-snippet
None
robots.txtwww.theverge.com/robots.txt
Found
Structured dataNewsMediaOrganization, WebSite
Found

Crawler by crawler

CrawlerCompanyUsed forrobots.txtFirewallResult
OAI-SearchBotCompanyOpenAIUsed forSearchrobots.txtAllowedFirewall200 OKResultCan read
ChatGPT-UserCompanyOpenAIUsed forOn requestrobots.txtBlockedFirewall200 OKResultBlocked
GPTBotCompanyOpenAIUsed forTrainingrobots.txtAllowedFirewallNot testedResultAllowed
Claude-SearchBotCompanyAnthropicUsed forSearchrobots.txtBlockedFirewall403 BlockedHTTP 403.ResultBlocked
Claude-UserCompanyAnthropicUsed forOn requestrobots.txtBlockedFirewall403 BlockedHTTP 403.ResultBlocked
ClaudeBotCompanyAnthropicUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out
PerplexityBotCompanyPerplexityUsed forSearchrobots.txtBlockedFirewall403 BlockedHTTP 403.ResultBlocked
Perplexity-UserCompanyPerplexityUsed forOn requestrobots.txtBlockedFirewall403 BlockedHTTP 403.ResultBlocked
GooglebotCompanyGoogleUsed forSearch & AI Overviewsrobots.txtAllowedFirewallNot testedResultAllowed
Google-ExtendedCompanyGoogleUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out
BingbotCompanyMicrosoftUsed forSearch & Copilotrobots.txtAllowedFirewallNot testedResultAllowed
ApplebotCompanyAppleUsed forSearch & Sirirobots.txtAllowedFirewallNot testedResultAllowed
Applebot-ExtendedCompanyAppleUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out
CCBotCompanyCommon CrawlUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out
meta-externalagentCompanyMetaUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out
AmazonbotCompanyAmazonUsed forTrainingrobots.txtAllowedFirewallNot testedResultAllowed
BytespiderCompanyByteDanceUsed forTrainingrobots.txtBlockedFirewallNot testedResultOpted out

Where we can, we request the page the way each AI crawler does, giving its name (its user agent). We don’t imitate Google, Bing or Apple, because sites check those by IP address and would block our test as a fake.

Good to know

  • You block 6 AI training crawlers

    ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Bytespider can’t collect this page for model training. That doesn’t affect whether you appear in AI answers, which use the search crawlers above.

  • No llms.txt file

    You don’t need one. No major AI company has said its crawlers use llms.txt.

Want us to fix it?

Send us this report and we’ll take it from there.