Skip to main content
    Proof
    Aussie Plumbing — ad spend scaled 1.8× with enquiries up 2.1×, so each enquiry cost lessI-ELEC — a 231-page electrician website for Inner SydneyWhat a plumbing lead costs on Google AdsWhat a lead costs on Google Ads, industry by industrySLS Solicitors — cost per recorded Google Ads conversion down 58.48%Wilco Plumbing — search campaigns and service pagesWilco Plumbing — service menus and guided enquiriesHow much should a trade business spend on marketing?State Choice Plumbing — a drains, hot water and gas websiteAussie Drains — a 600-page drain website for SydneyWhat an electrician lead costs on Google AdsAussie Electrical — a 650-page electrical website for SydneyThe most efficient ways for a trade business to get leadsEOL Melbourne — click-to-enquiry rate up from 28% to 33.4%What a cleaning lead costs on Google AdsAlliance Plumbing — ad spend scaled 1.8× with enquiries up 2.1×, so each enquiry cost lessImpress Blinds — cost per recorded Google Ads conversion down 62.63%FixCare Property - cost per enquiry down 28%Rubbish Removal WA — cost per enquiry down 53.18%Floral Cakery — cost per enquiry down 49.82%ILLUMINATE Laser Emporium — cost per enquiry down 48.54%Sydney Fence Painting — cost per enquiry down 33.68%Alliance Plumbing — cost per enquiry down 29.6%Gridless Build Solutions — cost per enquiry down 29.16%FacilityWorx — cost per enquiry down 23.62%Cornerstone Roofing — cost per enquiry down 20.47%Council finance — budgeting and workforce designCouncil planning — clearer approvals and reportingCultural audiences — media strategy and creative conceptsCommunity participation — surveys, maps and project pagesPublic-sector research — survey design and analysisPublic art — site studies and visual conceptsRegional identity — visitor guides and wayfinding conceptsCouncil systems — integration and migration planningA pressure washing business — service-led search campaignsA pressure washing business — a clear enquiry journeyA carpet cleaner — Google Ads built around cleaning servicesA roofing company — campaigns for repairs and restorationA CCTV installer — campaigns for security enquiriesA fence painter — search campaigns for specific surfacesA maintenance business — live in 8 weeks, 3 stacks gone5.0 across every Google review$120M+ in media under management250+ active engagements across five countries
    Small Business

    Does Claude read my website? How AI assistants find and use business information

    Yes, Claude can read your website if your robots.txt allows Anthropic's Claude-SearchBot and Claude-User bots and your page text is in the HTML, not loaded by JavaScript. AI assistants find business information through web search, pages users ask about, and training, then use clear facts like services, areas and prices. Check robots.txt, view your page source and fix broken links.

    • By Vaishu, Head of SEO
    • 17 November 2026
    • 6 min read
    Illustration of Claude on a laptop and a phone. On the laptop, someone asks who can help them get more plumbing jobs in Melbourne; Claude reads soudcoh.com and suggests SoudCoh, a Melbourne marketing agency that helps plumbers get more jobs. On the phone, someone asks for the best marketing agency for plumbers; Claude searches the web and suggests SoudCoh for Google Ads, SEO and websites.

    Can Claude read your site? Three quick checks

    Each takes a couple of minutes.

    Tick a check as you review it.

    Fix access first; better content can't help a page the AI can't read.

    You've written good service pages. But when someone asks Claude about your business, does it actually read them? Here's how AI assistants find and use business information, what stops them reading a website, and how to check yours.

    Does Claude read my website?

    Yes, if it's allowed to and the page can be read without running JavaScript. Anthropic, the company behind Claude, says its bots follow your robots.txt file, and it uses three of them for different jobs:

    • Claude-SearchBot visits websites to improve Claude's search results.
    • Claude-User fetches a page when someone asks Claude a question that needs it.
    • ClaudeBot collects content that could be used to train future models.

    Anthropic's help article on its crawlers explains each one. Our guide to getting your business mentioned by ChatGPT and Claude has the full table of AI bots, from Claude, ChatGPT and others, and what blocking each one does.

    How do AI assistants find business information?

    In three main ways. They search the web and read the pages the search returns, they fetch a specific page when a user pastes a link or asks about it, and they draw on what they learned in training. ChatGPT works the same way, with OAI-SearchBot for its search features.

    Your website is only one of the sources. When an assistant searches for a business like yours, it can also read directories, review sites and articles that mention you. That's why the same name, services, address and phone number everywhere matters, and why we explain why ChatGPT recommends a competitor instead of you.

    What stops AI assistants reading your website?

    • A robots.txt block. Some security plugins and hosting settings block AI bots by default. One line can make you invisible.
    • Text that only appears after JavaScript runs. In a study of AI crawler traffic, Vercel found that none of the major AI crawlers it tracked ran JavaScript; only Google's Gemini (using Googlebot) and Apple's crawler did. If your service details load in the browser, ChatGPT and Claude may see an almost empty page.
    • Broken links. The same study found that about 34% of ChatGPT's and Claude's page requests hit 404 errors, against about 8% for Googlebot. Pages that moved without a redirect cost you visits that could have read a real page.
    • Information behind a login, a form or a download. If a customer has to click through to find your prices or service areas, an AI bot probably won't.

    The first three are technical SEO problems, and they're cheap to fix compared with writing new content nobody can read.

    A page an AI bot can't read can't be quoted, however good it is.

    How do AI assistants use what they read?

    They look for clear, specific answers they can repeat: what you do, where you work, who you work for and roughly what it costs. Short paragraphs, headings that match real questions and structured data help you make your content easier for AI tools to read. Vague claims like "quality service" give them nothing to use.

    Reading your page isn't the same as recommending you. Which business an assistant names is a separate question; we cover how ChatGPT chooses which business to recommend.

    Check your site in 10 minutes

    1. Open yourwebsite.com.au/robots.txt. Look for "Disallow: /" under Claude-SearchBot, Claude-User, OAI-SearchBot or "User-agent: *". If it's there, your site is blocked.
    2. View the page source. Open your main service page in Chrome, right-click, choose View page source and search for a sentence from the page. If you can't find it, the text only appears after JavaScript runs.
    3. Ask Claude. Paste your page's address into Claude and ask: "What services does this business offer, and in which areas?" A vague or wrong answer means it couldn't read the page properly.
    4. Find broken pages. In Google Search Console, open Pages and look for "Not found (404)". Redirect the old addresses people and bots still visit.

    For new pages, Google still matters too: see getting a new page indexed by Google.

    What we did on soudcoh.com

    Our site is built with a JavaScript framework, exactly the setup that can hide text from AI bots. So every page is prerendered: the server sends the finished text, and JavaScript only adds the interactive parts. When we request one of our service pages as Claude-SearchBot, with no JavaScript, the full page comes back, around 8,000 words of it. Our robots.txt names each search and AI bot and lets them in.

    If you're planning a new website, ask how the pages will reach bots that don't run JavaScript before the build starts. It's much easier than fixing it after launch.

    How SoudCoh helps

    We check whether Claude, ChatGPT and other AI assistants can reach and read your site, fix what's in the way, then make your pages clear enough to be used and recommended. If you're looking for the best marketing agency for your business, start with a free marketing audit and we'll tell you what Claude and ChatGPT can and can't see.

    Common questions about AI bots and your website

    Does Google read JavaScript?

    Googlebot can render JavaScript, though rendering can happen later than the first crawl. The AI crawlers from OpenAI and Anthropic were found not to run it, so text that only appears after JavaScript is safest moved into the HTML.

    How long does a robots.txt change take to work?

    OpenAI says it can take about 24 hours for ChatGPT search to reflect a robots.txt change. Anthropic doesn't publish a timeline, so check again after a few days.

    Can Claude read pages behind a login?

    Treat anything behind a login, a form or a paywall as invisible. If you want AI assistants to use the information, put it on a public page.

    Questions and answers

    Why can't ChatGPT or Claude see my website text?

    Usually because the text loads with JavaScript after the page opens. Vercel's study found the major AI crawlers, apart from Gemini and Apple's, don't run JavaScript, so they only see what's in the HTML.

    Does Claude only use my website?

    No. When Claude searches the web, it reads the pages the search returns, which can include directories, review sites and articles that mention you. Keep your business details the same everywhere.

    Not sure what Claude can see?

    Send us your website and we will check whether Claude and ChatGPT can reach and read it, and what to fix first. Start with a free marketing audit.

    Book a free marketing audit

    Sources and related projects

    1. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?": ClaudeBot (training), Claude-User (fetches for Claude users), Claude-SearchBot (search quality); all honour robots.txt: support.claude.com/en/articles/8896518
    2. Vercel and MERJ, "The rise of the AI crawler" (17 December 2024): none of the major AI crawlers render JavaScript (Gemini via Googlebot and AppleBot do); 34.82% of ChatGPT and 34.16% of Claude fetches hit 404 pages vs 8.22% for Googlebot: vercel.com/blog/the-rise-of-the-ai-crawler
    3. OpenAI, "Overview of OpenAI crawlers": about 24 hours for ChatGPT search to reflect robots.txt changes: developers.openai.com/api/docs/bots
    4. SoudCoh's own test, 17 November 2026: soudcoh.com service pages requested with the Claude-SearchBot user agent and no JavaScript returned the full page text (about 8,000–9,000 words each); robots.txt allows each named AI crawler.
    Read next

    Filed under the same desk first. The full index is searchable and filters by reader.

    Explore the services and related work

    See how the ideas in this article connect with our services and client projects.

    • AI search optimisation — Pages AI assistants can reach, read and quote, with your business facts kept consistent.

    • Technical SEO — robots.txt, rendering, redirects and broken-link fixes so search and AI bots can read every page.

    Related client work.

    These are individual engagements where the market, system or measurement problem overlaps with the briefing. Each study names its window and the evidence available; none is a forecast for another account.