Agentic Browsing, AI Catalog JSON and llms.txt: Do They Matter for SEO?

Who this guide is for, and how we wrote it#

This guide is for small and mid-sized businesses, SaaS teams and marketers deciding whether agentic browsing deserves a slot on this quarter's SEO roadmap. It's based on Google Search Central's guidance on AI features, Chrome for Developers documentation on agent-ready websites and Lighthouse, the draft AI Catalog spec and the llms.txt proposal at llmstxt.org.

We also publish both files on our own site, so parts of this guide come from doing the work rather than reading about it. A quick disclosure: Uplift AI sells SEO software. We don't sell llms.txt or AI catalog setup as a separate service, and the examples below are illustrations unless we say they're from our own site.

What is agentic browsing?#

Agentic browsing is a way of using the web where an AI agent, not a person, opens pages, reads them and takes actions to finish a task. The person gives a goal, such as "find a dentist near me open on Saturday and book a check-up", and the agent searches, compares sites, clicks buttons and fills in forms until the job is done or it needs approval.

You'll find agentic browsing in tools like Perplexity's Comet browser, Claude in Chrome, Google's Gemini features in Chrome and the agent mode inside ChatGPT. The products change fast. OpenAI retired its Atlas browser on 9 August 2026 and moved its browsing agent into ChatGPT itself. That's a good reason to build for open standards rather than for any single agent.

How do AI agents read a website?#

AI agents read a website mainly through its accessibility tree, the same structured view of headings, links, buttons and form fields that screen readers use. Chrome's developer documentation calls the accessibility tree the primary data model agents rely on. Some agents also take screenshots, but a page with clear, labelled elements is far easier for them to use correctly.

That has a practical consequence. A button drawn as an image with no label, a form field with placeholder text and no real label, or a price that only appears after a script runs can all stop an agent mid-task. When that happens, the agent moves to the next site on its list, and you never see the visit in your analytics as a lost customer.

• Accessibility tree: element roles and names, such as "button: Book appointment" or "textbox: Email".
• Page text and headings: the content an agent uses to judge whether your page answers the task.
• Structured data: Schema.org markup such as Product, Offer, LocalBusiness and FAQPage.
• Screenshots: a fallback for visual layout, which makes layout shifts a real problem.
• Declared tools: functions a site exposes through WebMCP, which agents can call directly instead of clicking.

Is agentic browsing important for SEO?#

Agentic browsing is important for SEO because agents still start with search. Chrome's own agent-ready guidance says that when agents are just searching for websites, the principles of SEO still apply. If your page doesn't rank or can't be crawled, an agent is unlikely to find it, however agent-friendly it is.

Where agentic browsing changes things is after the click. A human visitor forgives a clunky form or a hidden price. An agent often doesn't. So agentic browsing adds a second test on top of rankings: once an agent lands on your page, can it understand your offer and finish the task? Think of it as conversion rate optimisation for a visitor who reads code instead of looking at colours.

What is AI Catalog JSON?#

AI Catalog JSON is a machine-readable file that lists the AI-facing resources a website offers, such as an MCP server, an agent card, agent skills or other AI-readable documents. The draft AI Catalog spec suggests serving it at /.well-known/ai-catalog.json with the media type application/ai-catalog+json, so an agent can find everything in one request instead of guessing URLs.

The spec is still a draft that its authors say may change at any time, and it comes from the community behind the A2A and MCP agent protocols. It defines three levels: a minimal list of entries, a discoverable level that adds a host block and the well-known address, and a trusted level that adds signed trust manifests. Most sites that adopt it today stop at level 2.

• specVersion: the version of the spec the file follows, such as "1.0".
• host: who publishes the catalog, with a display name and optional logo and documentation URL.
• entries: one item per resource, each with an identifier, a type, and a URL or inline data.
• Registered types: A2A agent cards, MCP server cards, agent skills, plugins and nested catalogs.
• Trust manifests (level 3): signatures that let an agent check the catalog really comes from you.

What does a real AI catalog look like? Our own file#

Our own catalog at upliftai.co/.well-known/ai-catalog.json is a level 2 file with a host block and exactly one entry: our llms.txt. We don't run a public MCP server, an A2A agent card or agent skills yet, so we didn't list any. An entry that points at something you don't serve is worse than no entry, because it sends agents to a dead end.

One detail surprised us when we built it. The spec's registered types cover agent cards, MCP servers, skills and plugins, but llms.txt isn't one of them, so we list it with the plain text/markdown type. That tells you where the spec's focus is: AI Catalog JSON is built for sites with agent tools, and a content site with only an llms.txt gets very little from it today.

• specVersion: "1.0"
• host: displayName "Uplift AI", identifier https://www.upliftai.co, plus documentation and logo URLs.
• entries[0].type: "text/markdown", pointing at https://www.upliftai.co/llms.txt
• entries[0].tags: "llms.txt", "seo" and "ai-search"
• Served with Content-Type application/ai-catalog+json and a one-day CDN cache.

What is llms.txt?#

llms.txt is a Markdown file served at the root of a website, such as example.com/llms.txt, that gives language models a short, curated summary of the site and links to its most useful pages. Jeremy Howard proposed it in September 2024. The only required part is an H1 with the site's name. Everything else is optional but recommended.

The idea is simple. A model reading raw HTML has to strip menus, cookie banners and scripts before it reaches your content, and it can't tell which of your pages matter most. A clean Markdown index fixes both. Our full guide to llms.txt covers how to write one and who benefits most, so here we'll stick to how it fits next to agentic browsing.

• H1: the name of the site or project (required).
• Blockquote: a one or two sentence summary of what you do.
• Short notes: plain paragraphs or lists with key details, no extra headings.
• H2 file lists: grouped links in the form [Page name](url): short description.
• "Optional" section: secondary links an agent can skip when it's short on context.

llms.txt vs AI Catalog JSON vs robots.txt vs sitemap.xml#

llms.txt, AI Catalog JSON, robots.txt and sitemap.xml solve four different problems, and none of them replaces another. robots.txt controls access. sitemap.xml lists every URL you want indexed. llms.txt picks out your most useful content for a model. AI Catalog JSON lists the AI tools and resources you offer.

If you only have time for one of the two new files, publish llms.txt. It takes about half an hour, Lighthouse already checks for it, and AI Catalog JSON mostly points to things a typical business site doesn't have yet. Keep robots.txt and your sitemap accurate first, because search engines and agents both rely on them today.

• robots.txt: tells crawlers what they may access. Widely honoured. Essential.
• sitemap.xml: lists every page you want search engines to index. Used by Google. Essential.
• llms.txt: a curated Markdown summary for language models. Not used by Google Search. Cheap to add.
• AI Catalog JSON: a JSON list of AI resources such as MCP servers and agent cards. Draft spec. Add it when you have resources to list.
• Schema.org structured data: describes your business, products and content on each page. Used by Google and read by agents. High priority.

What do Google and Chrome actually say?#

Google Search and Google Chrome say different things, and that's where most of the confusion comes from. Google Search Central's guidance on AI features says you don't need to create new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode. For Google rankings, llms.txt and AI Catalog JSON do nothing.

Chrome is a different team with a different job. In June 2026 Chrome for Developers published a toolkit for agent-ready websites, and Lighthouse in Chrome 150 and later includes an experimental Agentic Browsing category. It reports a pass ratio rather than a 0 to 100 score, and it checks four things: llms.txt, WebMCP tools, accessibility for agents and layout stability.

• Discoverability: does the site serve an llms.txt file?
• WebMCP integration: are tools registered, do forms have declarative WebMCP, and are the schemas valid?
• Accessibility for agents: does every interactive element have a programmatic name?
• Layout stability: does the page avoid layout shifts, measured with Cumulative Layout Shift (CLS)?

How do you make your website ready for agentic browsing?#

You make a website ready for agentic browsing by fixing the basics agents depend on, in order of impact. Most of these steps also improve accessibility, conversions and classic SEO, so none of the work is wasted if agent traffic grows more slowly than the headlines suggest.

Run the Lighthouse Agentic Browsing category on your three most important pages first, such as the homepage, a service or product page and your contact or booking page. That gives you a short, specific list instead of a vague project, and it shows you which forms an agent would struggle with.

• 1. Make sure search engines and the AI crawlers you want can reach your pages, and that robots.txt and your CDN's bot settings match what you intend.
• 2. Render key content in the HTML. Prices, opening hours, services and stock status shouldn't depend on scripts loading first.
• 3. Give every button, link and form field a clear name. Use real <button> and <label> elements, not clickable images or placeholder-only fields.
• 4. Add Schema.org structured data for your business type, such as Organization, LocalBusiness, Product, Offer and FAQPage.
• 5. Fix layout shifts so buttons don't move while an agent is about to click them.
• 6. Publish a short, curated llms.txt of your most useful pages and keep it current.
• 7. Add AI Catalog JSON and WebMCP only when you have agent tools, an API or booking and checkout flows worth exposing.

Which of these should your business do first?#

Your priority depends on what agents would do on your site. A local service business mostly needs agents to read hours, prices and booking options correctly. A SaaS company with documentation gets real value from llms.txt, because coding agents read docs directly. An online store needs product data and checkout forms that agents can use.

Whatever your type, structured data is the closest thing to a sure win, because Google uses it and agents read it too. Our guide to schema markup for AI search covers which types to add first, and our technical SEO essentials guide covers the speed and mobile basics that agents and people both need.

• Local businesses: LocalBusiness schema, a crawlable booking page and clearly labelled contact forms. llms.txt is optional.
• SaaS and developer tools: llms.txt with your docs, a clean API reference, then an AI catalog once you ship an MCP server.
• Ecommerce: Product and Offer schema, visible prices and stock, and accessible cart and checkout forms. WebMCP is worth testing later.
• Publishers and content sites: crawlability, clear answers and structured data first. AI Catalog JSON adds little until you offer agent tools.

Common mistakes with agentic browsing, llms.txt and AI catalogs#

The most common mistake we see is treating llms.txt as a ranking hack. Teams publish the file, wait for traffic, and conclude AI search doesn't work. The file was never going to move Google rankings, and the hours would have done more on content and structured data.

The second mistake is blocking the agents you want. Aggressive bot protection, a CAPTCHA on every page or a blanket block on AI user agents can stop an agent before it reads a single word. Decide which AI crawlers and agents you want, then check your robots.txt and CDN settings actually match that decision.

• Listing resources in an AI catalog that you don't actually serve, such as an MCP server that's still in development.
• Dumping every URL into llms.txt. The value is in the selection, so keep it to your most useful pages.
• Letting llms.txt go stale after pages move. A file full of broken links misleads anything that reads it.
• Hiding prices or key details behind scripts, pop-ups or "contact us for a quote" when you could show a range.
• Assuming a good Lighthouse agentic score means agents will choose you. Rankings and a strong offer still decide that.

How can you tell if AI agents and assistants find your site?#

You can tell if AI agents and assistants find your site by checking your server logs for AI user agents and by tracking whether assistants mention or cite you for your key topics. Many agent visits run inside a normal browser, so they won't always show up as a separate bot. Look for unusual referral sources and quick, form-heavy sessions too.

Our guide to AI visibility tracking across ChatGPT, Claude and Perplexity explains how to measure mentions and citations. If you'd rather have the audit done for you, Uplift's AI SEO agent checks crawlability, structured data and AI visibility in one place, with plans from $49/month for a limited time per website.

Final verdict: does agentic browsing change SEO?#

Agentic browsing doesn't replace SEO. It adds a second test after the ranking: can an AI agent understand your page and finish the task? AI Catalog JSON and llms.txt are worth adding, but they're small, cheap steps, not ranking factors. Google has said plainly that AI search doesn't need special files.

Our recommendation is simple. Fix crawlability, labels, structured data and layout stability first, because they help search engines, AI agents and people at once. Then add llms.txt in an afternoon, and save AI Catalog JSON for when you have agent tools to list. If you want help with the bigger wins, see current plans.