You ask an AI assistant: "How long is the warranty at Shop X?" It answers with total confidence and gets it wrong, because the shop's policy page is a tangle of buttons, banners and scripts with the real wording buried inside. The customer trusts the answer and walks into the shop expecting the wrong thing.
More people ask an AI before they ever open a website. This post goes through each thing being discussed in 2026 and says plainly which are official standards and which are only proposals. I use phatdev.net as the example, because I built these pieces for real and checked them with commands.
First: what Google says
If you worry about the AI parts of Google Search (AI Overviews, AI Mode), Google Search Central's documentation (checked 21 October 2026) is direct: you do not need new machine-readable files, AI text files or special markup to appear in them. There is no special structured data to add either. The requirement is that the page is indexed and eligible to show a snippet in Search. That page does not mention llms.txt.
So the foundation is still ordinary SEO. Everything below is an extra layer for other AI agents, not a replacement for that foundation. For more, see Making Sure Google Sees Both Languages of Your Next.js Site.
Standard or proposal
| Item | What it is | Status (checked 21 Oct 2026) |
|---|---|---|
| robots.txt | Tells bots which paths they may visit | Long-standing convention; polite bots obey, but it is only a request |
| Structured data (schema.org JSON-LD) | Labels telling machines "this is an article, a product, an author" | Widespread practice, supported by Google |
| Google-Extended | A robots.txt token to opt content out of Gemini use | Official Google control |
| llms.txt | An index file for AI | Community proposal, not a standard |
| Markdown version of a page | A flat, clean page for AI | Part of the llms.txt proposal |
| MCP Server Card | A business card describing an MCP server | Official, optional MCP extension |
| AI Catalog (ai-catalog.json) | A per-domain catalog of AI resources | Still being drafted |
| WebMCP | A page registers tools for AI inside the browser | Community group draft, not stable |
Now each one.
robots.txt and AI bot permissions
robots.txt is a file at your site root telling bots which paths they may visit. Google describes it as a way to reduce server load, not a way to hide pages: a blocked page can still show up in results if other sites link to it, and bots that are not polite may ignore it. To truly hide something, use noindex or a password.
For AI there are two separate decisions. One is which bots may read. The other is whether your content may be used for training. Google-Extended is the token for the part where content is used for Gemini models (training and "grounding", meaning search data fed into answers), and Google says it does not affect whether your site appears in Google Search. Other AI companies have their own bot names, so read each one's documentation before you write rules.
phatdev.net currently has only a general rule: all bots may read the public part, while the admin area and API are blocked. There are no rules for individual AI bots, because the site wants to be found by as many people as possible. If your content is something you sell, think differently.
Structured data
These are JSON-LD blocks in a page that tell machines: this is an article, who wrote it, when. Google says no AI-specific schema is needed. Just keep it correct and matching what is visible on the page.
llms.txt and Markdown pages
This proposal came from Jeremy Howard on 3 September 2024; per llmstxt.org the current version was edited on 10 August 2026, and the site itself calls it "a proposal". The idea: put an /llms.txt file at your site root, written in Markdown, with the site name, a short summary and a list of links to key pages. Alongside it, each page should offer a clean Markdown version at the same address plus .md, and the HTML page should point to it with link rel="alternate" type="text/markdown".
I found no official Google documentation saying Google uses llms.txt. Treat it as a cheap extra that helps agents and tools willing to read it, not as a ranking lever.
phatdev.net does all three layers: /llms.txt (an index with a description of each page), /llms-full-vi.txt and /llms-full-en.txt (all public content, combined by language, about 1.1 MB each; /llms-full.txt is an index pointing to these two files) and a .md version of every page. You can fetch the Markdown two ways: append .md to the address, or send a request with Accept: text/markdown. HTML pages carry the rel="alternate" link to the Markdown copy.
MCP Server Card and AI Catalog
If your site runs an MCP server for AI to query (see What Is MCP? How AI Connects to Your Tools and Data, and the Risks to Know), AI needs to find it. An MCP Server Card is a file describing the server before connecting: address, connection type, protocol versions. It is an official, optional MCP extension (SEP-2127, status Final). phatdev.net serves it at https://phatdev.net/api/mcp/server-card.
AI Catalog (/.well-known/ai-catalog.json) is a general catalog of a domain's AI resources, which can point to a Server Card. The Agent-Card/ai-catalog repo calls itself a "temporary working repo" maintained under the Linux Foundation, and the specification is still in progress. phatdev.net publishes the file, but treat it as an early experiment and recheck the current docs because the path may change.
WebMCP
WebMCP lets a page register "tools" inside the browser so an AI agent running there can use the right function instead of guessing at buttons. The specification is currently a "Draft Community Group Report" dated 20 October 2026 from the Web Machine Learning Community Group, which means it is not on the W3C standards track. Per Chrome's documentation (updated 7 October 2026), it is a "proposed web standard", available through an origin trial from Chrome 149 or an experimental flag, and subject to change: the call is currently document.modelContext.registerTool(...), and the older name navigator.modelContext was changed between drafts.
phatdev.net loads WebMCP only when the browser supports it, and registers tools such as site search, reading the current page as Markdown, switching language and prefilling the contact form. No tool submits a form; the user still presses Send. Shop owners do not need WebMCP yet. Just watch it.
For those who write code
This Python ran for real on 21 October 2026 to check the layers above on phatdev.net (standard library only):
import urllib.request
def get(url, accept=None):
req = urllib.request.Request(url, headers={"Accept": accept} if accept else {})
with urllib.request.urlopen(req) as r:
return r.status, r.headers.get("content-type"), r.read(200).decode("utf-8", "ignore")
site = "https://phatdev.net"
for path, accept in [("/llms.txt", None), ("/vi/blog/mysql-index-explain.md", None),
("/vi/blog/mysql-index-explain", "text/markdown"), ("/.well-known/ai-catalog.json", None)]:
s, ct, head = get(site + path, accept)
print(path, s, ct, "|", head[:40].replace("\n", " "))
Result: all four return 200. /llms.txt is text/plain, both blog addresses (with .md, and with Accept: text/markdown) return text/markdown, and ai-catalog.json returns application/ai-catalog+json.
Mistakes to avoid
The most common one: making llms.txt and forgetting to update it. A file listing deleted pages or missing new ones is worse than none. Generate it from real data, the way phatdev.net does. Second, believing a robots.txt block is security. Private content needs a password, not a Disallow line. Third, a Markdown version that differs from the HTML, leaving you with two contradictory versions of your policy.
What to do next
In order of effort: check your key pages are indexed, check the structured data is correct, and only then add llms.txt and Markdown pages. An MCP server and WebMCP make sense only if your site has a function an AI should be able to call, like product lookup or booking. If you want me to review how far AI can read your current site and suggest what to do first, send me a message.
