---
title: "Making Your Website Readable for AI in 2026: What Is an Official Standard and What Is Only a Proposal"
url: https://phatdev.net/en/blog/website-ready-for-ai-agents
lang: en
updated: 2026-10-21
alternate:
  vi: https://phatdev.net/vi/blog/website-ready-for-ai-agents
markdown: https://phatdev.net/en/blog/website-ready-for-ai-agents.md
---

# Making Your Website Readable for AI in 2026: What Is an Official Standard and What Is Only a Proposal

> An AI answers wrongly about your shop because it cannot read the policy page. A walk through robots.txt, structured data, llms.txt, Markdown pages, MCP Server Cards, AI Catalog and WebMCP, saying which are standards and which are proposals, with phatdev.net as the example.

Categories: SEO | Published: 2026-10-21 | Level: Intermediate | 7 min read

Tags: llms-txt, markdown, structured-data, webmcp, robots-txt, ai-agents

You ask an AI assistant: "How long is the warranty at Shop X?" It answers with total confidence and gets it wrong, because the shop's policy page is a tangle of buttons, banners and scripts with the real wording buried inside. The customer trusts the answer and walks into the shop expecting the wrong thing.

More people ask an AI before they ever open a website. This post goes through each thing being discussed in 2026 and says plainly which are official standards and which are only proposals. I use phatdev.net as the example, because I built these pieces for real and checked them with commands.

## First: what Google says

If you worry about the AI parts of Google Search (AI Overviews, AI Mode), Google Search Central's documentation (checked 21 October 2026) is direct: you do not need new machine-readable files, AI text files or special markup to appear in them. There is no special structured data to add either. The requirement is that the page is indexed and eligible to show a snippet in Search. That page does not mention llms.txt.

So the foundation is still ordinary SEO. Everything below is an extra layer for other AI agents, not a replacement for that foundation. For more, see [Making Sure Google Sees Both Languages of Your Next.js Site](https://phatdev.net/en/blog/technical-seo-nextjs).

![A web building with a big front door for visitors and a side entrance carrying flattened pages for AI to read](https://phatdev.net/uploads/blog/website-ready-for-ai-agents/two-readers-1920.webp)

## Standard or proposal

| Item | What it is | Status (checked 21 Oct 2026) |
|---|---|---|
| robots.txt | Tells bots which paths they may visit | Long-standing convention; polite bots obey, but it is only a request |
| Structured data (schema.org JSON-LD) | Labels telling machines "this is an article, a product, an author" | Widespread practice, supported by Google |
| Google-Extended | A robots.txt token to opt content out of Gemini use | Official Google control |
| llms.txt | An index file for AI | Community proposal, not a standard |
| Markdown version of a page | A flat, clean page for AI | Part of the llms.txt proposal |
| MCP Server Card | A business card describing an MCP server | Official, optional MCP extension |
| AI Catalog (ai-catalog.json) | A per-domain catalog of AI resources | Still being drafted |
| WebMCP | A page registers tools for AI inside the browser | Community group draft, not stable |

Now each one.

## robots.txt and AI bot permissions

robots.txt is a file at your site root telling bots which paths they may visit. Google describes it as a way to reduce server load, not a way to hide pages: a blocked page can still show up in results if other sites link to it, and bots that are not polite may ignore it. To truly hide something, use noindex or a password.

For AI there are two separate decisions. One is which bots may read. The other is whether your content may be used for training. Google-Extended is the token for the part where content is used for Gemini models (training and "grounding", meaning search data fed into answers), and Google says it does not affect whether your site appears in Google Search. Other AI companies have their own bot names, so read each one's documentation before you write rules.

phatdev.net currently has only a general rule: all bots may read the public part, while the admin area and API are blocked. There are no rules for individual AI bots, because the site wants to be found by as many people as possible. If your content is something you sell, think differently.

![A gatekeeper booth at a site entrance, some visitors passing through an open lane, one held at a barrier, a rulebook board with an allow or block dial](https://phatdev.net/uploads/blog/website-ready-for-ai-agents/bot-gate-1920.webp)

## Structured data

These are JSON-LD blocks in a page that tell machines: this is an article, who wrote it, when. Google says no AI-specific schema is needed. Just keep it correct and matching what is visible on the page.

## llms.txt and Markdown pages

This proposal came from Jeremy Howard on 3 September 2024; per llmstxt.org the current version was edited on 10 August 2026, and the site itself calls it "a proposal". The idea: put an `/llms.txt` file at your site root, written in Markdown, with the site name, a short summary and a list of links to key pages. Alongside it, each page should offer a clean Markdown version at the same address plus `.md`, and the HTML page should point to it with `link rel="alternate" type="text/markdown"`.

I found no official Google documentation saying Google uses llms.txt. Treat it as a cheap extra that helps agents and tools willing to read it, not as a ranking lever.

phatdev.net does all three layers: `/llms.txt` (an index with a description of each page), `/llms-full-vi.txt` and `/llms-full-en.txt` (all public content, combined by language, about 1.1 MB each; `/llms-full.txt` is an index pointing to these two files) and a `.md` version of every page. You can fetch the Markdown two ways: append `.md` to the address, or send a request with `Accept: text/markdown`. HTML pages carry the `rel="alternate"` link to the Markdown copy.

![A signpost board at the site root with a list of links leading to tidy stacked plain pages, plus a card catalog](https://phatdev.net/uploads/blog/website-ready-for-ai-agents/doc-map-1920.webp)

## MCP Server Card and AI Catalog

If your site runs an MCP server for AI to query (see [What Is MCP? How AI Connects to Your Tools and Data, and the Risks to Know](https://phatdev.net/en/blog/mcp-explained)), AI needs to find it. An MCP Server Card is a file describing the server before connecting: address, connection type, protocol versions. It is an official, optional MCP extension (SEP-2127, status Final). phatdev.net serves it at https://phatdev.net/api/mcp/server-card.

AI Catalog (`/.well-known/ai-catalog.json`) is a general catalog of a domain's AI resources, which can point to a Server Card. The Agent-Card/ai-catalog repo calls itself a "temporary working repo" maintained under the Linux Foundation, and the specification is still in progress. phatdev.net publishes the file, but treat it as an early experiment and recheck the current docs because the path may change.

## WebMCP

WebMCP lets a page register "tools" inside the browser so an AI agent running there can use the right function instead of guessing at buttons. The specification is currently a "Draft Community Group Report" dated 20 October 2026 from the Web Machine Learning Community Group, which means it is not on the W3C standards track. Per Chrome's documentation (updated 7 October 2026), it is a "proposed web standard", available through an origin trial from Chrome 149 or an experimental flag, and subject to change: the call is currently `document.modelContext.registerTool(...)`, and the older name `navigator.modelContext` was changed between drafts.

phatdev.net loads WebMCP only when the browser supports it, and registers tools such as site search, reading the current page as Markdown, switching language and prefilling the contact form. No tool submits a form; the user still presses Send. Shop owners do not need WebMCP yet. Just watch it.

## For those who write code

This Python ran for real on 21 October 2026 to check the layers above on phatdev.net (standard library only):

```python
import urllib.request
def get(url, accept=None):
    req = urllib.request.Request(url, headers={"Accept": accept} if accept else {})
    with urllib.request.urlopen(req) as r:
        return r.status, r.headers.get("content-type"), r.read(200).decode("utf-8", "ignore")
site = "https://phatdev.net"
for path, accept in [("/llms.txt", None), ("/vi/blog/mysql-index-explain.md", None),
                     ("/vi/blog/mysql-index-explain", "text/markdown"), ("/.well-known/ai-catalog.json", None)]:
    s, ct, head = get(site + path, accept)
    print(path, s, ct, "|", head[:40].replace("\n", " "))
```

Result: all four return 200. `/llms.txt` is `text/plain`, both blog addresses (with `.md`, and with `Accept: text/markdown`) return `text/markdown`, and `ai-catalog.json` returns `application/ai-catalog+json`.

## Mistakes to avoid

The most common one: making llms.txt and forgetting to update it. A file listing deleted pages or missing new ones is worse than none. Generate it from real data, the way phatdev.net does. Second, believing a robots.txt block is security. Private content needs a password, not a Disallow line. Third, a Markdown version that differs from the HTML, leaving you with two contradictory versions of your policy.

## What to do next

In order of effort: check your key pages are indexed, check the structured data is correct, and only then add llms.txt and Markdown pages. An MCP server and WebMCP make sense only if your site has a function an AI should be able to call, like product lookup or booking. If you want me to review how far AI can read your current site and suggest what to do first, send me a message.

## References

- [llms.txt proposal](https://llmstxt.org/)
- [Google Search Central - AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google - Common crawlers (Google-Extended)](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- [Google - Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [WebMCP draft specification](https://webmachinelearning.github.io/webmcp/)
- [Chrome for Developers - WebMCP](https://developer.chrome.com/docs/ai/webmcp)
- [MCP SEP-2127 Server Cards](https://modelcontextprotocol.io/seps/2127-mcp-server-cards)
- [AI Catalog (Agent-Card/ai-catalog)](https://github.com/Agent-Card/ai-catalog)

## Related posts

- [Making Sure Google Sees Both Languages of Your Next.js Site](https://phatdev.net/en/blog/technical-seo-nextjs.md)
- [What Is MCP? How AI Connects to Your Tools and Data, and the Risks to Know](https://phatdev.net/en/blog/mcp-explained.md)
- [Basic SEO for Online Shops: What Owners Can Do and What Needs a Developer](https://phatdev.net/en/blog/seo-basics-online-shop.md)