# How Readable works

Source: https://agentreadable.site/docs/

Docs

Everything Readable does, what it doesn't, and how to undo it.

## What Readable does

You give Readable a website address. It reads up to 30 public pages of that site, removes navigation, scripts, cookie banners and hidden elements, converts what is left to markdown, and writes three things:

- **`llms.txt`**: the site's name, a one-line summary, and links to every page, grouped into _About_, _Tokenomics_, _Docs_ and _Optional_.
- **`llms-full.txt`**: the markdown of every page in one file.
- **`pages/*.md`**: one markdown file per page.

You download them as a zip and upload them to your own site. Readable never hosts them for you, and never needs a login, a wallet or access to your hosting.

The same input always gives the same output, byte for byte: no timestamps, no random order.

## What llms.txt is

llms.txt is a [community proposal](https://llmstxt.org/): a markdown file at the root of a website (or of a folder) that tells language models and AI agents what the site is and where its important pages are, ideally as plain markdown. It is **not an official standard**, and no AI product is required to read it.

Version 2 of the proposal (August 2026) also describes where a page's markdown version lives: at the page URL with `.md` appended (`page.html.md`) or with the extension replaced (`page.md`), or wherever the page points with `<link rel="alternate" type="text/markdown">`. The checker accepts all of these. Readable writes its copies to `pages/<page>.md` and links them from llms.txt.

## Upload your files

Upload everything to your site's **web root**, so these addresses work:

```
https://your-site.com/llms.txt
https://your-site.com/llms-full.txt
https://your-site.com/pages/<page>.md
```

| Your site runs on | Put the files in |
| --- | --- |
| Next.js, Vite, Astro, SvelteKit, most frameworks | `public/` (in Next.js, **not** the `pages/` folder) |
| WordPress | the folder that contains `wp-config.php` |
| GitHub Pages, Netlify, Cloudflare Pages, static hosts | the folder you publish |
| Webflow, Wix, Squarespace, Framer | usually not possible: these builders don't allow it |

Then run the [checker](https://agentreadable.site/check/) to confirm your score.

### Scanning a folder

Enter a folder address ending in `/` (for example a GitHub Pages project site, `https://you.github.io/project/`) and Readable scans only that folder. The files are then meant for that folder: `https://you.github.io/project/llms.txt` and `…/project/pages/*.md`. Version 2 of the proposal says an llms.txt covers the pages under its folder. Enter the homepage to scan the whole site.

### Optional: point pages to their copies

The result page also gives you snippets that let agents go straight from a page to its markdown copy, as version 2 of the proposal describes:

- a `<link rel="describedby" href="/llms.txt">` tag for every page,
- a `<link rel="alternate" type="text/markdown" href="/pages/….md">` tag per page,
- or the same as `Link:` headers, in the `_headers` format of Netlify and Cloudflare Pages.

In Next.js, set `alternates.types['text/markdown']` in a page's metadata. All of this is optional: agents that read llms.txt find the copies anyway.

Readable keeps every copy in one `pages/` folder on purpose. The proposal also allows copies next to each page (`/about/index.md`), but that creates real folders like `/about/`, and on WordPress and many PHP or Nginx setups a real folder takes priority over the page with the same address, which can break it.

## The score

The checker fetches `/llms.txt` and the pages it lists, the way an agent would. It checks at most 10 links per site. For a URL in a folder (like a GitHub Pages project site), it uses the most specific llms.txt it finds, as version 2 of the proposal describes.

| Check | Points |
| --- | --- |
| `/llms.txt` exists, has an H1 title (`# Name`) and at least one section (`## Section`) | 20 |
| A summary blockquote (`> …`) under the title | 15 |
| Markdown copies of listed pages: the link is itself markdown, or the page has a markdown version | 25, proportional |
| Listed pages show at least 200 characters of text without running JavaScript | 20, proportional |
| Links in llms.txt answer with 2xx | 20, proportional |

80 and up is **AGENT-READABLE**, 50 to 79 **NEEDS WORK**, below 50 **NOT READABLE**. Results are cached for 10 minutes.

## Pages that need JavaScript

Readable does not run JavaScript and has no headless browser. If a page shows almost no text until scripts run (a typical single-page app), Readable flags it and leaves it out. To fix it, render the page on the server (static generation or server-side rendering), or write its markdown copy by hand.

## Badge

```
https://agentreadable.site/b/your-site.com.svg
https://agentreadable.site/b/your-site.com.svg?theme=light
```

The badge shows your live score and is cached for 24 hours. Make one on the [badge page](https://agentreadable.site/badge/).

## Command line

The same scan and checker run in your terminal with Node.js 20 or later. Nothing goes through agentreadable.site:

```sh
npx agentreadable scan your-site.com --out public
npx agentreadable check your-site.com --min 80
```

- `scan` writes `llms.txt`, `llms-full.txt` and `pages/*.md` into `--out` (default `readable/`) and prints the score the site will get once they are uploaded.
- `check` prints the five checks and the score. With `--min`, it exits with code 1 when the score is lower, so a CI build can fail when agents can no longer read your site. `--json` prints the result as JSON.
- `--allow-private` lets it fetch a development server on `localhost`.

In GitHub Actions:

```yaml
- uses: actions/setup-node@v4
  with:
    node-version: 22
- run: npx --yes agentreadable@0.1 check https://your-site.com --min 80
```

## The crawler

Readable fetches pages with this user agent:

```
Readable/0.1 (+https://agentreadable.site/docs#crawler)
```

- It only fetches pages when someone asks for it: a scan, a check, or a badge view. The only scheduled fetches are the daily re-check of sites someone chose to monitor and the weekly [Pons Readability Index](https://agentreadable.site/docs/#pons).
- It reads `robots.txt` before anything else and obeys it for the `Readable` user agent and for `*`, in scans, checks, badges, the monitor and the index, `llms.txt` included. If `robots.txt` can't be fetched because of a server error, it fetches nothing. Links in your llms.txt that `robots.txt` blocks are skipped, not counted as broken.
- It never visits login, sign-up, admin, account or dashboard paths, and never sends cookies.
- At most 30 pages per scan, 2 MB per page, 3 redirects per request, and only public `http`/`https` addresses: private networks are blocked.

To keep Readable off your site:

```
User-agent: Readable
Disallow: /
```

## Monitor

The [monitor](https://agentreadable.site/monitor/) re-checks a site about once a day and alerts you on Telegram or by webhook when:

- the score drops below 80 (while it stays low, you get a reminder at most once a week),
- a link in your llms.txt stops working,
- a page linked from the watched page is missing from your llms.txt. Pages already there at the first check count as the baseline.

Each check reads your llms.txt, up to 3 of the links it lists, and the watched page. The last 30 scores are kept.

**Telegram:** message the Readable bot; it replies with your chat ID.

**Webhook:** an `https` address on the public internet. Redirects are not followed. Readable sends a `POST` with a JSON body like:

```json
{
  "type": "readable.alert",
  "url": "https://your-site.com/",
  "score": 55,
  "label": "NEEDS WORK",
  "reasons": ["The score is 55/100 (NEEDS WORK)."],
  "brokenLinks": [],
  "newPages": ["/staking"]
}
```

There is no account. When you start watching, you get a private link to see the history or stop the watch; we only store a hash of its secret, so keep the link safe. Capacity is limited to keep the service free (250 sites).

## Pons Readability Index

The [index](https://agentreadable.site/pons/) ranks the websites of tokens launched on Pons, the token launchpad on Robinhood Chain, by the same score as the [checker](https://agentreadable.site/check/).

- **Which tokens:** tokens that graduated from their bonding curve (most launches never do) and whose creator entered a website. Links to X, Telegram, Discord, Linktree, GitHub, chart sites and the like are not websites and are left out. The index holds the 150 most recent of them.
- **Where the data comes from:** the public launch records of the Pons V2 factory on Robinhood Chain (chain ID 4663), read through the chain's public RPC: graduation events, and each token's name, symbol and website as stored in its contract. Nothing is signed and no wallet is involved.
- **How it updates:** sites are checked through the week, each once, with up to 5 links from its llms.txt. Every Monday (UTC) the finished week is published. The 7-day change compares a token's score with last week's index; "new" means the token was not in it.
- **Launched this week** means the token launched in the same week it is listed for.
- A site whose `robots.txt` blocks Readable is left out. So is, for that week, a site that needs too many requests to check (long redirect chains).

The index says nothing about a token's value, team or safety. It is not investment advice, and it is not affiliated with Pons or Robinhood. To keep your site off the list, block Readable in `robots.txt`; it drops out within two weekly updates.

## Privacy

- No accounts, no cookies, no tracking scripts.
- A scan runs in your browser. Our server only fetches the pages for you and does not store them. The result stays in your browser tab.
- If you click **Publish result page**, we store your llms.txt and the scores for 60 days at a public link.
- If you start watching a site, we store its address, your Telegram chat ID or webhook URL, and the last 30 scores until you stop the watch.
- We don't keep request logs. Cloudflare runs our servers and processes requests to deliver them. Fonts load from Google Fonts.

## Limits

- 30 pages per scan, closest pages first; 2 MB per page.
- Scans, checks and publishing are rate-limited per network to keep the service free.
- The checker scores at most 10 links of an llms.txt.
- Published result pages expire after 60 days.

## Not guaranteed today

- AI products are not required to read llms.txt. Some agents and developer tools fetch it when they find it; many don't.
- llms.txt is a proposal, and its format can still change.
- It is not an SEO tool. Don't expect search ranking changes.

## What this is not

- Not a security audit, and not a safety check for any token.
- Not investment advice.
- Not hosting, and not a browser: no JavaScript is run.