Making a Static Site Crawlable: HTTP Headers, Structured Data, and AI Discovery

This site is deployed as a Cloudflare Worker with static assets. The hub at mohamedzaam.com is minimal HTML with no build step, so there is no framework handling SEO for me. I had to wire up crawlability signals manually across a few layers: HTTP headers, robots.txt, HTML, and a separate discovery file for LLM systems. Here is what I changed and why.

Fixing the sitemap.xml Content-Type

When I ran a verification test by asking an LLM to fetch the key URLs on the site, it reported sitemap.xml as binary data. The server was returning Content-Type: application/xml without a charset declaration.

The issue is that application/ types cover both binary formats (PDF, ZIP, protobuf) and text-based formats (JSON, XML). Some HTTP clients use this ambiguity to apply heuristic binary detection: if the type is application/ and there is no explicit charset, the content might be binary. text/* types do not have this problem because they are always character-encoded by definition.

Cloudflare Workers Assets supports a _headers file in the asset directory for setting custom response headers per path:

/sitemap.xml
  Content-Type: application/xml; charset=utf-8

/llms.txt
  Content-Type: text/plain; charset=utf-8

/robots.txt
  Content-Type: text/plain; charset=utf-8

After deploying, curl -sI https://mohamedzaam.com/sitemap.xml confirmed:

content-type: application/xml; charset=utf-8
x-content-type-options: nosniff

X-Content-Type-Options: nosniff is applied by Cloudflare by default. It tells conforming clients to respect the declared type and skip MIME sniffing, which is exactly the heuristic that was causing the binary misclassification.

The LLM browsing tool was wrong

Worth flagging: even before the fix, the sitemap was being served as valid UTF-8 XML and search engine crawlers had no problem with it. The LLM's browsing tool reported Content-Type: application/xml while simultaneously calling the content unreadable. After verifying with curl, the content was fine.

LLM browsing tools are primarily built for HTML. When they receive a non-HTML content type, many of them classify the response as binary regardless of the actual encoding. It is not a reliable way to test whether your XML assets are working. Use curl -sI for headers and curl -s for content.

The fix was still worth making for correctness, but I want to be clear that the sitemap was already functional for its actual consumers before it.

Explicit AI Crawler Directives in robots.txt

My robots.txt had a single wildcard rule:

User-agent: *
Allow: /

Sitemap: https://mohamedzaam.com/sitemap.xml

This permits everything and is technically complete. The reason to add explicit per-agent entries is that some AI systems distinguish between implicit permission from a wildcard and explicit permission via a named user-agent entry. A site with only a wildcard may be treated as having not specifically considered AI crawling; a site with explicit entries signals deliberate opt-in. This affects crawl prioritization in some pipelines.

I added explicit Allow: / entries for every major AI crawler:

User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: meta-externalagent
Allow: /

User-agent: Applebot-Extended
Allow: /

GPTBot is OpenAI's training and browsing crawler. Google-Extended is Google's AI-specific crawler, separate from Googlebot. CCBot is Common Crawl, which feeds training data to many open-weight models. anthropic-ai and Claude-Web are Anthropic's crawlers for training and live browsing. The rest cover Perplexity, ByteDance, Meta, and Apple AI.

Nothing functionally changes. The effective permission is identical to the wildcard. The point is to be explicit about intent.

HTML Discovery Signals

I added three and elements to the head of each page. These are for crawlers that arrive through link traversal rather than at the domain root, so they do not need to first find and parse robots.txt to know what to do.

Meta robots directive

<meta name="robots" content="index, follow" />

Redundant with robots.txt but evaluated independently by some crawlers on a per-page basis. If a crawler fetches a page without consulting robots.txt first, this makes the intent clear.

Sitemap link

<link rel="sitemap" type="application/xml" title="Sitemap" href="/sitemap.xml" />

Creates a machine-readable association between the HTML document and the XML sitemap. A crawler arriving at any page through a link will find the sitemap directly without a separate robots.txt lookup.

RSS autodiscovery

<link rel="alternate" type="application/rss+xml" title="Technical Writing RSS"
    href="https://docs.mohamedzaam.com/rss.xml" />
<link rel="alternate" type="application/rss+xml" title="Casual Writing RSS"
    href="https://blog.mohamedzaam.com/rss.xml" />

The standard RSS autodiscovery mechanism. Feed readers and content aggregators that check for this tag will find both subdomain feeds from the hub page. Both feeds contain full article text, which makes them useful for AI systems doing bulk content ingestion.

JSON-LD Structured Data

Without JSON-LD, search engines and AI systems infer entity information from prose. That inference is probabilistic and inconsistent across crawlers. JSON-LD gives them a deterministic channel that does not depend on natural language parsing.

I added a Person and WebSite entity to the hub page, linked together:

{
    "@context": "https://schema.org",
    "@graph": [
        {
            "@type": "Person",
            "@id": "https://mohamedzaam.com/#person",
            "name": "Mohamed Zaam",
            "url": "https://mohamedzaam.com",
            "email": "contact@mohamedzaam.com",
            "sameAs": ["https://github.com/mohamedzaam"],
            "jobTitle": "Systems and Security Engineer",
            "description": "Systems and security engineer building secure backend systems, conducting offensive security research, and writing on cybersecurity, systems programming, OSINT, and privacy tools."
        },
        {
            "@type": "WebSite",
            "@id": "https://mohamedzaam.com/#website",
            "url": "https://mohamedzaam.com",
            "name": "Mohamed Zaam",
            "description": "Canonical entry point for professional and personal work by Mohamed Zaam.",
            "author": { "@id": "https://mohamedzaam.com/#person" }
        }
    ]
}

The @graph array lets me define multiple entities in one block and link them via @id references. The sameAs property on the Person entity tells crawlers that the GitHub profile is the same real-world entity. Knowledge graph systems use sameAs to consolidate information from multiple sources into a single entity representation, which affects how the entity appears in rich results and AI-generated summaries.

llms.txt

llms.txt is a plain text Markdown file at /llms.txt. It is not the same as sitemap.xml. The sitemap lists URLs with crawl metadata and lets a URL-based crawler decide what to fetch. llms.txt provides a structured index with descriptions so an LLM can assess what content exists without fetching every URL individually. One request, full catalog.

The format I use:

A # header with my name
A blockquote summarizing who I am and what the site covers
## sections per content domain, each starting with the RSS feed link, followed by individual article links with descriptions
An ## Optional section for secondary resources

The file is regenerated automatically by a GitHub Actions workflow that fetches both subdomain RSS feeds on push to master and on a weekly schedule. The same workflow regenerates sitemap.xml. Both files stay in sync with published content without any manual updates.

The LLM verification test confirmed this is the most useful thing on the site for AI discoverability. It reported llms.txt as immediately useful and described it as a complete content catalog in one request, which is exactly what it is designed to be.

DOM Fix

Both pages had a paragraph element nested inside another paragraph element in the footer:

<p>
    &copy; 2026
    <a href="https://mohamedzaam.com">Mohamed Zaam</a>
    <br /><p class="lead">systems &middot; security &middot; solutions</p>
</p>

A paragraph element cannot contain block-level descendants. When a parser hits an opening paragraph element inside an open paragraph element, it implicitly closes the outer one first. The resulting DOM does not match what was written. Crawlers that operate on the parsed DOM rather than the raw source see a different structure.

Fixed to sibling paragraphs:

<p>&copy; 2026 <a href="https://mohamedzaam.com">Mohamed Zaam</a></p>
<p class="lead">systems &middot; security &middot; solutions</p>

Minor in terms of crawlability impact, but invalid DOM construction is a source of inconsistency that compounds across every tool that touches the page.

Summary

The changes split into two groups. Correctness fixes: the content-type header and the invalid HTML. Explicitness fixes: everything else. The JSON-LD, meta tags, link elements, and per-agent robots.txt entries are all technically redundant with default crawler behavior, but they remove ambiguity and give crawlers machine-readable channels for information they would otherwise have to infer.

sitemap.xml and llms.txt serve different consumers. The sitemap is for URL-based search engine crawlers. llms.txt is for LLM systems that want to assess content before committing to crawling individual URLs. A site without llms.txt is poorly positioned for AI discovery regardless of how well the sitemap is configured.