Never Hand-Write a Sitemap — Where to Draw the Line Between Derived and Declared

sitemap.xml tells search engines which URLs exist on a site. The format is simple, which makes it look like a file you can maintain by hand.

That's how this site started. It quickly became this:

The volume isn't the problem. The problem is that drift has no consequence you can see. Miss an entry and the build succeeds, the site is fine, and that one URL is simply invisible to search engines. Nobody notices until someone opens Search Console weeks later.

1. A Sitemap Is Output, Not a Document

In this repository sitemap.xml does not exist. The build produces it.

That distinction is the whole idea. A sitemap is a rendering of the site's state, not a second fact living beside the site. Two copies of one fact always drift apart eventually.

2. Derive Whatever Can Be Derived

Post URLs already exist in the repo. There's a file; its front matter has a slug and a date. Writing them into a sitemap is stating the same fact twice.

So posts come straight out of a collection.

Putting the locale loop on the inside is deliberate. We iterate one list — sorted_ko — and generate both the ko and en URLs from each slug. Iterate both locale collections separately and a post present in only one locale quietly lands in the sitemap half-way. Generating both from one list makes that state impossible.

This works because the build guards already enforce ko/en symmetry. With the guard in place, the template is safe to assume symmetry. Validate once up front and everything downstream gets simpler.

3. Scheduled Publishing Comes Along for Free

One thing this structure gave us for nothing: scheduled posts stay out of the sitemap automatically.

Future-dated posts are excluded from the collection, and the sitemap iterates that collection. There is no condition written anywhere for it.

Maintained by hand, this becomes a rule to remember every single time. List an unpublished post's URL and search engines arrive to a 404. Keep the condition in one place and the rule stops existing.

4. Declare What Can't Be Derived

Non-post pages are a different story. Landing, policy, and lab pages are plain .html files copied through — no front matter, no collection. From the file listing alone you cannot tell which ones belong in a sitemap.

Some .html under src/ genuinely must stay out:

The file system carries no signal separating these. Same extension, same directory tree. The difference lives entirely in intent.

So that part is declared as a list:

Note that the locale prefix isn't written here either. The path is /lab/, and the template emits /ko/lab/ and /en/lab/. Half the hand-written lines, and listing only one locale becomes structurally impossible.

Which Pages Fall on Which Side

"Can this be derived?" has a fixed answer per kind of page. For this site:

Five middle rows all funnel into staticPages.js. The list started at six entries and is eight today — the per-category post lists (/posts/athlentic/, /posts/dev/) added two later. Those eight expand across two locales into the sitemap's 16 non-post URLs.

What's worth noticing is that every "no" row is decided by something outside the file. Extension and location are indistinguishable from the "yes" rows. The verification files stand out to a human because their names are odd — but that's human knowledge, not a signal code can read.

5. Say Out Loud What's Still Manual

staticPages.js is the one sitemap-related thing the build won't catch for you. Move or delete a page and this list has to move with it; the build has no idea.

There were two options.

• Add a guard — check that every path in the list corresponds to a real file
• Mark it as manual — a comment at the top of the file, an entry in the rules doc

We took the second. Listed-but-missing is checkable, but present-but-unlisted can't be judged by a build — it has no way to know whether the omission was a mistake or the point. The google*.html files above are exactly that case.

A guard that covers half the problem manufactures the belief that "the build passed, so the sitemap is correct." That belief is more dangerous than having no check at all. So instead of the check, the manual part is labeled where you'll see it.

The Four Ways This List Drifts

"It drifts" is too vague to defend against. There are four concrete shapes, and each announces itself differently.

The first row is the quiet one. Absence is never reported as an error. The other three eventually turn up as red rows in Search Console; an omission turns up nowhere. Noticing that a URL was never indexed requires independently knowing what the list should contain.

Rows two and three arrive as a pair whenever you move or retire a path. You remember the redirect; you don't remember this list. And a sitemap that keeps listing a redirecting URL is telling crawlers to index an address you then answer with "it's over there."

The fourth row is what this design costs. Dropping the locale prefix eliminated "listing only one locale" — and put "building only one locale" in its place. One path entry unconditionally produces both the ko and en URL. Create the actual file only under ko/ and the en URL is a 404. Posts are protected by the symmetry guard; passthrough .html files are not checked at all.

With no way to remove the manual list, the only real defense is that it's short enough to scan. Eight lines get read. Eighty would not, and this defense would quietly stop existing.

6. lastmod Counts, priority Doesn't

Every row carries three values — lastmod, changefreq, priority — which invites treating all three with equal care. Search engines don't read them equally at all.

Google has said outright that it ignores changefreq and priority. Relative importance within a site is judged from internal link structure, not numbers you typed into a sitemap. Set priority: "0.9" on every page and precisely nothing happens.

lastmod is different, with a condition attached — it's used only while it stays trustworthy. Stamp every page's lastmod with the build time and crawlers soon start discounting it. "Everything changed today, every day" carries no information.

So why keep changefreq and priority at all? They're in the spec, non-Google crawlers may still consult them, and keeping the row shape uniform is worth more than the bytes saved. What we don't do is treat them as a dial for rankings. They aren't that lever.

7. A Sitemap Alone Doesn't Decide Indexing

An accurate sitemap feels like it should guarantee indexing, but a sitemap is one signal among several — and when it contradicts the others, the sitemap loses.

Row three is the one this site actively guards against. The root src/index.html is noindex, which is exactly why it is absent from staticPages.js. Do only one of those two and your signals split.

Row four is nastier. A URL blocked in robots.txt can't be fetched, so the crawler has no way to read the noindex inside it. You blocked it to get it out of the index and made it more likely to stay in. To deindex something, allow the crawl and let the noindex be read.

Row five is a cost, not an error. A page with inbound links gets indexed with or without the sitemap. What a sitemap really says is not "index this" but "it's here, come sooner."

8. Where the Line Goes

The rule this work settled on:

The question to ask is "does this information already live somewhere?" If yes, derive it. If no, declare it. The worst option is restating something that already exists — that's the moment you create two places that can disagree.

Forcing derivation where no fact exists fails the other way. "Every HTML file goes in the sitemap" sounds clean, right up until you're asking Google to index your ownership verification files. Intent doesn't derive from code.

9. What We Check After a Build

Making the sitemap an output removes the place a human could break it — and the place a human could look at it. There's no file in the repo, so it never appears in a diff. So we read the output directly.

Step 1's number is checkable in your head. At the time of writing, 16 posts are published across two locales for 32, plus 16 non-post URLs, giving 48. A number that's off usually means a post sits on the scheduled-publishing boundary — which is correct behavior, not a bug. Being able to do the arithmetic is what keeps you from panicking at it.

Step 3 is the only check that actually catches the four drift shapes above. The build doesn't do it. Running it on deploys that added, moved, or deleted a page is enough — a deploy that only added posts has nothing that can drift.

All of that inspects the build output; how the index actually responded shows up in Search Console days later. That lag is why sitemap mistakes are always reported back after whoever made them has forgotten. Looking once, right after deploy, is far cheaper.

10. Questions That Keep Coming Up

Can't we just commit the sitemap?
You can. The moment you do, the same fact lives in two places — the real pages and the committed file. Drift is a matter of time, and the build passes either way. If you're going to commit it, commit what the build generated and adopt a rule that nobody hand-edits it.

When do we need to split the sitemap?
The limit is 50,000 URLs and 50MB uncompressed per file. Past that, split into multiple sitemaps and tie them together with a sitemap index. This site is at 48. Splitting early buys nothing and adds structure.

Should we ping search engines on every deploy?
No. A Sitemap: line in robots.txt is enough for crawlers to find it. Per-deploy pings are mostly wasted, and Google has already retired its sitemap ping endpoint.

Would listing scheduled posts early get them indexed faster on publish day?
It does the opposite. Listing a URL that has no page yet earns a 404 from the crawler and costs that URL some trust. That's exactly why scheduled publishing ties page generation and sitemap membership to the same condition.

Does hreflang belong in the sitemap or in <head>?
Either one suffices. This site does both — the sitemap copy falls out of the template's locale loop for free, and the <head> copy falls out of the layout for free. Had either needed hand-writing, we'd have picked one. Duplication is only harmless when it's free.

Shouldn't you eventually add that guard checking the list against real files?
The four-row table above makes the temptation concrete: "listed but missing" is always an error and therefore checkable, and it catches rows two, three, and four. The reason not to ship it hasn't changed, though — the row it can't catch, "added a page and never listed it," is the quietest and most common mistake of the four, and a passing build would report it as passing. Covering half while looking like full coverage is the problem, and that false confidence costs more than eyeballing eight lines. When the list grows past what an eye can scan, the arithmetic flips.

11. Summary

• The sitemap is output, not a document. Keep it out of the repo; let the build make it.
• Derive facts that already exist. Writing them twice creates something that can drift.
• Generate both locales from one list. That removes half-populated states structurally.
• Decide "published" in exactly one place. Share it and scheduled publishing follows for free.
• Declare intent. Whether a page should be indexed doesn't come from the file system.
• Don't ship half a guard. Label the manual part conspicuously instead, and keep the list short enough to scan.
• Maintain lastmod and nothing else. Google ignores changefreq and priority, and a build-clock lastmod gets ignored too.
• A sitemap loses to contradicting signals. A noindex page has to be absent from the list as well.
• Output never shows up in a diff, so read it after the build. Checking the URL count in your head is the cheapest test there is.

A hand-written list disagreeing with reality is a matter of time, not of carelessness. Any manual list kept long enough will drift. Deriving what can be derived doesn't remove that risk — it shrinks the manual list to something small enough to actually maintain.