Inventory-driven sites — vehicle marketplaces, property portals, anything where the catalogue is the content — get most of their traffic from organic search. They are also, disproportionately, built as client-rendered single-page applications, because that is what the frontend tooling makes easy.
Those two facts are in direct conflict, and the conflict got sharper when AI crawlers became a meaningful discovery channel. This is what the problem actually looks like and what to do about it.
What a crawler sees
A client-rendered SPA serves every URL the same HTML document:
<!doctype html>
<html lang="en">
<head>
<title>Cars — Marketplace</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/assets/index.js"></script>
</body>
</html>Identical for the homepage, for every category page, and for all 200,000 listing URLs. Same title, no description, no canonical, no content.
Google will render the JavaScript and eventually see the real page. But rendering happens in a second pass, on a separate queue, and crawl budget is allocated from the first. What Googlebot sees on pass one is a large number of URLs with an identical title and no text — which is, structurally, the profile of a low-quality auto-generated site. On a site whose whole proposition rests on being trusted, that is a bad first impression to make at scale.
The fix, in order of impact
1. Real URLs and real anchors
Prerequisite for everything else, and frequently the missing piece. If navigation is <button onClick={setPage}>, there is nothing to crawl and nothing to link to — no amount of meta-tag work helps a site with one URL.
Every page needs its own path, and every navigation control needs to be an <a href> that a crawler can follow and a user can middle-click. Intercept the click for client-side routing if you want the SPA feel; keep the anchor.
2. Prerender the pages you can enumerate
Marketing pages, documentation, articles, category pages — anything with a known, bounded URL list — should be rendered to static HTML at build time. For a React app this is not a rewrite: render each route with renderToString, inject the result and the per-page head into the built HTML template, and hydrate on the client.
You get complete HTML for crawlers, a faster first paint for users, and no framework migration.
3. Server-render the head for pages you cannot enumerate
Individual listing pages are the hard case: there can be hundreds of thousands, they change constantly, and prerendering all of them at build time is not viable. The pragmatic answer is to inject the head server-side per request — title, description, canonical, OG tags and structured data — from a single database lookup, while leaving the body to the client.
It is perhaps sixty lines of string replacement on the HTML template, and it converts a mass of identical shells into individually described pages.
<title>2021 Kia Niro 1.6 GDi HEV, 77,205 km — auction lot | Example</title>
<meta name="description" content="2021 Kia Niro hybrid, 77,205 km, automatic, from a Dutch auction. Current price €9,800. Full specification and 34 photographs.">
<link rel="canonical" href="https://example.com/vehicle/openlane/11125938">The URL count problem
Inventory sites generate URLs faster than they generate value, and submitting all of them is actively harmful. Three filters worth applying before anything reaches your sitemap:
- 1Only index listings worth landing on. Live, with photographs, with a resolved make and model. A lot with no images and an unparsed title is not a page you want a searcher to arrive at.
- 2Handle the dead ones deliberately. A sold lot should return
noindex, followor redirect to the model category, not sit in the sitemap indefinitely. Never leave a soft 404 returning HTTP 200 with an empty shell. - 3Set a minimum threshold on category pages. A category page with two results is thin content. Pick a floor — three, five — and noindex below it. Critically, make the sitemap use the same threshold as the meta robots tag, or you are submitting URLs you are simultaneously telling Google to ignore.
Filter permutations are the other multiplier. ?fuel=diesel&sort=price_low&page=3 should not compete with the clean category URL. Emit noindex, follow on filtered variants and canonical them to the base.
Structured data that earns its place
JSON-LD is one of the few places where the same work serves both search engines and AI systems, because both read it directly rather than inferring from layout. Worth having:
Organizationon the homepage — with a legal name and address if you have them. For anyone asking customers to send money, machine-readable identity is half the trust argument.BreadcrumbListon every nested page. Cheap, and it renders in results.FAQPagewhere you have genuine questions and answers. This is the format AI answers quote from most readily.ArticlewithdatePublishedanddateModifiedon editorial content. Dates matter more than people expect for freshness signals.ProductorVehicleon listing pages, where the vehicle data supports it honestly.
One rule: structured data must match the visible page. Marking up a price that is not shown, or an FAQ that does not exist on the page, is a manual-action risk and not a clever shortcut.
Writing for machines as well as people
AI systems reading your pages are doing extraction, not skimming. A few things measurably help, and none of them hurt human readers:
- Answer the question in the first paragraph. Extraction favours a direct answer near the heading over one buried after eight hundred words of preamble.
- Use real heading hierarchy. One
<h1>,<h2>for sections, in order. Headings styled with a div and a font size are invisible as structure. - Put comparative data in tables. A genuine
<table>extracts cleanly; a grid of divs does not. - State dates and figures explicitly. "As of August 2026" is usable; "recently" is not.
- Be specific about uncertainty. "Terms vary by country — verify with the platform" is more useful to an extraction system than false confidence, and it is more honest to your readers.
Machine-readable mirrors
An emerging convention worth adopting: publish /llms.txt — a plain-text map of your site for language models — and serve Markdown versions of your articles alongside the HTML. It costs almost nothing if your content is structured rather than stored as HTML strings, and it removes any ambiguity about what your pages say.
Decide the AI crawler question deliberately
Many sites now block AI crawlers, often by accepting a CDN default rather than by making a decision. The distinction that matters is between two different kinds of bot:
| Type | Examples | What blocking costs you |
|---|---|---|
| Search / citation crawlers | OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot | Your site can never be cited in an AI answer. This is a real discovery channel. |
| Training harvesters | CCBot, Bytespider, and similar | Little, in most cases. Blocking these is a defensible default. |
Blocking everything removes the upside while keeping all the downside. Worth noting too that blocking Google-Extended does not affect Google Search or AI Overviews, which run on standard Googlebot — so it is a lower-stakes setting than it is often assumed to be.
Whatever you choose, check what is actually being served. CDN-managed rules frequently prepend directives to robots.txt and apply firewall rules above it, so the file in your repository may not be the file the crawler receives. Verify with a request carrying the relevant user agent.
An audit worth running
Ten minutes, and it finds most of what is wrong:
- 1
curla category page and a listing page. Is there content in the HTML, or an empty div? - 2Do the two pages have different titles and descriptions?
- 3Is there a canonical, and does it point where you expect?
- 4Do your navigation links appear as
<a href>in the raw HTML? - 5How many URLs are in your sitemap, and what fraction render real content?
- 6What does
robots.txtreturn over the network, not in your repository? - 7Do sold or expired items still return HTTP 200 with a shell?
- 8Are filtered URL variants indexable and competing with clean ones?
If the first question fails, the rest do not matter yet — start with prerendering. Building a marketplace covers where this sits in the wider architecture.
Frequently asked questions
Do AI crawlers execute JavaScript?
Generally not. The crawlers behind AI answers and citations fetch HTML and read it directly rather than running your bundle. A client-rendered single-page application therefore appears to them as an empty page, not a partially loaded one — so the entire AI citation channel is unavailable regardless of how good the content is.
Is client-side rendering bad for Google specifically?
Google does render JavaScript, but in a second pass on a separate queue, while crawl budget is allocated from the first pass. At scale that means large numbers of URLs sharing one title with no text on initial crawl — which resembles a low-quality auto-generated site and wastes budget that could go to pages capable of ranking.
How do I add SEO to an existing React SPA without rewriting it?
Add real URLs and real anchor tags first, then prerender the enumerable routes at build time by rendering each with renderToString and injecting the output plus a per-page head into the built HTML template, hydrating on the client. For unbounded listing URLs, inject just the head server-side per request from a single database lookup. Neither step requires a framework migration.
Should I put every listing URL in my sitemap?
No. Filter to listings worth landing on — live, photographed, with a resolved make and model — and remove sold or expired items rather than leaving them indefinitely. Make sure the sitemap and your meta robots tags use the same threshold, since submitting URLs you are simultaneously telling Google to ignore is a contradiction that wastes crawl budget.
Should I block AI crawlers in robots.txt?
Distinguish the two kinds. Blocking pure training harvesters is a defensible default. Blocking search and citation crawlers means your site can never be the source an AI answer cites, which is a real discovery channel — so blocking everything removes the upside while keeping the downside. Also verify what is actually served, since CDN-managed rules often override the file in your repository.
What is llms.txt?
An emerging convention: a plain-text file at the root of a site that maps its content for language models, often paired with Markdown versions of pages served alongside the HTML. It is inexpensive to generate if your content is stored as structured data rather than HTML strings, and it removes ambiguity about what your pages actually say.
- SEO
- prerendering
- structured data
- AI crawlers
One API, seven auction sources
Normalized search, source-aware detail, CDN image galleries, price history and archive access — all included on every plan.