Integrating a vehicle auction API: pagination, caching, retries and rate limits

This is the article to read after you have made your first successful call and before you put anything in front of users. Everything below comes from integrations that broke in a specific, avoidable way.

Authentication and key handling

Keys go in a header, never in a query string — query strings end up in access logs, browser history, referrer headers and error trackers.

# Either form works
curl -H "X-API-Key: $API_KEY"          "https://api.thecarapi.com/api/search?limit=1"
curl -H "Authorization: Bearer $API_KEY" "https://api.thecarapi.com/api/search?limit=1"

One operational detail worth knowing before it bites: repeated authentication failures typically trigger lockouts. A misconfigured staging environment hammering the API with a stale key can lock out the IP that production shares. Keep environments on separate keys and fail fast on a 401 rather than retrying it.

Cache by data type, not by endpoint

The single highest-leverage decision in the whole integration. The four data layers change at completely different rates, and one TTL across all of them is either far too aggressive or far too conservative.

The detail row deserves care. Specification and price arrive in the same payload but have opposite requirements. The clean resolution is to cache the detail response for the specification fields and re-read price separately on any screen where the user might act on it.

Many GET endpoints are already cached at the edge and return an X-Cache header. Log it — an unexpectedly low hit rate usually means you are varying a parameter you did not intend to, such as passing a timestamp or an unstable sort into an otherwise identical query.

Pagination without falling off the end

Three rules that cover almost every pagination problem in this domain.

Skip the count when you do not need it

Counting matching rows across a large filtered set is frequently the most expensive part of a search. On infinite scroll or a "load more" pattern you never display the total, so pass include_total=false and drop the cost entirely.

Respect the depth limit

Deep offsets are expensive everywhere, and most APIs cap them. If you need to walk an entire result set — for a nightly reconciliation, say — do not page to offset 50,000. Partition the query instead: iterate by brand, or by year, or by price band, keeping each partition shallow.

# Wrong: falls off the depth limit and gets slower every page
offset = 0
while True:
    page = search(brand="bmw", offset=offset, limit=100)
    if not page["results"]:
        break
    offset += 100

# Right: partition so every query stays shallow
for year in range(2015, 2027):
    offset = 0
    while True:
        page = search(brand="bmw", year_from=year, year_to=year,
                      offset=offset, limit=100, include_total=False)
        if not page["results"]:
            break
        yield from page["results"]
        offset += 100

Seed your random sorts

If you sort randomly for variety, the ordering must be stable across pages or page two will repeat rows from page one. The search response returns a random_seed; pass it back on subsequent pages.

Retries, and what not to retry

A retry policy that retries the wrong things turns a small problem into an outage. The rule is: retry only what could plausibly succeed next time.

Use jitter. Without it, every client that failed during a brief outage retries at the same instant and produces a thundering herd on recovery — turning a thirty-second blip into a sustained incident.

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

async function call(url, { attempts = 4 } = {}) {
  for (let i = 0; i < attempts; i++) {
    const res = await fetch(url, { headers: { 'X-API-Key': process.env.API_KEY } });
    if (res.ok) return res.json();
    if (!RETRYABLE.has(res.status)) {
      throw new Error(`${res.status} ${await res.text()}`);   // fail fast, do not retry
    }
    const after = Number(res.headers.get('retry-after')) * 1000;
    const backoff = after || 2 ** i * 250 + Math.random() * 250;  // jitter
    await new Promise(r => setTimeout(r, backoff));
  }
  throw new Error('exhausted retries');
}

Log the request id

Responses carry a request id, also returned as X-Request-ID. Log it with every call, and surface it in your own error messages. It converts a support conversation from "search was slow this morning" into a specific call someone can look up. This costs one line and is the highest-value logging you will add.

Log contract_version too. When a response shape changes, the version in your logs tells you exactly when, which is otherwise a genuinely difficult thing to reconstruct.

Handle sparse fields properly

Fields are present when the source provides them, which means absent fields are the normal case rather than an error. Two habits:

• Never assume a field exists. VIN, horsepower, photo count and equipment lists are all commonly absent. Destructuring without a default is the most common cause of a crash in a vehicle frontend.
• Distinguish null from zero. A mileage of 0 means the odometer read zero. null means the source did not say. Rendering both as "0 km" makes every unknown-mileage car look new.

Health checks and graceful degradation

A vehicle marketplace does not need to go down because one endpoint is slow. Decide in advance what degrades and how:

1. 1If images fail, render cards with a placeholder. Do not fail the page.
2. 2If facets fail, render search without filter counts. Users can still search.
3. 3If price history fails, hide the chart. It is an enhancement, not the product.
4. 4If search fails, that is a real outage — surface it honestly rather than showing an empty result set, which users read as "no cars match" and act on.

That last distinction matters more than it looks. An empty result set and a failed request are completely different messages to a user, and conflating them makes your product look like it has no inventory.

curl -s "https://api.thecarapi.com/api/health"

A pre-launch checklist

• API key is server-side only, and staging uses a different key from production.
• Cache TTLs are set per data type; live prices are not cached.
• Retries cover only retryable statuses, with jitter and an attempt cap.
• include_total=false on any view that does not display a count.
• Deep pagination is partitioned rather than offset-walked.
• Request ids are logged and surfaced in errors.
• Every field access tolerates absence; null and zero are distinguished.
• Degradation behaviour is defined per endpoint, and empty results are visibly distinct from failures.
• Fill-rate monitoring is running before launch, not after the first data incident.

Endpoint-level detail for everything referenced here is in the API reference, and building a marketplace covers the architecture these mechanics sit inside.