Scraping car auction sites: the real cost, versus buying the data

Someone on your team has already built a scraper for one auction site. It took a weekend, it worked, and it produced a CSV that made everyone briefly optimistic. That prototype is the reason this conversation is difficult: it is real evidence that the problem is easy, and it is measuring the wrong thing.

The weekend prototype solves collection for one source, one page layout, one moment in time, with no availability requirement. Production solves collection for seven sources, continuously, while the sources actively change, at a volume where storage and bandwidth stop being rounding errors. This article prices that second thing.

The cost model, line by line

Figures below are order-of-magnitude for a team collecting roughly a million live listings across seven European and overseas sources with images. Your numbers will differ; the structure of the model will not. Treat the ranges as a framework to fill in with your own quotes, not as a benchmark.

Why maintenance never converges

The intuition that scrapers stabilise over time is wrong, and it is wrong for a specific reason: you are not maintaining code against a fixed specification. You are maintaining it against seven independent teams who ship whenever they like and owe you nothing.

Markup drift

A frontend redesign, an A/B test, or a component library upgrade breaks selectors. The A/B test case is the worst: your extraction succeeds for 70% of requests and silently returns nulls for the other 30%, which looks like sparse data rather than a bug and can run for weeks before anyone notices.

Anti-bot escalation

Bot mitigation improves continuously and asymmetrically. Every provider upgrade is a step change for you, arrives without notice, and is followed by an urgent unplanned week. You cannot schedule around it and you cannot predict it.

Semantic drift

The subtlest failure. A source renames a fuel category, changes a currency, starts including VAT where it previously excluded it, or adds a fee to a displayed price. Nothing errors. Your pipeline ingests the new meaning under the old field name, and every downstream number is quietly wrong. This is the class of bug that reaches customers.

Broken scrapers announce themselves. Semantically drifted scrapers do not, and those are the ones that damage trust in your product.

Coverage decay

Rate limits force you to prioritise. You start crawling only the first N pages of each source, or refreshing prices only for lots under a certain age. Coverage drops from 99% to 85% over a year without a single incident, and nobody notices until a customer asks why a car they can see on Auto1 is not in your product.

The legal and contractual dimension

This is genuinely jurisdiction-dependent and this article is not legal advice — but it belongs in the decision, because it is a real risk that rarely makes it into the spreadsheet.

• Most auction platforms' terms of service restrict automated access. Whether those terms bind you, and with what consequence, varies by country and by how you access the site.
• The EU Database Directive creates a sui generis right over substantial investment in compiling a database, separate from copyright in the individual records.
• Photographs are copyrighted works. Re-hosting a source's images is a distinct question from extracting factual specifications, and the answer is not the same.
• Personal data occasionally appears in listing text — seller names, phone numbers, locations — and GDPR applies to it regardless of how you obtained it.

The practical point is not that collection is forbidden. It is that the risk is real, unquantified, and sits with you. Buying from a provider does not make the underlying questions disappear, but it does move the operational relationship with the sources onto someone whose business depends on maintaining it, and it gives you a contract to point at.

Time to first useful product

Cost is one axis. Calendar time is usually the one that decides it.

That last row deserves emphasis. If your product needs price history — comparables, market value estimation, anything trained on past sales — then starting collection today means your first credible model is a year away. There is no way to accelerate it, because the data is generated by the passage of time.

When building it yourself is the right answer

Not a rhetorical section. There are cases where in-house collection is clearly correct, and pretending otherwise would be dishonest.

• One source, one narrow slice. You need Copart Germany, salvage only, one brand. That is a maintainable script, not a platform.
• Data collection is your product. If you sell the feed, the pipeline is your moat and outsourcing it makes no sense.
• You need a field nobody exposes. A specific inspection-sheet annotation, a seller-level signal — if it is not in any commercial feed, you have no alternative.
• A source no provider covers. Regional platforms outside the major seven are often only reachable if you build it.
• Contractual data residency requirements that a third-party API cannot satisfy.

Making the comparison honestly

If you are running this decision internally, build the comparison over 36 months rather than 12, and include these four things that standard estimates omit:

1. 1Loaded engineering cost for ongoing maintenance, not just the initial build. Use a real fully-loaded figure, not salary.
2. 2The opportunity cost of that engineering time — what does not get built.
3. 3The revenue impact of coverage gaps and stale prices during the months your pipeline is degraded.
4. 4The value of the archive you will not have for the first year.

Run that model and the answer is usually clear in one direction or the other, which is the point. For a concrete comparison of the two approaches against our own surface, see the side-by-side; for what the data looks like once it arrives, start with the auction data guide.