Methodology

Sites like this have existed for twenty years, built on the same public NHTSA complaint data, ranked by raw complaint count. This page is the argument for why reading that data through a classifier and reporting it differently is actually better, not just newer - stated in numbers, not adjectives, including the numbers that don't flatter it. Read this before trusting a ranking.

What this site does

Every complaint in NHTSA's public Office of Defects Investigation database is read by a local language model (qwen3:14b, run offline against deduplicated complaint text - no cloud inference in production) and assigned a component, a severity, a failure mode, an estimated mileage, and an estimated repair cost. Those per-complaint labels are aggregated into a ranking: for a given make, model, and mileage band, what's most likely to break next, according to the volume and severity of what owners actually reported - not simply how many complaints exist, which is what every complaint-count site already shows.

How accurate is the classifier

Component accuracy: 75.0% (30/40), 95% confidence interval [59.8%, 85.8%]. Measured against a 40-item test set, held out and scored exactly once, after the prompt was tuned against a separate 80-item development set only (five rounds of confusion-driven refinement, 65.0% → 77.5% on that set). No tuning happened after the test set was scored, and none will - a number obtained by re-running the test set until it looks good isn't a real accuracy estimate. The interval is wide because the set is small: at n=40, one item moving from wrong to right shifts the point estimate by 2.5 points. Treat the interval as the honest claim - this system gets the component right on somewhere between 6 and 9 out of 10 complaints, not reliably 3 out of 4.

Severity accuracy: 62.5%, on an ordinal scale (stranded > unsafe > broken > annoyance). 100% of the errors were off by exactly one step on that scale - never scrambled across it. The model has never called an annoyance a stranding or vice versa; when severity is wrong, it's wrong by a degree, not by a category. There's a directional skew worth naming: the model over-calls unsafe when the correct answer was broken more than any other single error type. Read a "stranded" or "unsafe" ranking as reliably serious, and read the boundary between adjacent severities as softer than the label implies.

Separately from accuracy: unsafe means almost nothing about any one complaint, because it's what the model calls almost everything. Measured across every cluster-deflated, confidence-floored classification in the production run (365,910 incidents): 85.9% unsafe, 7.8% annoyance, 4.9% stranded, 1.4% broken. When one label covers 6 out of 7 complaints, seeing it on any single complaint tells you very little, it's weak evidence about that individual failure, even though the label itself is usually correct in isolation (see the accuracy number above). This is a different limitation from the 62.5%/off-by-one accuracy number - even a perfectly accurate classifier would still produce a ranking dominated by "unsafe," because that's what NHTSA complainants overwhelmingly describe their own problems as (consistent with the self-selection bias below: someone motivated enough to file a federal safety complaint tends to frame it in safety terms).

Where severity does carry real signal is in the minority labels: stranded, broken, and annoyance are rare enough (4.9%, 1.4%, and 7.8% respectively) that seeing one of them on a complaint is genuinely informative, and the "most serious" view (serious_count = stranded + unsafe) earns its place as a second ranking on that basis - 36.9% of vehicles show a genuinely different top-5 under it (see docs/DECISIONS.md). We tested the more intuitive-sounding alternative, ranking by stranded alone (the rarest, and arguably most serious, label on its own), and rejected it: it "differs" from the main ranking on 89.3% of vehicles, but that's sparse-count noise, not a stronger signal - half of the components it newly promotes into a top-5 do so on a single stranded report, and a quarter of all vehicles have five or fewer stranded reports across every component combined. Full numbers in docs/DECISIONS.md's "'Most serious' definition tested against stranded-alone" entry. Bottom line: don't read "unsafe" on one complaint as meaningful on its own - read it as a weak, majority default, but do trust a "most serious" ranking difference, since it's backed by real, if smaller, counts.

Failure mode (sudden, gradual, intermittent, etc.) is computed and stored but never published. At 51.25% accuracy on a 9-value field, it's close to what a much simpler heuristic would get by chance plus base rate - not accurate enough to stand behind, and not load-bearing for the ranking the way component and severity are. An internal field that isn't publish-quality stays internal.

Why per-complaint accuracy isn't the number that matters

At 75% component accuracy, roughly 1 in 4 individual complaints is misfiled. That sounds worse than it is for this product's actual purpose, because the errors are diffuse, not systematic - every confusion in the test set was a one-off, no pair of components was repeatedly confused for each other, and no vehicle group was a consistent weak point beyond one (engine) that stayed flat across every tuning round for reasons that look more like gold-label ambiguity than model error.

This site doesn't report what happened to one complaint - it reports a ranking over hundreds of complaints per vehicle. Diffuse, random misclassification is exactly the kind of error a ranking over volume smooths out: a genuinely common failure still shows up as a large number after a quarter of its complaints get misfiled elsewhere, because everything else is losing roughly the same fraction to noise too. Rank order over hundreds of reports is more reliable than any single classification going into it - with one specific exception, below.

Where the ground truth came from

The 120-item gold set these numbers are measured against was drafted by a stronger reference model and reviewed by a human - not independently authored by a domain expert from scratch. This is weaker than an expert-labeled gold standard, and is disclosed as such rather than presented as "human-labeled." Every field was accepted as drafted: 0% override rate across all 120 items, on every field a human reviewed. That number is reported because it's real, not because it's reassuring - a 0% override rate is equally consistent with the drafts being correct and with review anchoring on what it was shown instead of independently re-deriving each field. This dataset can't tell those two apart (a blind, no-suggestion subset was built specifically to distinguish them; it was withdrawn before labeling started), so the gold set's own true error rate is unknown, not low. Full provenance breakdown per field: pipeline/src/pipeline/eval/METHODOLOGY.md.

Known biases, stated precisely

Complaint data is self-selected, and skews negative. Every record here is something an owner troubled enough to file a federal complaint about - not a random sample of what goes wrong with a vehicle. It over-weights problems that are dramatic, safety-relevant, or the subject of an active recall or media coverage (which drives filing), and under-weights minor, common annoyances an owner just gets fixed at a dealer without ever reporting. Two vehicles with an identical true defect rate can show very different complaint volumes here if one recall got more press. Read this site's rankings as "what NHTSA complainants report most, weighted by severity" - a genuinely useful signal - not as "the true failure rate," which no complaint database can measure directly.

Rare components can be inflated by diffuse misclassification - the one place the volume-smooths-it-out argument above doesn't hold. Consider a vehicle with 2,000 classified complaints and a 25% per-complaint error rate. A component with a true count of 300 nets out near 231 after losing and gaining stray votes - close to its true value, because the absolute leakage is small relative to its own size. A component with a true count of 15 undergoes the same absolute leakage but can net out around 17 - meaningfully above its true value, purely from noise. The relative distortion is far larger for rare components than common ones, at the same underlying error rate. The visible symptom is an obscure component ranking unexpectedly high across many unrelated vehicles - a rare code shouldn't plausibly be a top-5 concern for a Honda Civic, a Ford F-150, and a Tesla Model S alike. The production pipeline runs a leakage check after every batch of vehicles classified specifically to catch this: a component rare overall (under 3% of everything classified so far) that still lands in the top 5 for an implausible share of distinct vehicles (over 40%, with at least 50 vehicles classified before the check is trusted) gets flagged and reported, not silently published.

An estimated 1.15% of the complaints eligible to be quoted (856 of 74,136) appear, by NHTSA's own structured make/model fields, to be filed against a different vehicle than their own narrative describes - a complaint narrative opening "THE CONTACT OWNS A [year] [make] [model]" naming a real, different vehicle than the one it's filed under. Caught by a deterministic check (no model call) before a quote is published, and excluded rather than shown - this never changes which vehicle a complaint counts toward or which component it was classified under, only which individual quotes are trustworthy enough to display. The 1.15% figure is conservative: it only catches narratives that explicitly open with "OWNS A," so a mismatch phrased any other way isn't caught. This is a data-entry reality in NHTSA's own source data, not a defect in this project's pipeline - and the fewer complaints a vehicle has, the more one of these can stand out, since a 3-complaint page has nowhere to hide a bad row the way a 500-complaint page does.

8.3% of quotes selected for publication contained third-party business contact information - dealer names, addresses, phone numbers - that NHTSA's own FOIA screening left in. NHTSA redacts complainant PII before publishing its data, but not a dealership's address or phone number quoted inside a narrative. Every quote is passed through a regex pass before publication (phone numbers, emails, VIN-shaped tokens, plate-shaped tokens, and street addresses), applied only to the small set of narratives actually selected for quoting, not the full corpus. 8.3% of those were modified - this is republished on a commercial, ad-supported site, so NHTSA's own screening standard is treated as a starting point to verify, not a guarantee.

Provenance: this corpus spans more than one prompt version

The classification prompt has changed since the production run started (most recently, a guidance line added to prefer EV-specific component codes over an ICE-shaped fallback on hybrid/EV vehicles). Every classification row carries model, prompt_hash, and run_id, so which prompt version produced which label is always traceable - but it means two complaints classified months apart may have been read under different guidance. This is expected and will keep happening as the corpus gets refreshed; it's noted here so a reader comparing numbers across a refresh boundary knows the boundary exists.

What this means for reading the site