The Other Certificate: GPU Performance Grading as ITAD Due Diligence

By GPU Resource Editorial Staff

Certifying that a GPU’s data is gone is not the same as certifying that it still works — and the secondary market is now sophisticated enough to demand both certificates.

GPU Resource, in partnership with Cirkadis | June 2026 | Part 4

Research conducted by GPU Resource in partnership with Cirkadis, a leading aftermarket processor of GPUs. Cirkadis is a sponsor company to GPU Resource.

The earlier articles in this series examined a single question in depth: when an enterprise AI GPU leaves its owner, is the data gone? Part 1: Hardware Architecture and Per-Layer Sanitization mapped the VBIOS, HBM, and BMC layers to the sanitization tiers each one can actually reach. That work answers the question the ITAD industry has spent two decades learning to answer well: is the data gone?

It leaves a second question almost entirely unaddressed: is the GPU still performing to spec?

These are different questions, and only one of them has a mature answer in the aftermarket today. A retired H100 or A100 can pass every data-sanitization step flawlessly and still carry a documented history of HBM memory degradation, accumulated ECC errors, and permanently retired memory pages that quietly reduce both its value and its reliability in the next deployment. The diagnostic due diligence that hyperscalers already run as standard practice before returning hardware to their own production pools should become standard practice in ITAD before hardware enters the secondary market.

The ITAD industry has a mature answer to “is the data gone?” It has almost no standardized answer to “is the GPU still performing to spec?” The secondary market is increasingly sophisticated enough to ask both. A graded, tested H100 with a clean DCGM Level 4 report and a documented page-retirement count commands a different price — and a different level of buyer confidence — than one that ships with a sanitization certificate and nothing else.

GPUs Degrade — and the Degradation Is Measurable

Enterprise AI accelerators do not age gracefully. An H100 running continuous training workloads at a 700W board power sustains thermal and electrical stress far beyond anything a consumer card experiences, and that stress leaves a measurable mark on the silicon.

This is not a theoretical concern. Peer-reviewed academic research drawing on 11.7 million GPU-hours of A100 and H100 telemetry from the Delta supercomputer found that the H100 exhibits a 3.2× lower mean time between uncorrectable ECC memory errors than the A100 — a difference the authors attribute to the H100’s higher memory density, lower HBM3 signaling voltage, and increased stack count, all of which make heat dissipation harder. Degradation is a real, quantified phenomenon, and the newest, most valuable silicon is in some respects the most error-prone.

Source: “A Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs” — peer-reviewed analysis of 11.7M GPU-hours, arxiv.org/html/2503.11901 [1]

ECC Errors and Page Retirement: The Permanent Health Record

Every NVIDIA A100 and H100 maintains a permanent, hardware-level log of memory errors in the InfoROM — a small non-volatile chip on the GPU board. When individual memory cells accumulate errors beyond a threshold, the GPU permanently retires the affected memory pages, excluding them from all future allocations. This action cannot be undone, cannot be reset, and cannot be hidden.

That permanence is exactly what makes it useful. A single command — nvidia-smi -q -d PAGE_RETIREMENT — exposes the full lifetime page-retirement count for each GPU, with single-bit (ECC) and double-bit (ECC) retirements listed separately. It is the closest thing the GPU secondary market has to an odometer reading. NVIDIA caps total retirements at 64 pages combined; a GPU approaching that ceiling is functionally degraded regardless of how pristine it looks on a bench.

Source: NVIDIA Dynamic Page Retirement documentation — 64-page cap, InfoROM logging, query commands — docs.nvidia.com/deploy/dynamic-page-retirement/ [4]; NVIDIA GPU Memory Error Management application note — row remapping and error containment on A100/H100 [5]

XID Errors: The Fault-Event Timeline

Where page retirement is the odometer, the driver’s XID error log is the service history. NVIDIA’s driver records XID codes that capture hardware fault events across a GPU’s operational life — accessible after the fact through dmesg or DCGM. The most consequential of these, XID 79 (“GPU has fallen off the bus”), affected an estimated 3.2% of H100 deployments in their first year of large-scale operation, according to infrastructure reliability data published from Meta’s training clusters. Read in sequence, the XID log gives a grader a timeline of fault events that a physical inspection never could.

Source: Meta Llama 3 training-cluster reliability data — 16,384 H100s, 419 interruptions over 54 days, with GPU and HBM3 failures accounting for nearly half — as reported by Tom’s Hardware [2]

DCGM: NVIDIA’s Diagnostic Toolchain

The instrument that turns these scattered records into a grade is NVIDIA’s Data Center GPU Manager (DCGM), the primary tool for active GPU health assessment. DCGM exposes four diagnostic run levels of increasing depth:

Level 1 (~30 seconds) — driver responsiveness, basic queries, PCIe configuration
Level 2 (~2 minutes) — memory bandwidth, SM compute validation, PCIe throughput
Level 3 (~12–15 minutes) — exhaustive HBM bit-pattern testing, extended SM stress, and NVLink bandwidth validation; catches marginal faults that Levels 1 and 2 miss
Level 4 (~90 minutes) — full memtest suite (the GPU-memory equivalent of memtest86), Input EDPp pulse testing, and extended stress; the most comprehensive diagnostic available

The hyperscaler benchmark is instructive. Microsoft Azure has described running DCGM-based diagnostics across its GPU fleet on a nightly cadence and automatically draining from production any GPU that shows roughly 15% performance degradation. That is the standard the largest operators already hold their own hardware to. ITAD remarketing should apply, at minimum, a Level 3 diagnostic — reserving Level 4 for high-value SXM inventory.

Source: NVIDIA DCGM documentation — Level 1–4 diagnostic descriptions, EUD and memtest details — docs.nvidia.com/datacenter/dcgm/ [3]; Dell DCGM Installation Guide — documents Level 4 at ~90 minutes [6]; Together.ai Practitioner’s Guide to GPU Cluster Testing — real-world acceptance-testing methodology [7]

A Pre-Remarketing Health-Check Workflow

None of this requires tooling an ITAD does not already have access to. A defensible performance grade can be assembled from a short, repeatable sequence run at intake or refurbishment, every step logged against the device serial number:

nvidia-smi -q -d PAGE_RETIREMENT — record the lifetime page-retirement count (single- and double-bit) against the serial number.
nvidia-smi -q -d ECC — capture aggregate ECC error counts, both volatile and lifetime.
DCGM Level 3 minimum (Level 4 for H100 SXM inventory) — log pass/fail against the serial number.
nvidia-smi -q — capture clock speeds, power draw, and thermal readings under load.
nvidia-smi nvlink -s — verify interconnect integrity on SXM form-factor cards.
Output — a per-device health report, tied to the serial number, issued alongside the sanitization certificate.

The result is a second document that travels with the asset: not a claim that the data is gone, but a claim — evidenced, reproducible, serial-keyed — that the hardware still performs.

Bottom Line

The aftermarket has spent years building confidence in the sanitization certificate. The performance certificate is the other half of the same diligence, and the data to produce it is already sitting in every retired GPU’s InfoROM and driver logs waiting to be read. Buyers are beginning to price the difference: a card graded against a documented page-retirement count, ECC history, and a clean DCGM report is a known quantity; a card with a sanitization certificate and nothing else is not.

GPU Resource and Cirkadis see the two-certificate standard — the data is gone, and the hardware works — as the natural maturation of enterprise AI GPU ITAD. The tooling exists, the hyperscaler precedent is established, and the secondary market is already asking the question. What remains is for the ITAD channel to make the answer standard practice.

Sources

[1] “A Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs” — peer-reviewed, 11.7M GPU-hours, H100 vs A100 ECC error rates and degradation patterns — arxiv.org/html/2503.11901

[2] Meta Llama 3 training-cluster failure report — 16,384 H100s, 419 failures over 54 days, GPU and HBM3 failures accounting for nearly half of all interruptions — tomshardware.com (covering Meta’s published data)

[3] NVIDIA DCGM Documentation — Level 1–4 diagnostic descriptions, EUD documentation, memtest suite details — docs.nvidia.com/datacenter/dcgm/

[4] NVIDIA Dynamic Page Retirement Documentation — 64-page retirement cap, InfoROM logging, nvidia-smi query commands — docs.nvidia.com/deploy/dynamic-page-retirement/

[5] NVIDIA GPU Memory Error Management (application note) — row remapping, error containment, page retirement on A100 and H100 — docs.nvidia.com/deploy/pdf/nvidia-gpu-mem-error-mgmt.pdf

[6] Dell DCGM Installation Guide — documents Level 4 at ~90 minutes, practical ITAD implementation detail — dell.com/support/kbdoc/en-us/000219485

[7] Together.ai Practitioner’s Guide to GPU Cluster Testing — real-world acceptance-testing methodology; establishes hyperscaler standard — together.ai/blog

[8] Silicon Value Book: GPU Server Buying Guide — secondary-market buyer expectations around ECC errors, page retirement, and thermal history as grading factors — siliconvaluebook.com/blog/gpu-server-buying-guide-a100-h100