HDD Overwrite Verification: What a Random Sample Can Prove

Short answer

A random read sample after an HDD overwrite can quantify the chance of missing a specified number of mismatching blocks in a defined, readable population. It cannot prove that every physical location was erased, establish zero residual data, or convert a host overwrite into a purge method.

Use sampling only as an additional check required or justified by your policy. NIST SP 800-88 Rev. 2 does not require elaborate content sampling universally after clear or purge. Method selection, operation status, anomalies, and the validation decision still matter. Start with the sanitization process guide if those decisions are not yet documented.

Who this is for

Assume an authorized operator has already performed an approved overwrite, and a separate verification procedure can read the expected final bytes of each selected block. This article calculates detection probabilities; it does not provide or execute a destructive disk command.

Define the population before sampling. For example, divide an explicitly identified readable logical address range into equal-size blocks. State whether a block fails when any byte differs from its expected value. Record how a partial final block is handled. A block count is not automatically a sector count, and a failed-block fraction is not automatically a byte fraction.

The model needs a fixed population, uniform random selection of distinct block indices, and a reliable comparison result. The contents must remain unchanged during checking. A filesystem view, a list of allocated files, or repeated reads of the first few sectors does not supply that sampling design.

Derive the probability of missing residual mismatches

Let:

• N be the total number of blocks in the defined population.
• K be the number of blocks that would fail the comparison if every block were examined.
• n be the number of distinct blocks sampled uniformly without replacement.
• C(a,b) mean the number of ways to choose b objects from a objects.

All counts are dimensionless integers: N >= 1, 0 <= K <= N, and 0 <= n <= N. Of the N blocks, N-K pass. There are C(N,n) possible sample sets and C(N-K,n) sets containing only passing blocks. Therefore:

P(miss | N,K,n) = C(N-K,n) / C(N,n)
P(detect at least one mismatch | N,K,n) = 1 - P(miss | N,K,n)

When n > N-K, missing every failing block is impossible, so the miss probability is zero. When K=0, detection probability is zero because no mismatching blocks exist in this population. Neither boundary says anything about data outside that population.

The same miss probability can be computed by multiplying conditional probabilities: the first draw passes with probability (N-K)/N, the next with (N-K-1)/(N-1), and so on. This is the zero-failure case of sampling without replacement. The draws are not independent.

Worked example: one thousand reads are not a universal guarantee

Consider a hypothetical population of N=1,000,000 equal-size blocks and a sample of n=1,000. Suppose K=1,000 blocks would mismatch: a failing-block fraction of 0.1%.

The exact calculation gives roughly a 63.25% chance of detecting at least one mismatch, leaving about a 36.75% chance of missing all of them. These are conditional probabilities for the stated hypothetical K, not measured properties of a drive.

Now suppose only one block fails. Detection probability becomes exactly n/N = 0.1%; the sample misses it 99.9% of the time. A sample useful for widespread overwrite failure can be weak at finding a small residual region.

The following Python 3.8+ calculation uses only the standard library and does not open a disk. It computes the finite-population probability using logarithms, avoiding enormous factorials:

import math

def detection_probability(N, K, n):
    if not (N >= 1 and 0 <= K <= N and 0 <= n <= N):
        raise ValueError("Invalid block counts")
    if K == 0 or n == 0:
        return 0.0
    if n > N - K:
        return 1.0
    log_miss = math.fsum(
        math.log1p(-K / (N - i)) for i in range(n)
    )
    return -math.expm1(log_miss)

for K in (1000, 1):
    print(K, f"{detection_probability(1_000_000, K, 1000):.6f}")

Expected mathematical output is approximately 1000 0.632488 and 1 0.001000. This example is for moderate counts; it is not a general-purpose arbitrary-precision statistics implementation or a media verification tool.

Choose a sample size from a stated failure threshold

If the sampled fraction is small, the miss probability can be approximated by (1-p)^n, where p=K/N. This approximation treats draws as effectively independent. For a desired miss probability no greater than alpha, solve:

(1-p)^n <= alpha
n >= log(alpha) / log(1-p)

Both logarithms are negative for 0<p<1 and 0<alpha<1, so the resulting sample size is positive. Round up. For p=0.001 and alpha=0.05, this gives n=2995. Check the finite-population calculation for the actual N and chosen minimum integer K.

For a fixed failing set under uniform sampling without replacement, each successive passing draw leaves a larger failing fraction behind. Consequently the exact miss probability is no greater than (1-p)^n; the bound is conservative under these assumptions. It stops being a justified bound when selection is biased or comparisons are unreliable.

This is a detection design for a specified failure threshold. It is not a statement that a passed sample makes the drive “95% erased,” nor is it a probability that no data remains. Avoid turning a conditional test probability into a posterior claim about an unknown drive.

Where the real verification procedure can break the model

Reading only convenient locations gives some blocks no chance of selection. Selecting the same index repeatedly reduces distinct coverage. Sampling only mounted files excludes other logical addresses. A random seed generated after seeing results creates an opportunity for cherry-picking; preserve the selected indices and selection procedure as part of the evidence.

Spatial clustering of bad blocks does not invalidate the exact formula if every individual block is sampled uniformly from the full population. But choosing a few contiguous read windows changes the sampling unit and can make the individual-block formula inappropriate. Define that design separately.

Controller caches, translation layers, remapped sectors, inaccessible areas, and enclosure behavior also constrain what a successful read comparison establishes. Do not reclassify read errors as matching data. Record an unreadable selected block as an exception requiring review.

For SSDs, additional flash locations can exist outside the host-visible address map. This HDD-oriented logical sampling model supplies no proof about those locations and should not be sold as a flash sanitization guarantee.

Verification note: what a useful sample report must contain

No HDD was overwritten or examined for this article. Its numerical examples are reproducible calculations. An operator's report should identify the asset and overwrite operation, block size, population boundaries, sampling algorithm, distinct indices, expected comparison rule, read errors, mismatches, and the acceptance policy used.

A passing sample means only that the sampled comparisons passed under that recorded procedure. Full readback can remove sampling uncertainty for the readable population, but cannot by itself demonstrate coverage of inaccessible physical storage. Accept or reject the sanitization using the complete method evidence and policy, not a percentage printed by a sampling calculator.

When to use this

Use this calculation to review a proposed random-block verification plan with a defined minimum failing-block threshold. Agree on the population and detection target before collecting results.

When not to use this

Do not apply the formula to selected files, repeated first-sector reads, or a few contiguous windows as though they were uniform individual-block samples. Do not use it to claim physical coverage outside the readable population.

Workflow

1. Define the asset, readable population, block size and comparison rule.
2. Choose the minimum failing-block count to detect and acceptable miss probability.
3. Calculate the sample size and preserve the distinct random indices.
4. Execute the approved read comparison, preserving all errors and mismatches.
5. Review the results with the original operation evidence and acceptance policy.

Verification checklist

• Confirm the sample population matches the recorded address range.
• Count distinct sampled indices rather than read attempts.
• Treat short reads and errors as exceptions, not matching blocks.
• State the failure threshold and conditional probability alongside a passing result.

Common mistakes

The most consequential mistake is translating “95% detection at this failure threshold” into “95% probability that no data remains.” Another is calling every sampled block a sector when the sampling unit actually contains many sectors. Preserve both the units and the direction of the statistical claim.

CliDeck workflow fit

Keep the sample design and its result attached to the same evidence record as the overwrite operation. The code here is an independent mathematical example; it is not a CliDeck wiping feature, a disk reader or a certification tool.

FAQ

Does clustering alone invalidate the calculation?

No, provided the fixed population is sampled uniformly at the individual-block level. Sampling contiguous windows changes that design and needs its own analysis.

Does a full logical read prove every physical storage location is clean?

No. It covers what that read interface actually exposes. Inaccessible storage and separate data copies remain separate scope questions.

Related guides

Use the NIST Revision 2 process guide to connect this optional check with method selection, operational verification and the final validation decision.