Merkle Chains for ML Audit Trails

“We trained the model on this data with these hyperparameters.”

That statement is easy to make and difficult to verify. In most ML pipelines, training is a black box. Data goes in, weights come out. If something goes wrong six months later, good luck reconstructing what actually happened.

For safety-critical systems, “trust me” isn’t sufficient. Regulators want evidence. Incident investigators want audit trails. Certification bodies want provable claims.

Merkle chains provide that evidence.

The Core Idea

A Merkle chain is a sequence of cryptographic hashes where each hash depends on the previous one. Change any step in the sequence, and every subsequent hash changes. It’s the same principle that secures blockchain transactions — applied to ML training.

For each training step, we compute:

h_t = SHA256(h_{t-1} || H(θ_t) || H(B_t) || t)

Where:

• h_t is the hash at step t
• h_{t-1} is the previous hash
• H(θ_t) is the hash of the weights after this step
• H(B_t) is the hash of the batch used in this step
• t is the step number

The chain is cryptographically bound. If someone claims “step 5,000 used batch X,” you can verify it. If someone claims “the final weights came from this training run,” you can trace the entire history.

What Gets Committed

In certifiable-training, every training step commits:

Weight State (θ_t) The complete model weights after the update, serialised in canonical form. Canonical means: fixed byte order (little-endian), fixed layout, no padding ambiguity. The same weights always produce the same hash.

Batch Composition (B_t) Which samples were used, in what order. For a batch of indices [42, 17, 891, 3], we hash the indices themselves. Combined with the deterministic Feistel shuffle, this lets you reconstruct exactly which training examples influenced each step.

Step Number (t) Prevents replay attacks. You can’t take step 5,000’s data and claim it was step 3,000.

Previous Hash (h_{t-1}) The chain link. Each hash depends on all previous hashes, creating an immutable history.

The Genesis Block

Every chain needs a starting point. The genesis hash commits the initial state:

h_0 = SHA256(H(θ_0) || H(config) || seed)

This captures:

• Initial weights (random initialisation or pre-trained)
• Training configuration (learning rate, batch size, epochs)
• Random seed (for deterministic reproduction)

From h_0, the entire training run is deterministically specified. Given the same genesis state, any compliant implementation will produce the same sequence of hashes.

Implementation

Here’s the core step function from certifiable-training:

ct_error_t ct_merkle_step(ct_merkle_ctx_t *ctx,
                          const ct_tensor_t *weights,
                          const uint32_t *batch_indices,
                          uint32_t batch_size,
                          ct_training_step_t *step_out,
                          const ct_fault_flags_t *faults)
{
    /* Check for fault invalidation */
    if (ct_has_fault(faults)) {
        ctx->faulted = true;
        return CT_ERR_FAULT;
    }
    
    /* Hash current weights */
    uint8_t weights_hash[CT_HASH_SIZE];
    ct_tensor_hash(weights, weights_hash);
    
    /* Hash batch indices */
    uint8_t batch_hash[CT_HASH_SIZE];
    ct_sha256(batch_indices, batch_size * sizeof(uint32_t), batch_hash);
    
    /* Build preimage: prev_hash || weights_hash || batch_hash || step */
    uint8_t preimage[CT_HASH_SIZE * 3 + 8];
    memcpy(preimage, ctx->current_hash, CT_HASH_SIZE);
    memcpy(preimage + CT_HASH_SIZE, weights_hash, CT_HASH_SIZE);
    memcpy(preimage + CT_HASH_SIZE * 2, batch_hash, CT_HASH_SIZE);
    
    /* Encode step as little-endian 64-bit */
    uint64_t step = ctx->step;
    for (int i = 0; i < 8; i++) {
        preimage[CT_HASH_SIZE * 3 + i] = (uint8_t)(step >> (i * 8));
    }
    
    /* Compute step hash */
    ct_sha256(preimage, sizeof(preimage), ctx->current_hash);
    ctx->step++;
    
    return CT_OK;
}

The key property: this function is deterministic. Same inputs, same hash. No timestamps, no random nonces, no platform-dependent values.

Fault Invalidation

What happens when something goes wrong during training? An overflow, a division by zero, a NaN that would have been?

The chain records it. When a fault flag is set, the Merkle context is marked as “faulted.” Subsequent steps can continue (for debugging), but the chain is cryptographically invalidated.

This prevents a subtle attack: “the training had some numerical issues, but we fixed them and continued.” With fault invalidation, you can’t hide problems. The chain either represents a clean run or it doesn’t.

Verification Without Replay

The Merkle chain enables two levels of verification:

Hash-Only Verification (Fast) Given the chain hashes and the claimed final weights, verify that the chain is internally consistent. This doesn’t prove the training was correct — it proves the claimed history wasn’t tampered with.

Full Replay Verification (Slow) Re-run the entire training from genesis, comparing hashes at each step. If every hash matches, the training is bit-identical to the claimed history. This is expensive but provides the strongest guarantee.

certifiable-verify implements both modes. For most audits, hash-only verification is sufficient. For certification or incident investigation, full replay provides cryptographic proof.

The Checkpoint Problem

Training runs can take days. If the machine crashes at step 50,000, you don’t want to restart from zero.

Checkpoints break the simple chain model — you’re resuming from a saved state, not computing continuously. The solution: checkpoints commit to the chain state at the save point.

typedef struct {
    uint64_t step;
    uint32_t epoch;
    uint8_t merkle_hash[CT_HASH_SIZE];
    uint8_t weights_hash[CT_HASH_SIZE];
    uint8_t config_hash[CT_HASH_SIZE];
    ct_prng_t prng_state;
    uint64_t timestamp;  /* EXCLUDED from commitment */
    uint32_t version;
    ct_fault_flags_t fault_flags;
} ct_checkpoint_t;

Note that timestamp is excluded from the cryptographic commitment. Timestamps are useful for humans but shouldn’t affect determinism — a checkpoint saved at 3pm should be identical to one saved at 3am.

What This Enables

Incident Investigation When a deployed model misbehaves, you can trace back: which training run produced this model? What data was used? Were there any numerical faults?

Regulatory Compliance For DO-178C (aerospace), IEC 62304 (medical devices), ISO 26262 (automotive), auditors want evidence that the development process was controlled. A Merkle chain is that evidence — cryptographically signed, tamper-evident, independently verifiable.

Reproducibility Claims “Our results are reproducible” is a strong claim. With a Merkle chain, it’s a provable claim. Anyone with the genesis state and the chain hashes can verify that your training run is reproducible.

Model Provenance In a world of fine-tuned models and transfer learning, “where did this model come from?” is increasingly important. The chain provides complete provenance: initial weights → training data → final weights.

The Cost

Merkle chains aren’t free:

• Storage: Each step adds ~100 bytes (hashes + step record)
• Compute: SHA-256 of weights at each step (can be expensive for large models)
• Complexity: More code, more tests, more things that can go wrong

For a 10,000-step training run, you’re looking at ~1MB of chain data. For a million-step run, ~100MB. That’s negligible compared to the model weights themselves.

The hash computation is more significant. For a 1-million-parameter model (4MB in Q16.16), hashing takes roughly 10ms. For a 1-billion-parameter model, it’s 10 seconds per step. That’s meaningful overhead.

The mitigation: hash only at checkpoint boundaries for large models, with the option for full-step hashing when audit requirements demand it.

Conclusion

Merkle chains transform “we trained this model” from a claim into evidence. Every step is committed. Every batch is recorded. Every weight update is cryptographically bound to its history.

For systems that matter — medical devices, autonomous vehicles, aerospace — this audit trail isn’t overhead. It’s a requirement. When something goes wrong, you need to know what happened. When regulators ask questions, you need provable answers.

The certifiable-* ecosystem builds this in from the start. Not as an afterthought, not as a logging feature, but as a fundamental architectural property.

As with any architectural approach, suitability depends on system requirements, risk classification, and regulatory context. For systems that must be auditable, Merkle chains provide the foundation.

Explore the implementation in certifiable-training or see certifiable-verify for the verification tooling.