Unlearning Is Not Deletion

The gap

A checkpoint can forget while the deployed ecosystem still remembers

Machine unlearning is certified almost entirely on the updated model \(M_U\): low forget accuracy, a weak membership signal, closeness to a retrained model. A deployed vision system releases far more than a checkpoint.

After training on \(D = D_r \cup D_f\), a deployment can retain or publish:

• Eembedding banks
• Pclass and identity prototypes
• GkNN and retrieval indices
• Ccached feature stores
• Kquery access to the updated model

Editing \(M \rightarrow M_U\) never revisits an artifact that was stored independently of the model. So the privacy question is not "did the model forget?" but "can the forget set still be recovered from what was released?"

The metric

Artifact Residual Risk

For an artifact \(a\) and a recoverability attack \(R(D_f;a) \in [0,1]\), ARR normalises what the shipped artifact leaks between two anchors: what the original model's artifact leaks, and what a retrained model's artifact leaks.

Five channels are audited: E embedding-bank membership, P nearest-prototype recovery, G retrieval mAP on forgotten classes, C linear-probe recovery from cached features, and K loss-based API membership. We report the worst channel, \(\mathrm{ARR}_{\max}(U) = \max_{a \in \mathcal{H}} \mathrm{ARR}_a(U)\), because an adversary uses the single best-leaking channel, not a polite average.

Main result

Re-derived artifacts still leak

Frozen artifacts are ARR = 1 for every method by construction. The interesting regime is re-derived: rebuild the artifact from \(M_U\) and ask whether the representation itself actually forgot. Over 3 seeds × 3 forget classes on CIFAR-100:

Checkpoint metrics do not rank artifact safety

Among methods that pass the checkpoint forget test, output-level behaviour is already saturated while artifact risk spans most of the scale. Checkpoint forgetting explains only R² = 0.22 of the variance in mean re-derived ARR (r = −0.46 over n = 58 valid configurations); worst-channel rank agreement is only Kendall τ = 0.36.

Three methods with identical checkpoint forgetting — exactly 0.000 — and very different worst-channel ARR.

Evidence

"Forgotten" embeddings still reconstruct

On CIFAR-100 class 0, RandLabel drives checkpoint forget accuracy from 0.89 to 0.01. A decoder trained only on retain data still recovers class-defining content from the unlearned embeddings.

The stronger case

Open-set retrieval has no deletion floor

Market-1501 identity unlearning. Forget 20 training identities, then compare the original encoder against a from-scratch retrain on the remaining ones. The two anchors collapse onto each other:

on prototype recovery, retrieval mAP and linear-probe recovery alike.

A frozen CLIP index is stronger still. Over a CLIP ViT-B/32 index of CIFAR-100 the "forgotten" class stays recoverable at \((R_P, R_G, R_C) = (0.79,\, 0.44,\, 0.92)\), and there is no "retrain CLIP without \(D_f\)" to define a floor at all — so no model-side operation moves these numbers. Pruning the entries drives all stored-artifact channels to 0.

Closing the gap

What actually works

1. Re-derive and prune

Rebuild every artifact from \(M_U\), then remove the forgotten entries. In the controlled CIFAR setting raw recoverability \((R_P, R_G, R_C) = (0.89,\, 0.91,\, 1.00)\) falls to \((0,\,0,\,0)\). Note this zeroes the stored channels P, G and C only — pruning an index does nothing for the API channel K.

2. If you cannot enumerate every artifact, prefer feature-damaging unlearning

SSD and FT are the only methods in this study that hold the strict worst-channel risk low on every audited channel.

3. Relocating features is not enough

Four relocation and feature-matching variants, including an oracle pointed at the retrain encoder, still leave re-derived ARR at 0.76–0.81. Destruction generalises to held-out forgotten samples; relocation does not.

Limitations. Billion-entry production indices are untested; Market-1501 uses a single ReID encoder; channel E is ill-conditioned for well-generalising classifiers; and no single method is artifact-safe everywhere while preserving utility.

Code

Reproduce it

The ARR audit core is pure NumPy and torch-free by design, so the metric and its tests install and run on any interpreter. Only training and the attacks need torch.

git clone https://github.com/tirth5828/unlearning-is-not-deletion
cd unlearning-is-not-deletion

pip install -e .
pytest                    # ARR core, metrics, forget-sets, artifacts, ReID metrics

Both paper tables and the decoupling figure regenerate straight from the shipped per-configuration CSVs, with no training and no GPU:

python paper_tables/make_agg_table.py results/cifar100_multi/per_config.csv
python paper_tables/make_tiny_table.py results/tinyimagenet/per_config.csv tab_tiny.tex
python experiments/plot_decoupling.py results/cifar100_multi/per_config.csv

The end-to-end runners, the validity gate, the Market-1501 retrieval protocol and the per-file column reference are documented in REPRODUCING.md. Trained checkpoints (~1 GB) and the raw datasets are not in the repo; CIFAR-100, Tiny-ImageNet, Market-1501 and CLIP ViT-B/32 are all public.

Poster

U&ME @ ECCV 2026

1400 × 1000 mm, print-ready CMYK. The LaTeX source and its print checks — page geometry, crop marks, embedded and subsetted fonts, trim clearance — are in poster/.

Citation

BibTeX

@inproceedings{joshi2026unlearning,
  title     = {Unlearning Is Not Deletion: Auditing Residual Information in
               Released Vision Artifacts},
  author    = {Joshi, Tirth and Wang, Honggang},
  booktitle = {Proceedings of the 3rd Workshop on Unlearning and Model Editing
               (U\&ME), European Conference on Computer Vision (ECCV)},
  year      = {2026},
}