CNN confidence under corruption.

What was compared

Six CIFAR-adapted ResNet-18 models trained from random initialization for 100 epochs each: baseline cross-entropy and mixup with alpha 0.2, paired across seeds 42, 43, and 44. Architecture, optimization, augmentation, and data partitions were held fixed. Checkpoints were selected by minimum validation NLL.

A fixed stratified split reserved 40,000 CIFAR-10 images for training, 5,000 for validation, and 5,000 for fitting temperature. Each checkpoint was evaluated on all 10,000 official test images and the 15 standard CIFAR-10-C corruption types at five severities. No temperature was fitted on test or corrupted data.

The primary paired comparison

Mixup minus baseline, after averaging the 75 corruption conditions within each run. Negative NLL differences favor mixup; positive accuracy differences favor mixup. Values are means ± sample standard deviations across three paired seed differences.

Mean clean accuracy rose from 94.19% to 94.62%; mean corruption accuracy rose from 73.64% to 75.88%. Raw corruption NLL fell from 1.274 to 0.853. The direction of the primary NLL contrast favored mixup in all three seeds, both before and after scaling.

Calibration did not transfer uniformly

Temperature scaling reduced clean-test NLL for both methods. Under corruption, baseline NLL improved from 1.274 to 0.967, while mixup NLL increased from 0.853 to 0.907. This opposite direction occurred in all three seeds. For mixup, mean corruption Brier score increased from 0.3593 to 0.3701 and ECE15 from 0.1019 to 0.1219.

Positive scalar temperature leaves argmax predictions and accuracy unchanged. The fitted temperatures were above 1 for baseline and below 1 for mixup. They soften baseline probabilities and sharpen mixup probabilities; the measured effects depend on the test condition and metric.

NLL is mean negative natural log probability of the true class. Brier is the mean sum of squared probability errors over all ten classes. ECE15 uses 15 equal-width confidence bins. Lower values are better for these three metrics; each measures a different aspect of the predictions.

Original analysis figures

What the results support

• Three seeds describe this experiment. Sample SD shows between-seed variability. It is not a confidence interval, and these runs do not establish statistical significance or equivalence.
• Corrupted images share their source cases. The 75 versions of an image are correlated. The summaries weight each condition equally within each run; they do not treat 750,000 views as independent test cases. This mean is not normalized mCE.
• Metric gains are not universal. After scaling, mean corruption ECE15 is 0.1218 for baseline and 0.1219 for mixup, despite mixup's lower NLL. ECE depends on the bin definition and is calculated per condition before averaging.
• Severe corruptions still cause substantial errors. Averaged over corruption types, severity-5 accuracy is 55.79% for baseline and 60.02% for mixup. These results concern CIFAR-10-C and this training configuration; they do not establish adversarial robustness, OOD detection, or general deployment performance.
• Test results are the final evaluation. Choosing a new temperature or changing training based on these observations would require a separately reported follow-up study.

Run evidence

This page is derived from the completed study's comparison.json, comparison.csv, and original generated figures. The implementation preserves per-condition predictions, IDs, checkpoint and source hashes, calibration records, and evaluation identities. The public repository provides the code and frozen protocol. The full technical report preserves the measured tables and provenance; the results discussion explains the findings and limitations.