A hundred templates

RMNIST/10 gives ten handwritten examples of each digit. Fitting all hundred and classifying by mean energy per class is the same experiment as before with twice the templates — but getting there first required fixing something about how the templates were fitted, and that turned out to be the more interesting half.

100 templates · 29 parameters each · 48 restarts per training digit · 200 unseen test digits, the same ones throughout · from classify_n.py.

1Restarts that were being thrown away

The fifty templates of the last study included a few poor traces — five of fifty below an IoU of 0.80, the worst at 0.65. The obvious remedy is more restarts. It does not work:

The restarts were never the problem. They were being spent and then discarded, because the restart kept was the one with the lowest squared error, and squared error is not what a template is wanted for:

This is the optimiser's curse in miniature. Generating more candidates and selecting on a proxy reliably finds a candidate that is excellent by the proxy and merely ordinary by the thing you meant — and the better the search gets, the more of its gain lands on the proxy alone. The fix is not more search; it is selecting on the quantity that matters. Squared error still does the optimising — it has to, it is the differentiable one — but overlap does the choosing, and it is free, since every candidate has already been rendered.

2All one hundred, fitted

3Choosing the anchor

As before, one number is fitted on a disjoint tuning sample: how hard to charge for deformation during the fit.

4Results

Each test digit is fitted by all 100 templates; each fit yields a total energy — the mean data loss over the whole run — and the hundred energies are collapsed to ten.

5One, five, ten

The same two hundred test digits and the same anchor at every point, so these are directly comparable.

So the verdict of the last study does not survive contact with more data. With five examples per class the mean was beaten decisively, and the reason given — that a class is several distinct styles and averaging dilutes a good match with mismatched ones — is still true. What was wrong was treating that as a defect of the rule rather than of the sample. Five is simply too few to estimate a class mean when the class is multi-modal; ten is enough for it to start working.

The selective rules go the other way. Neither the best of a class nor the mean of its best two improves significantly from five examples to ten, and the plain minimum is flat. That is the same selection effect as section 1, now inside the classifier: a minimum taken over more candidates drifts down for every class, so the extra examples buy resolution for all ten at once and the ordering barely moves.

6The spread that caused it

The ten templates of a class are still not noisy copies of one thing — they are distinct styles that happen to share a label, and the within-class spread is still comparable to the gap between classes. Averaging still asks whether 8s in general are a cheap start, when the sharper question is whether any 8 is. The difference is that with ten samples the vaguer question is answered precisely enough to be nearly as useful.

7Per class

And where the mean rule's errors go:

8The hundred energies behind one decision

9What I would take from this

• More restarts did not fix the bad fits; selecting differently did. From 6 to 96 restarts, mean template IoU is flat while the restarts are kept by lowest loss. Keeping them by overlap instead turns the same computation into a steady improvement and removes almost all the poor traces.
• The mechanism is a fixed percentile against a moving tail. The lowest-loss fit sits at roughly the same quarter of the overlap distribution whether there are six candidates or ninety-six; what changes is that the top of that distribution keeps rising and a fixed percentile never reaches it. Search amplifies whatever mismatch exists between the objective and the goal.
• The mean was not wrong, it was under-sampled. At five examples per class it lost decisively to the selective rules; at ten the difference is no longer significant. It is also the only rule that improves significantly from five to ten. The last study's conclusion was right about the data it had and wrong as a statement about the rule.
• The selective rules saturate, for the same reason section 1 does. A minimum over more candidates drifts down for every class at once, so extra examples sharpen all ten equally and move the ordering very little. Selection effects cut both ways: they wasted the restarts and they cap the best-of-class rule.
• Better templates helped the selective rules and barely moved the mean. Fixing the selection lifted the best-of-class rule from 0.925 to 0.950 on RMNIST/5 and the mean from 0.850 to 0.855 — averaging washes out template quality along with everything else.
• A hundred traced curves label four fifths to nine tenths of unseen handwriting. No learning across classes, no discriminative training, a hand-built generative model and 29 numbers per example — against 54% for the same hundred templates compared without any fitting.

---

Reproduce with python classify_n.py 10 then python build_page6.py. The restart sweep is python classify_n.py 5 restarts. The model is described in the digits.html write-up and the scoring in classify.html.