MDL counts the model's length because a fitted model has to be sent

A probabilistic model HH is a code, and the data's length under it, L(D∣H)L(D \mid H), is its cross-entropy under HH. If sender and receiver agreed on the code in advance, as a phone line or two copies of the same pretrained language model do, that is the whole cost. Minimum description length (MDL) scores a model by the total a sender needs when they didn't.

Model selection is that case. The sender picks HH after seeing the data, so the receiver can't have agreed to it in advance, and a code fitted after the fact can absorb the data. Take ten points and the degree-9 polynomial through all of them: it predicts every point exactly, so L(D∣H)L(D \mid H) is zero, but only because its ten coefficients are the ten values re-expressed in another basis.

So the sender has to transmit the fitted code first [1], and the message costs

L(H)+L(D∣H)L(H) + L(D \mid H)

bits. The first term is the price of what the model learned from the data, not a complexity penalty anyone chose: a code agreed in advance would cost nothing, and a fitted one costs as much as it took from the data.

Three polynomial fits to the same ten pointsA straight line misses the curve and leaves large residuals. A cubic follows the pattern and leaves small residuals. A degree-9 polynomial passes through every point and swings wildly between them.straight line2 numbers, large residualscubic4 numbers, noise-sized residualsdegree 910 numbers, no residuals
The same ten points, three models. Thin lines are residuals: what the model still has to write down. The line is cheap to state and leaves a lot; the degree-9 polynomial leaves nothing but needs a number per point.

The degree-9 polynomial took everything, so its coefficients cost about as much as sending the raw data. A model that predicts nothing takes nothing and leaves every value to the second term, which also costs the raw data. The only way to get below that is structure: a regularity sent once in HH and used at every point. The straight line sends too little, and its residuals still trace the curve. The cubic sends four coefficients and leaves residuals the size of the noise, and the two parts together are shorter than the data. Give MDL data with no structure and nothing gets shorter, so it finds no pattern.

This is what Grünwald calls crude MDL: how precisely to send HH is left to you, and the message commits to a single hypothesis. That commitment wastes bits, which is why refined MDL agrees on one code for the whole model class in advance.

References

  1. A tutorial introduction to the minimum description length principle [link]
    Grunwald, P., 2004. arXiv. DOI: 10.48550/ARXIV.MATH/0406077