Minimum Description Length

The conceptual ground for entropy was prepared by Sadi Carnot's 1824 memoir Réflexions sur la puissance motrice du feu. Carnot showed that no heat engine could exceed a universal efficiency limit determined solely by its operating temperatures — implying something fundamental about the direction of physical processes. 1

This was not yet entropy, but it demanded the concept. Why cannot all heat be converted to work? The answer — something is always irretrievably lost — would become the second law, and entropy would name that loss.

Carnot died of cholera in 1832 at age 36, his 1824 memoir nearly forgotten. Émile Clapeyron’s 1834 paper “Mémoire sur la puissance motrice de la chaleur” rescued it — reformulating Carnot’s arguments using calculus and the pressure-volume indicator diagram, giving them the mathematical form the scientific community could absorb.

Without Clapeyron, Clausius and Lord Kelvin would almost certainly never have encountered Carnot’s theorem. Clausius explicitly credited Clapeyron’s memoir as the route by which he came to the work. The chain Carnot → Clapeyron → Clausius → Boltzmann → Gibbs → Shannon is unbroken — and Clapeyron is its indispensable link. 50

Rudolf Clausius is the architect of entropy as a named physical quantity. His 1850 paper displaced caloric theory, establishing that heat flows from hot to cold and cannot spontaneously reverse. 2 Over the following decade he developed the "equivalence value" of transformations — his precursor to entropy — and formalized the inequality:

In 1865 he named this quantity entropy (Greek: transformation content), deliberately echoing energy: "so closely related in physical significance that a certain similarity in their names appears to be appropriate." 3 His conclusion: "The energy of the universe is constant. The entropy of the universe tends to a maximum." This is the first clean articulation of irreversible directional change — a concept that reappears, centuries later, as the cost of model description. 2

In a private letter to Peter Guthrie Tait, Maxwell conceived an imaginary demon capable of sorting fast and slow molecules without apparent energy expenditure — seemingly violating the second law. 4 Unresolved for sixty years, the demon posed a profound question: what is the relationship between knowledge — the information the demon holds — and entropy? This question became central to MDL's intellectual foundations.

Boltzmann made the decisive leap from thermodynamic entropy to probability. His 1872 H-theorem established that a gas's approach to equilibrium is fundamentally probabilistic. He wrote: "the problems of the mechanical theory of heat are really problems in probability calculus." 5

The synthesis came in 1877. Boltzmann saw that entropy increases because uniform molecular distributions vastly outnumber non-uniform ones — a purely combinatorial insight. The formula engraved on his Vienna tombstone: 6

The logarithmic form — probability mapped to additive scalars through the logarithm — is structurally identical to what Shannon would later call entropy. Shannon acknowledged this connection when naming his measure after Boltzmann's H-function. The thermodynamic ancestry of MDL is embedded in this equation. 7

Gibbs extended Boltzmann's work into a general statistical mechanics. His 1875–1878 memoir used entropy maximization as the condition of thermodynamic equilibrium. 8 His 1902 textbook coined "statistical mechanics" and defined entropy for an arbitrary probability distribution:

This is the direct algebraic precursor to Shannon's information entropy. The chain Clausius → Boltzmann → Gibbs → Shannon was not accidental: it was a continuous mathematization of the insight that disorder and missing information are the same thing. 9

Szilard resolved Maxwell's demon with a landmark 1929 paper. He showed the demon must acquire knowledge of each molecule's state, and that this information acquisition carries a thermodynamic cost precisely sufficient to compensate the entropy reduction achieved. 10 Szilard established that information is physical — knowing a system's state is energetically equivalent to reducing its entropy. Measurement and knowledge are subject to thermodynamic accounting. 11

Turing's "On Computable Numbers" constructed a universal Turing machine capable, in principle, of any computable process. 12 The deeper implication: any computable object can be described by a finite program, and the length of that program is its minimum description. Turing planted the foundational intuition that complexity equals the length of the shortest description — the core of MDL, formalized three decades later. 13

Shannon's 1948 paper is the single most consequential publication in MDL's genealogy. He defined the entropy of a discrete distribution: 14

…and proved this is the minimum average bits needed to encode messages — a fundamental lower bound on lossless compression. Shannon named his measure after Boltzmann's H-function. Warren Weaver noted it "roots back to Boltzmann's observation that entropy is related to 'missing information.'" 15

Shannon's source coding theorem is the direct ancestor of MDL: any statistical regularity permits compression below raw bit count, and the degree of compression measures how much regularity a model has captured. 16

Landauer proved that erasing one bit of information dissipates at least k_BT ln 2 joules — closing the loop on Maxwell's demon, whose memory erasure pays the entropy price. 17

Description length and thermodynamic work are in precise quantitative correspondence. Any description that stores a bit of information about the world is implicitly performing work. This is not metaphor — it is physical equivalence. 18

"Plurality must never be posited without necessity." This principle — prefer the simpler of two equally explanatory hypotheses — has guided scientific model-building for seven centuries. 19 The modern MDL principle is, in the precise technical sense, the computational and information-theoretic formalization of Ockham's razor: shorter descriptions receive exponentially higher probability under the universal prior. 20

Solomonoff is the unheralded originator of the computational branch of MDL. His 1964 paper "A Formal Theory of Inductive Inference" constructed a rigorous framework: the a priori probability of any sequence is proportional to the sum of probabilities of all programs on a universal Turing machine that generate it. 21

Shorter programs receive exponentially higher probability — a mathematically perfect realization of Occam's razor with provably optimal convergence for any computable data-generating process. 22 Rissanen explicitly acknowledged Solomonoff's influence when developing MDL. 23

Kolmogorov defined the complexity K(x) of a string x as the length of the shortest program on a universal Turing machine that outputs x. 24 The Invariance Theorem guarantees that the choice of programming language affects K(x) by at most a constant. This "algorithmic entropy" is the theoretically ideal, objective measure of information in any individual object.

Kolmogorov complexity is the theoretical ideal toward which MDL strives. Because it is uncomputable, practical MDL methods are computable approximations that trade theoretical perfection for tractability. 25

Chaitin independently discovered algorithmic complexity in 1966 and extended it in 1969 to the prefix-free (self-delimiting) variant, more directly connected to probability theory through the Kraft inequality — the foundation for NML and the universal codes used in practical MDL. 26 Chaitin also demonstrated deep links between algorithmic randomness and Gödel incompleteness. 27

Chris Wallace and David Boulton published "An Information Measure for Classification," introducing Minimum Message Length (MML): the best model minimizes the combined bit-length of the hypothesis and the data encoded under that hypothesis. 28 MML is a parallel invention to Rissanen's later MDL, differing primarily in its explicit Bayesian framing and treatment of continuous parameters. 29

Akaike derived AIC from the relationship between maximum likelihood estimation and Kullback-Leibler divergence, penalizing complexity by the number of free parameters. 30 AIC, BIC (Schwarz, 1978), and MDL share philosophical ancestry in Occam's razor and Shannon's theory — their near-simultaneous emergence reflects the same underlying insight pressing through different disciplines. 31

The term Minimum Description Length was coined by Jorma Rissanen in "Modeling by the Shortest Data Description," published in Automatica. The central thesis: all statistical learning is about finding regularities in data, and the best hypothesis is the one that compresses the data most effectively. 32

His original two-part code — describe the model, then encode data given the model, minimize total length — is closely related to Schwarz's BIC (also 1978), though Rissanen's motivation came entirely from information theory and universal coding, not Bayesian inference. 33

Rissanen extended MDL with stochastic complexity — the shortest description length achievable by any universal code for a parametric model class, moving beyond the two-part code. 34 He derived the celebrated parametric penalty:

where k is the number of parameters and n the sample size. This connects MDL to the Fisher information matrix and establishes a precise relationship between model complexity and sample-size requirements. 35

Geoffrey Hinton and Drew van Camp brought MDL into neural network theory in "Keeping Neural Networks Simple by Minimizing the Description Length of the Weights." 36 Their argument: networks generalize well when the information in the weights is less than the information in the training outputs. They proposed controlling weight information by adding Gaussian noise — the first direct application of MDL to deep learning, prefiguring Bayesian deep learning and the information bottleneck framework. 37

Rissanen introduced Normalized Maximum Likelihood (NML) — what he considered the purest form of MDL. For each data sequence, NML selects the model under which it has maximum probability, normalized across all possible sequences. 38 The stochastic complexity under NML has a clean minimax interpretation: it achieves the smallest achievable regret over all data-generating distributions in the model class. 39

"The Minimum Description Length Principle in Coding and Modeling" synthesized two decades of research, showing that NML, mixture coding, and predictive coding all achieve stochastic complexity to within asymptotically vanishing terms. 40 This remains the canonical reference for MDL's relationship to statistical modeling theory. 41

Tishby et al. framed representation learning as a rate-distortion problem: find the most compressed representation of X that maximally preserves information about target Y. 42 A close cousin of MDL — both operationalize compression as a criterion for structure extraction — the Information Bottleneck gained renewed prominence in 2017 when Tishby argued that deep networks implicitly solve an IB objective layer by layer.

Grünwald's 736-page MIT Press textbook The Minimum Description Length Principle became the definitive reference, covering universal coding, stochastic complexity, NML, Bayesian model selection, and MDL for regression, density estimation, and non-parametric problems. 43 MDL was established as a mature, unified theory of inductive inference, with several previously open theoretical questions about consistency and convergence resolved. 44

Arora et al.'s 2018 NeurIPS paper showed that compressing trained network weights leads to provable generalization bounds: the networks that generalize best are the ones most compactly described. 45 Ollivier's 2018 NeurIPS paper demonstrated that MDL-inspired prequential coding yields tighter compression bounds than standard variational inference — an empirical vindication of the algorithmic information theory tradition. 46

Alemi et al. introduced the Deep VIB, parameterizing the information bottleneck objective via a neural network and the reparameterization trick. 47 Models trained with the VIB objective outperformed standard regularization on generalization and adversarial robustness. Deep VIB brought the MDL-compression tradition and the IB-representation tradition into explicit contact within modern deep learning architectures.

Grünwald and Roos published "Minimum Description Length Revisited" in 2020, surveying three decades of development and identifying frontier directions: MDL for discrete data, finite-sample NML behavior, and PAC-Bayes generalization bounds. 48 Grünwald's 2025 JASA paper "Learning with the Minimum Description Length Principle" catalogued new developments including connections to deep learning theory, conformal prediction, and safe anytime-valid inference. 49 The central question — identifying models that capture genuine structure rather than overfitting noise — remains as urgent as it was in 1978.