## The Model Complexity Myth

An oft-repeated rule of thumb in any sort of statistical model fitting is "you can't fit a model with more parameters than data points".
This idea appears to be as wide-spread as it is incorrect.
On the contrary, if you construct your models carefully, **you can fit models with more parameters than datapoints**, and this is much more than mere trivia with which you can impress the nerdiest of your friends: as I will show here, this fact can prove to be very useful in real-world scientific applications.

A model with more parameters than datapoints is known as an *under-determined system*, and it's a common misperception that such a model cannot be solved in any circumstance.
In this post I will consider this misconception, which I like to call the "model complexity myth".
I'll start by showing where this model complexity myth holds true, first from from an intuitive point of view, and then from a more mathematically-heavy point of view.
I'll build from this mathematical treatment and discuss how underdetermined models may be addressed from a frequentist standpoint, and then from a Bayesian standpoint.
(If you're unclear about the general differences between frequentist and Bayesian approaches, I might suggest reading my posts on the subject).
Finally, I'll discuss some practical examples of where such an underdetermined model can be useful, and demonstrate one of these examples: quantitatively accounting for measurement biases in scientific data.