Skip to main content

Do you see the Matrix? Derivation of Linear chi^2 minimization

This blog post is primarily for my Ay117 students. However, if you've ever wondered where chi-squared minimization comes from, here's my derivation.

Figure 1: Either a scene from The Matrix or the hallway in your astronomy building.
Yesterday in class we reviewed the concept of "chi-squared minimization," starting with Bayes' Theorem
$P(\{a\} | {d}) \propto P(\{d\} | {a}) P(\{a\})$
In other words, if we wish to assess the probability of a hypothesis that is expressed in terms of the parameters $\{a\}$ conditioned on our data $\{d\}$, we first calculate how likely we were to get our data under the hypothesis (first term on the right), and multiply this "likelihood" by our prior notion that a given set of parameters is representative of the truth.

Supposing that we have data that are independent from one another, and normally distributed, then our likelihood term can be written
$P(\{d\} | {a}) = \prod_i \frac{1}{\sqrt{2\pi \sigma^2}} \exp{\left[\frac{1}{2}\left( \frac{y_i - f(x_i)}{\sigma_i}\right)^2\right]}$
As for the priors, we'll make the fast and loose assumption that they are constant ($P(a_0) = P(a_1) = ... = {\rm const}$). It is computationally advantageous to compute the log-likelihood
$l = \ln{P(\{d\} | {a})} = C - \frac{1}{2} \sum_{i=0}^{N-1} \left[ \frac{y_i - f(x_i)}{\sigma_i}\right]^2 = C - \frac{1}{2} \chi^2$
Since our goal is to find the parameters that maximize the likelihood, this is equivalent to maximizing the log-likelihood, which is in turn equivalent to minimizing that $\chi^2$ thingy.

For the specific problem of fitting an Mth-degree polynomial, $f(x_i) = \sum_{j=0}^{M-1}a_j x_i^j$, and this results in a linear system of equations that can be solved for the best-fitting parameters.

In class, I got my notation all scrambled, and I neglected the measurement uncertainties $\sigma_i$. My bad! Here's what should have appeared on the board (worked out this morning over breakfast, so be sure to check my work!).


To be clear, the "weights" are $w_i = 1/\sigma_i^2$. Zooming in on the key part:


The first problem of the next Class Activity will be to write a function that takes abscissa and ordinate values, and the associated uncertainties, and computes the best-fitting coefficients for a polynomial of arbitrary dimension $M$. 

See also David Hogg's excellent line-fitting tutorial.

Comments

Popular posts from this blog

A view from your shut down

The Daily Dish has been posting reader emails reporting on their " view from the shutdown ." If you think this doesn't affect you, or if you know all too well how bad this is, take a look at the growing collection of poignant stories. No one is in this alone except for the nutjobs in the House. I decided to email Andrew with my own view. I plan to send a similar letter to my congressperson. Dear Andrew, I am a professor of astronomy at the Harvard-Smithsonian Center for Astrophysics (CfA). The CfA houses one of the largest, if not the largest collection of PhD astronomers in the United States, with over 300 professional astronomers and roughly 100 doctoral and predoctoral students on a small campus a few blocks west of Harvard Yard. Under the umbrella of the CfA are about 20 Harvard astronomy professors, and 50 tenure-track Smithsonian researchers. A large fraction of the latter are civil servants currently on furlough and unable to come to work. In total, 147 FTEs

The Long Con

Hiding in Plain Sight ESPN has a series of sports documentaries called 30 For 30. One of my favorites is called Broke  which is about how professional athletes often make tens of millions of dollars in their careers yet retire with nothing. One of the major "leaks" turns out to be con artists, who lure athletes into elaborate real estate schemes or business ventures. This naturally raises the question: In a tightly-knit social structure that is a sports team, how can con artists operate so effectively and extensively? The answer is quite simple: very few people taken in by con artists ever tell anyone what happened. Thus, con artists can operate out in the open with little fear of consequences because they are shielded by the collective silence of their victims. I can empathize with this. I've lost money in two different con schemes. One was when I was in college, and I received a phone call that I had won an all-expenses-paid trip to the Bahamas. All I needed to d

back-talk begins

me: "owen, come here. it's time to get a new diaper" him, sprinting down the hall with no pants on: "forget about it!" he's quoting benny the rabbit, a short-lived sesame street character who happens to be in his favorite "count with me" video. i'm turning my head, trying not to let him see me laugh, because his use and tone with the phrase are so spot-on.