The Chernoff bound is of critical importance in computer science and probability. When I need it, though, I almost always have to look it up to make sure I get all the parameters correct. Not any more, though. I’ll explain how the Chernoff bound is essentially the PDF of the “Gaussian you would expect” from the Central Limit Theorem.
The most common form of the Chernoff bound is the following: suppose you have independent and identically distributed coin flips , say the result of repeatedly flipping a coin that comes up Heads with probability . The number of Heads is by definition a Binomial distribution; let denote this random variable. Then with all but exponentially small probability, is within of its mean:
The Central Limit Theorem (CLT) tells us that as , the distribution approaches a standard normal distribution. Philosophically, approaches a normal distribution with the parameters you would expect: mean and variance (though the “scaled up” distributions may not necessarily converge from the CLT alone).
In our case, and . The PDF of the “Gaussian you would expect” is
for an appropriate normalizing constant . Let’s rewrite that with :
This is (up to the factor of ) the expression appearing in the Chernoff bound! Thus we can think of the Chernoff bound as expressing an “even when is small” version of the CLT, with a little bit of loss from to in the multiplicative factor.
Using the CDF instead of the PDF
We lost some credibility at one point in the technique above: we actually should have been looking at the cumulative probability
Instead we noticed that the Chernoff bound can be remembered by looking at the PDF (and ignoring a nonconstant factor)
But using the CDF instead of the PDF actually gives the same expression, and with a truly constant factor. Let’s compute:
If we break up the integral into little size pieces from to , the integral on a piece looks like
The exponent increases faster than linearly, and it is the only thing that changes in , so this contribution to the integral goes to 0 faster than geometrically. Therefore the integral is bounded by an infinite geometric series with initial term proportional to . Therefore the cumulative probability is (up to a constant) the expression in the Chernoff bound
and this time the constant is actually constant!