So Andy and I worked through this offline, and we now get the same result through different means, so I'm pretty confident this is correct.
The first step is to get the best numbers possible for the masses of Pluto and Charon and for the distance between them. It turns out there's some recent work using the orbits of Nix and Hydra to do exactly that:
http://iopscience.iop.org/1538-3881/132/1/...1_132_1_290.pdfPluto: 1.28726E+22 kg
Charon: 1.69741E+21 kg
center-to-center Distance: 1.9571E+07 m
My results are:
Charon-center to L1: 5.9716E+06 m
Charon-center to L2: 7.4611E+06 m
Andy's numbers differ from mine by just a few hundred meters now. In this case, his math and Excel were correct from the start; I was the one with the math error.
In case anyone's interested, here's how to compute it: let m be the mass of the smaller body divided by the mass of the larger one and let h be the distance from the center of the smaller body to the L1 point as a fraction of the distance from the smaller body to to larger one. Then
(1+m)*h^5 - (3+2*m)*h^4 + (3+m)*h^3 - m*h^2 + 2*m*h - m = 0
There are a variety of ways to find h given m. (I used Newton's method, but you can brute force it too.)
For L2, the only change is that the quartic and linear terms change sign, like so:
(1+m)*h^5 + (3+2*m)*h^4 + (3+m)*h^3 - m*h^2 - 2*m*h - m = 0
I hunted and hunted to find something that laid it out like this, but everything seemed focused on the harder problem of finding L4 and L5 and proving their stability.
--Greg