Least squares in calibration: weights, nonlinearity, and other nuisances

In the least-squares fitting of data, there is a unique answer to the question of how the data should be weighted: inversely as their variance. Any other weighting gives less than optimal efficiency and leads to unreliable estimates of the parameter uncertainties. In calibration, knowledge of the data variance permits exact prediction of the precision of calibration, empowering the analyst to critically examine different response functions and different data structures. These points are illustrated here with a nonlinear response function that exhibits a type of saturation curvature at large signal like that observed in a number of detection methods. Exact error propagation is used to compute the uncertainty in the fitted response function and to treat common data transformations designed to reduce or eliminate the effects of data heteroscedasticity. Data variance functions can be estimated with adequate reliability from remarkably small data sets, illustrated here with three replicates at each of seven calibration values. As a quantitative goodness-of-fit indicator, chi(2) is better than the widely used R(2); in one application it shows clearly that the dominant source of uncertainty is not the measurement but the preparation of the calibration samples, forcing the conclusion that the calibration regression should be reversed.