Uncertainty, Sensitivity, Convergence, and Rounding in Performing and Reporting Least-Squares Fits

This paper describes a procedure for optimal rounding of parameters determined from a linear or nonlinear least-squres fit in order to minimize the number of digits which must be quoted while ensuring that the resulting rounded constants can predict the input data with no significant loss of precision. Related problems concerning nonlinear least-squares convergence and taking account of model dependence of fitted or predicted parameters are also addressed. The recommended rounding procedure is illustrated by applications to electronic band data for the A-X system of I2 and to infrared and microwave data for HF (yielding optimal new Dunham expansion coefficients for ground state HF). An automated version of this sequential rounding procedure has been incorporated in a general subroutine for performing linear or nonlinear least-squares fits. Copyright 1998 Academic Press.