Finite Sample Rates for Logistic Regression with Small Noise or Few Samples

The logistic regression estimator is known to inflate the magnitude of its coefficients if the sample size n is small, the dimension p is (moderately) large or the signal-to-noise ratio $1/\sigma$ is large (probabilities of observing a label are close to 0 or 1). With this in mind, we study the logistic regression estimator with $p\ll n/logn$ , assuming Gaussian covariates and labels generated by the Gaussian link function, with a mild optimization constraint on the estimator's length to ensure existence. We provide finite sample guarantees for its direction, which serves as a classifier, and its Euclidean norm, which is an estimator for the signal-to-noise ratio. We distinguish between two regimes. In the low-noise/small-sample regime ( $\sigma \lesssim (plogn)/n$ ), we show that the estimator's direction (and consequentially the classification error) achieve the rate $(plogn)/n$ - up to the log term as if the problem was noiseless. In this case, the norm of the estimator is at least of order $n/(plogn)$ . If instead $(plogn)/n\lesssim \sigma \lesssim 1$ , the estimator's direction achieves the rate $\sqrt{\sigma plogn/n}$ , whereas its norm converges to the true norm at the rate $\sqrt{plogn/\left(n{\sigma }^3\right)}$ . As a corollary, the data are not linearly separable with high probability in this regime. In either regime, logistic regression provides a competitive classifier.