Linear systems characterization of the topographical spatial resolution of optical instruments

Lateral resolving power is a key performance attribute of Fizeau interferometers, confocal microscopes, interference microscopes, and other instruments measuring surface form and texture. Within a well-defined scope of applicability, limited by surface slope, texture, and continuity, a linear response model provides a starting point for characterizing spatial resolution under ideal conditions. Presently, the instrument transfer function (ITF) is a standardized way to quantify linear response to surface height variations as a function of spatial frequency. In this paper, we build on the ITF idea and introduce terms, mathematical definitions, and appropriate physical units for applying a linear systems model to surface topography measurement. These new terms include topographical equivalents of the point-, line-, and edge-spread functions, as well as a complex-valued transfer function that extends the ITF concept to systems with spatial-frequency-dependent topography distortions. As an example, we consider the experimental determination of lateral resolving power of a coherence scanning interference microscope using a step-height surface feature to measure the ITF directly. The experiment illustrates the proposed mathematical definitions and provides a direct comparison to theoretical calculations performed using a scalar diffraction model.