Optimal smoothing of coherence estimates

We have previously described the usefulness of the magnitude-squared coherence function in analysis of auditory evoked potentials (AEPs) (Ear Hear., 1989, 20: 2-13). For each frequency of interest, the coherence value can be compared to a 'critical value' to determine whether a response is present. Coherence functions must be smoothed across either multiple subaverages or adjacent frequencies (or both) to be reliable, but there are trade-offs: increasing the degree of smoothing increases computational time and (in the case of frequency smoothing) reduces spectral resolution. Using AEPs to clicks and amplitude-modulated tones we investigated the effects of variable degrees of smoothing on threshold estimates in 10 normal human subjects. Thresholds were found to be lower for coherence estimates than for visual detection and are also lower for longer data collection periods. However, there appears to be little if any advantage to segment smoothing beyond 8-16 subaverages. The optimal degree of frequency smoothing is more difficult to specify, depending on the spectrum of the AEP being analyzed (especially the rate of change of phase), and the spectral resolution of the analysis system.