Automated determination of bubble grades from Doppler ultrasound recordings

One of the difficulties in the development of automated algorithms for the detection of bubbles in Doppler ultrasound recordings is that expert labels are only available on an aggregate basis, i.e., the expert provides a single label for a recording which may contain many bubbles. It is thus very difficult to determine whether an algorithm is correctly identifying the actual bubbles or simply identifying the correct number of events, but mislabeling some events that are not due to bubbles.

The analysis presented here shows that the classification probabilities for the detection of bubble events and other artifacts can be determined if a large number of recordings are available.

Using a half-integer scoring system from 0-4 gives a bias of approximately 1-2% and a standard deviation that varies with the number of event sequences available, dropping from approximately 7.5% for 100 60-s recordings to 3% for 1000 60-s recordings. These values are larger if an integer scoring system is used, but using a scale finer than half-integers confers no extra benefit due to the fact that the expert labels the whole recording rather than individual bubbles.

It is thus possible to estimate the classification probabilities with a reasonably high degree of accuracy, but difficult to show that one bubble detection algorithm is superior to another to any degree of statistical significance. Expert labels can be used to validate, but not to compare the performance of bubble detection algorithms.