Analysis of agreement on traditional chinese medical diagnostics for many practitioners

In Traditional Chinese Medicine (TCM) diagnostics, it is an important issue to study the degree of agreement among several distinct practitioners. In order to study the reliability of TCM diagnostics, we have to design an experiment to simultaneously deal with both of the cases when the data is ordinal and when there are many TCM practitioners. In this study, we consider a reliability measure called "Krippendorff's alpha" to investigate the agreement of tongue diagnostics in TCM. Besides, since it is not easy to obtain a large data set with patients rated simultaneously by many TCM practitioners, we use the renowned "bootstrapping" to obtain a 95% confidence interval for the Krippendorff's alpha. The estimated Krippendorff's alpha for the agreement among ten physicians that discerned fifteen randomly chosen patients is 0.7343, and the 95% bootstrapping confidence interval for the true alpha coefficient is [0.6570, 0.7349]. The data was collected and analyzed at the Department of Traditional Chinese Medicine, Changhua Christian Hospital (CCH) in Taiwan.