Consistency in decision making by research ethics committees: a controlled comparison

There has been longstanding interest in the consistency of decisions made by research ethics committees (RECs) in the UK, but most of the evidence has come from single studies submitted to multiple committees. A systematic comparison was carried out of the decisions made on 18 purposively selected applications, each of which was reviewed independently by three different RECs in a single strategic health authority. Decisions on 11 applications were consistent, but disparities were found among RECs on decisions on seven applications. An analysis of the agreement between decisions of RECs yielded an overall measure of agreement of kappa = 0.286 (95% confidence interval -0.06 to 0.73), indicating a level of agreement that, although probably better than chance, may be described as "slight". The small sample size limits the robustness of these findings. Further research on reasons for inconsistencies in decision making between RECs, and on the importance of such inconsistencies for a range of arguments, is needed.