A reconsideration of operations research experimental designs

Of the three types of experimental designs used in social science research (true experiments, nonexperiments, and quasi-experiments), the most common in family planning OR are two quasi- and two nonexperimental designs. There has been a shift over time in the OR Program from true to quasi- and now, apparently, to nonexperimental designs. The strengths and weaknesses of these designs can be summarized as a tradeoff between greater validity and lower practicality, or greater practicality and lower validity. A number of suggestions have been made for improving the validity of nonexperimental designs or improving the practicality of quasi-experiments, but these suggestions do not significantly reduce the tradeoff. The designer, therefore, has to make a choice when selecting a design. With the current emphasis in the OR Program on providing useful information to managers, it may be time to include the managers in that decision, especially if they are to be the users of the results. It may also be time to look to other methodologies, particularly management science, for appropriate tools and techniques to use in family planning field settings. Nine criteria have been suggested for helping manager-researcher teams select an appropriate design: the time it takes to carry out a design, the cost involved, the expected utility of the results, the risk involved in accepting the results as valid, the audience that will use the results, the complexity of the issue to be studied, the state of existing knowledge about the subject, the difficulty of measuring the indicators selected, and the risk of contamination of control or experimental groups. Using these criteria, a researcher could help the manager decide how much of an investment to make in the study design. Nonexperimental designs could be applied to studies that can be done quickly, that are inexpensive, that involve low risk, that are limited to the manager as the audience, that are not complex, that have high utility, about which a good deal is already known, and for which the desired indicators are easy to measure. The more rigorous designs would be selected for studies that are at the other end of the continuum: that are lengthy, that are expensive, that are high-risk, that have multiple audiences, that are complex, about which little is known, that are difficult to measure, and that have a high chance of contamination. Obviously, no single methodology or study design should be applied to all OR problems. Experiments are important in certain circumstances, but not in all, perhaps not in many.(ABSTRACT TRUNCATED AT 400 WORDS)