Related Experiment Video
Updated: Dec 27, 2025
10:07
Using Learning Outcome Measures to assess Doctoral Nursing Education
Published on: June 21, 2010
19.3K
Applying inter-rater reliability to improve consistency in classifying PhD career outcomes
C Abigail Stayart1, Patrick D Brandt2, Abigail M Brown3
1University of Chicago, Chicago, IL, USA.
F1000Research
|May 13, 2020
Summary
Refining the Unified Career Outcomes Taxonomy (UCOT) improved PhD career outcome classification consistency. Further guidance and a flag system enhance accuracy for faculty roles.
Area of Science:
- Higher Education
- Career Development
- Data Taxonomy
Background:
- Growing demand for transparent tracking of PhD graduate career paths.
- Development of the Unified Career Outcomes Taxonomy (UCOT) in 2017 for standardized classification.
- Ambiguities in the initial UCOT version led to inconsistent application.
Purpose of the Study:
- Evaluate and enhance the inter-rater reliability of the UCOT.
- Improve the consistency of PhD career outcome classification across institutions.
- Develop a more robust and adaptable career taxonomy.
Main Methods:
- Conducted two rounds of iterative refinement on the UCOT.
- Calculated inter-rater reliability with nine coders classifying over 800 PhD alumni records.
- Developed a Guidance Document and proposed a flag system for faculty classifications.
Main Results:
- Identified and addressed areas of discordance within the UCOT.
- Improved inter-rater reliability across all three tiers of the taxonomy.
- Proposed a flag system for faculty job functions to enhance granularity and reporting.
Conclusions:
- Emphasized the need for detailed guidance, coder training, and regular review of career outcome taxonomies.
- Highlighted the importance of adaptability as PhD careers evolve globally.
- Discussed implications for training novice coders in career outcome classification.
Related Concept Videos
01:29Reliability and Validity
13.6K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.6K
01:23Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
340
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
340
01:15Receiver Operating Characteristic Plot
438
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
438
01:11Guidelines for Writing Outcome
3.6K
When developing expected outcomes for a patient care plan, the nurse should adhere to the following recommendations:
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
3.6K
02:45Self-Discrepancy Theory
18.8K
One influential perspective on what motivates people's behavior is detailed in Tory Higgin's self-discrepancy theory (Higgins, 1987). He proposed that people hold disagreeing internal representations of themselves that lead to different emotional states.
18.8K
01:24Confidence Coefficient
10.1K
The confidence coefficient is also known as the confidence level or degree of confidence. It is the percent expression for the probability, 1-α, that the confidence interval contains the true population parameter assuming that the confidence interval is obtained after sufficient unbiased sampling; for example, if the CL = 90%, then in 90 out of 100 samples the interval estimate will enclose the true population parameter. Here α is the area under the curve, distributed equally under...
10.1K