Related Experiment Video
Updated: Jul 28, 2026
08:27
Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
Published on: September 27, 2019
Examining reliability and validity of job analysis survey data
1Center for Education, Widener University, One University Place, Chester, PA 19013, USA. nwang@mail.widener.edu
Journal of Applied Measurement
|October 3, 2003
Summary
Reliable job analysis data is crucial for valid examinations. This study shows that even small samples (around 400 respondents) can yield dependable task importance estimates, and rating scale effectiveness varies by profession.
Area of Science:
- Psychometrics
- Occupational Psychology
- Educational Measurement
Background:
- Job analysis is foundational for creating and validating professional licensure and certification exams.
- There is a notable gap in research regarding the reliability and validity of job analysis data.
- Ensuring the quality of job analysis data is critical for the integrity of professional assessments.
Purpose of the Study:
- To present methods for evaluating the reliability and validity of job analysis survey outcomes.
- To determine optimal sample sizes for achieving reliable job analysis estimates.
- To assess the impact of different rating scales on the quality of job analysis data.
Main Methods:
- Application of Generalizability Theory (G Theory) to assess consistency in task importance ratings.
- Utilization of the Many-Facets Rasch Model (MFRM) to investigate data generalizability.
- Analysis of random samples from job analysis data across two diverse professions.
Main Results:
- A sample size of approximately 400 respondents provides reliable estimates of task importance with high generalizability.
- Smaller, representative samples can achieve similar generalizability to larger samples in job analysis.
- The effectiveness of various rating scales is contingent upon the specific characteristics of the profession being studied.
Conclusions:
- This research offers practical methodologies for enhancing the rigor of job analysis studies.
- Findings support the use of smaller, carefully selected samples in job analysis without compromising data reliability.
- Profession-specific considerations are essential when designing rating scales for job analysis surveys.
More Related Videos
08:33A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
10:39Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
Related Concept Videos
02:16Surveys
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:07Data Collection by Survey
The systematic method of obtaining and analyzing accurate information of a population is called data collection. A survey is a standard method of data collection that involves collecting information from a target human population about their experience, opinion, or knowledge of a product, service, or process. The responses are recorded and interpreted. The most common survey examples are written questionnaires, face-to-face or telephonic conversations, focus groups, and electronic (e-mail or...
01:16What is an ANOVA?
The Analysis of Variance or ANOVA is a statistical test developed by Ronald Fisher in 1918. It is performed on three or more samples to check for equality between their means.
Before performing ANOVA, one must ensure that the samples used for this analysis have three crucial characteristics or statistical assumptions. The first assumption states that the samples should be drawn from normally distributed samples, while the second requires that all the drawn samples should be randomly and...
Before performing ANOVA, one must ensure that the samples used for this analysis have three crucial characteristics or statistical assumptions. The first assumption states that the samples should be drawn from normally distributed samples, while the second requires that all the drawn samples should be randomly and...
01:15Data Validation
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
01:29Measures of Intelligence
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this; it...