A User-Driven Method for Using Research Products to Empirically Assess Item Importance in National Surveys

Area of Science:

• Survey methodology
• Data science
• Social sciences research

Background:

• Large-scale national surveys are essential but often lengthy and burdensome for participants.
• Reducing survey length can enhance data quality and respondent engagement.
• Item removal requires careful empirical assessment to balance burden and data utility.

Purpose of the Study:

• To propose and demonstrate a method for empirically assessing survey item importance and respondent burden.
• To guide the decision-making process for item reduction in national surveys.
• To identify and remove non-essential items from the Survey of Doctorate Recipients (SDR).

Main Methods:

• Utilized a mixed-methods approach combining bibliometric analysis, website data, and paradata.
• Coded a bibliography of documents using SDR data for variable usage and citation counts.
• Analyzed SDR website data for summary statistics access and web timing paradata, including break-off rates.

Main Results:

• Identified 35 unused SDR items, representing 17% of the survey instrument.
• Found a positive correlation between item burden and item importance, indicating that the most burdensome items are often highly valuable.
• Demonstrated the feasibility of empirically assessing item importance and burden.

Conclusions:

• The proposed methodology effectively identifies non-essential survey items, reducing respondent burden.
• Prioritizing the removal of unused items can streamline data collection without compromising essential research outputs.
• The findings offer a scalable approach for optimizing national survey design and respondent experience.