Related Experiment Video
Updated: Jun 19, 2026
08:25
Combining Eye-tracking Data with an Analysis of Video Content from Free-viewing a Video of a Walk in an Urban Park Environment
Published on: May 7, 2019
9.6K
Towards Practical, Best Practice Video Annotation to Support Human Activity Recognition
Hoan Tran1, Veronika Potter1, Umberto Mazzucchelli1
1Northeastern University, Boston, MA 02115, USA.
Summary
A new "silver-standard" annotation method for wearable sensor data saves 33% of annotation time compared to the gold-standard. This revised annotation approach maintains high accuracy for activity recognition model training and evaluation.
Area of Science:
- Biomedical Engineering
- Human-Computer Interaction
- Wearable Technology
Background:
- Accurate ground-truth annotations are crucial for training and evaluating wearable-sensor-based activity recognition models.
- Traditional
- gold-standard
- annotation involves two independent annotators and a third expert, which is time-consuming.
- Increasing demand for complex, 24/7, and granular activity data challenges existing annotation methods.
Purpose of the Study:
- To investigate a more efficient
- silver-standard
- annotation approach for wearable sensor data.
- To compare the efficiency and accuracy of the silver-standard method against the gold-standard and single-annotator methods.
Main Methods:
- The
- silver-standard
- method involves a second annotator revising the work of the first, instead of independent double annotation.
- Annotation quality and time efficiency were compared against the gold-standard (two independent annotators + expert resolution) and single-annotator methods.
- Inter-rater reliability was assessed using Cohen's kappa (κ).
Main Results:
- The silver-standard approach reduced total annotation time by 33% compared to the gold-standard.
- Silver-standard labels showed higher agreement with the gold-standard (κ=0.77) than single-annotator labels (κ=0.68) over 16.4 hours of video.
- Mean inter-rater reliability for silver-standard labels (κ=0.79) was higher than for single-annotator labels (κ=0.68) across 92 hours of footage.
Conclusions:
- The silver-standard annotation method offers a practical and efficient alternative for generating high-quality ground-truth data for activity recognition.
- This approach balances annotation efficiency with near-equivalent quality to the gold-standard, making it suitable for large-scale and complex datasets.
- The silver-standard method improves upon single-annotator reliability and is a viable option for researchers developing wearable sensor-based activity recognition models.
Related Concept Videos
00:48Optimal Foraging
How animals obtain and eat their food is called foraging behavior. Foraging can include searching for plants and hunting for prey and depends on the species and environment.
01:24Biodiversity and Human Values
Human civilization relies on biodiversity in many ways. Sudden changes in species biodiversity result in environmental changes that can modify weather patterns and therefore human civilizations.
02:30Naturalistic Observations
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...