MEPS 552:61-70 (2016)  -  DOI: https://doi.org/10.3354/meps11775

Comparison of image annotation data generated by multiple investigators for benthic ecology

Jennifer M. Durden1,2,*, Brian J. Bett1, Timm Schoening3, Kirsty J. Morris1, Tim W. Nattkemper4, Henry A. Ruhl1

1National Oceanography Centre, European Way, Southampton SO14 3ZH, UK
2Ocean and Earth Science, University of Southampton, National Oceanography Centre Southampton, European Way, Southampton SO14 3ZH, UK
3GEOMAR Helmholtz Centre for Ocean Research Kiel, 24148 Kiel, Germany
4Biodata Mining Group, Faculty of Technology, Bielefeld University, 33501 Bielefeld, Germany
*Corresponding author:

ABSTRACT: Multiple investigators often generate data from seabed images within a single image set to reduce the time burden, particularly with the large photographic surveys now available to ecological studies. These data (annotations) are known to vary as a result of differences in investigator opinion on specimen classification and of human factors such as fatigue and cognition. These variations are rarely recorded or quantified, nor are their impacts on derived ecological metrics (density, diversity, composition). We compared the annotations of 3 investigators of 73 megafaunal morphotypes in ~28000 images, including 650 common images. Successful annotation was defined as both detecting and correctly classifying a specimen. Estimated specimen detection success was 77%, and classification success was 95%, giving an annotation success rate of 73%. Specimen detection success varied substantially by morphotype (12-100%). Variation in the detection of common taxa resulted in significant differences in apparent faunal density and community composition among investigators. Such bias has the potential to produce spurious ecological interpretations if not appropriately controlled or accounted for. We recommend that photographic studies document the use of multiple annotators and quantify potential inter-investigator bias. Randomisation of the sampling unit (photograph or video clip) is clearly critical to the effective removal of human annotation bias in multiple annotator studies (and indeed single annotator works).


KEY WORDS: Expert knowledge · Scoring · Visual imaging · Multiple investigators · Data quality · Quality assurance/quality control


Full text in pdf format 
Supplementary material 
Cite this article as: Durden JM, Bett BJ, Schoening T, Morris KJ, Nattkemper TW, Ruhl HA (2016) Comparison of image annotation data generated by multiple investigators for benthic ecology. Mar Ecol Prog Ser 552:61-70. https://doi.org/10.3354/meps11775

Export citation
Mail this link - Contents Mailing Lists - RSS
- -