Back
Full-text
Journal article

Why rankings of biomedical image analysis competitions should be interpreted with care.

  • Maier-Hein L Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany. l.maier-hein@dkfz.de.
  • Eisenmann M Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Reinke A Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Onogur S Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Stankovic M Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Scholz P Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Arbel T Centre for Intelligent Machines, McGill University, Montreal, QC, H3A0G4, Canada.
  • Bogunovic H Christian Doppler Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology, Medical University Vienna, 1090, Vienna, Austria.
  • Bradley AP Science and Engineering Faculty, Queensland University of Technology, Brisbane, QLD, 4001, Australia.
  • Carass A Department of Electrical and Computer Engineering, Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA.
  • Feldmann C Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Frangi AF CISTIB - Center for Computational Imaging & Simulation Technologies in Biomedicine, The University of Leeds, Leeds, Yorkshire, LS2 9JT, UK.
  • Full PM Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • van Ginneken B Department of Radiology and Nuclear Medicine, Medical Image Analysis, Radboud University Center, 6525 GA, Nijmegen, The Netherlands.
  • Hanbury A Institute of Information Systems Engineering, TU Wien, 1040, Vienna, Austria.
  • Honauer K Heidelberg Collaboratory for Image Processing (HCI), Heidelberg University, 69120, Heidelberg, Germany.
  • Kozubek M Centre for Biomedical Image Analysis, Masaryk University, 60200, Brno, Czech Republic.
  • Landman BA Electrical Engineering, Vanderbilt University, Nashville, TN, 37235-1679, USA.
  • März K Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Maier O Institute of Medical Informatics, Universität zu Lübeck, 23562, Lübeck, Germany.
  • Maier-Hein K Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Menze BH Institute for Advanced Studies, Department of Informatics, Technical University of Munich, 80333, Munich, Germany.
  • Müller H Information System Institute, HES-SO, Sierre, 3960, Switzerland.
  • Neher PF Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Niessen W Departments of Radiology, Nuclear Medicine and Medical Informatics, Erasmus MC, 3015 GD, Rotterdam, The Netherlands.
  • Rajpoot N Department of Computer Science, University of Warwick, Coventry, CV4 7AL, UK.
  • Sharp GC Department of Radiation Oncology, Massachusetts General Hospital, Boston, MA, 02114, USA.
  • Sirinukunwattana K Institute of Biomedical Engineering, University of Oxford, Oxford, OX3 7DQ, UK.
  • Speidel S Division of Translational Surgical Oncology (TCO), National Center for Tumor Diseases Dresden, 01307, Dresden, Germany.
  • Stock C Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
  • Stoyanov D Centre for Medical Image Computing (CMIC) & Department of Computer Science, University College London, London, W1W 7TS, UK.
  • Taha AA Data Science Studio, Research Studios Austria FG, 1090, Vienna, Austria.
  • van der Sommen F Department of Electrical Engineering, Eindhoven University of Technology, 5600 MB, Eindhoven, The Netherlands.
  • Wang CW AIExplore, NTUST Center of Computer Vision and Medical Imaging, Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei, 106, Taiwan.
  • Weber MA Institute of Diagnostic and Interventional Radiology, University Medical Center Rostock, 18051, Rostock, Germany.
  • Zheng G Institute for Surgical Technology and Biomechanics, University of Bern, Bern, 3014, Switzerland.
  • Jannin P Univ Rennes, Inserm, LTSI (Laboratoire Traitement du Signal et de l'Image) - UMR_S 1099, Rennes, 35043, Cedex, France.
  • Kopp-Schneider A Division of Biostatistics, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Show more…
  • 2018-12-08
Published in:
  • Nature communications. - 2018
Biomedical Research
Biomedical Technology
Diagnostic Imaging
Humans
Image Processing, Computer-Assisted
Reproducibility of Results
Surveys and Questionnaires
Technology Assessment, Biomedical
English International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
Language
  • English
Open access status
gold
Identifiers
Persistent URL
https://folia.unifr.ch/global/documents/233398
Statistics
Document views: 13 File downloads:
  • fulltext.pdf: 0