The current gold standard for fair performance evaluation in the field of medical image analysis are international competitions (‘challenges’) enabling algorithmic benchmarking in a controlled environment while simulating real-world conditions. The mission of the Special Interest Group on Biomedical Image Analysis Challenges is to unite different societies and scientific communities to establish best practices and raise the level of quality in image analysis validation, with a special focus on biomedical image analysis challenges.

Our mission is twofold:
First, we strive for the professionalization and definitive introduction of best practices into the field of biomedical image analysis. To this end, we develop infrastructure and promote standardization in the hosting, communication, and evaluation of biomedical image analysis challenges. Committing to the ‘Findability, Accessibility, Interoperability, and Reuse (FAIR) Guiding Principles for scientific data management and stewardship’, we aim to enhance transparency and access to data and methods. Second, we strive to educate scientific communities on all aspects related to conducting challenges, with a particular focus on their organization, data quality control, and evaluation using appropriate statistics and metrics. We encourage and enable open knowledge transfer and collaboration between diverse scientific communities, and lead community discussions towards consensus on important topics, such as AI-readiness of data. We commit to our work being continuously informed by community feedback.

Mission

Best practices for the MICCAI community: The SIG will have the potential to build consensus regarding standards and best practices in the field, by fostering inclusively and integrating the expertise of the minor and major SIG-specific laboratories in academia and industry, from around the world. This, in turn, has the potential to lead to better quality,  reproducibility, interpretability, and transparency of benchmarking studies.

Leading role in method benchmarking: Major machine learning and related medical imaging conferences are increasingly attracting submission from researchers that were traditionally heavily involved in MICCAI. However, best practices with respect to benchmarking in the biomedical image analysis domain requires credibility with respect to the target domain (biomedicine). The MICCAI society is thus in a unique position to establish itself as the international lead organization with respect to the systematic benchmarking of biomedical image analysis algorithms.

Assistance in challenge-related aspects: The SIG will assist MICCAI in handling challenge-related aspects, such as the review of applications for challenge organization or endorsement.