SIG for Challenges


Every year, hundreds of new computational methods are published in the field of biomedical image analysis. For a long time, validation and evaluation of new algorithms was based on the authors’ private data sets, rendering fair and direct comparison of solutions impossible. In the meantime, common research practice has changed and involves the organization of international competitions (‘challenges’) that allow for comparative benchmarking algorithms in a controlled environment. After almost 15 years of biomedical challenges at MICCAI, we have carefully analyzed and critically questioned common practice related to the design and organization of biomedical challenges. Given this analysis, the mission of the Special Interest Group for Challenges is to bring biomedical image analysis challenges to the next level of quality. In this process, in line with the ‘FAIR Guiding Principles for scientific data management and stewardship’, we are committed to Findability, Accessibility, Interoperability, and Reuse of digital assets, as well as to openness, transparency, and knowledge dissemination among diverse academic, clinical, and industrial research groups.

Mission

Best practices for the MICCAI community: The SIG will have the potential to build consensus regarding standards and best practices in the field, by fostering inclusively and integrating the expertise of the minor and major SIG-specific laboratories in academia and industry, from around the world. This, in turn, has the potential to lead to better quality,  reproducibility, interpretability, and transparency of benchmarking studies.

Leading role in method benchmarking: Major machine learning and related medical imaging conferences are increasingly attracting submission from researchers that were traditionally heavily involved in MICCAI. However, best practices with respect to benchmarking in the biomedical image analysis domain requires credibility with respect to the target domain (biomedicine). The MICCAI society is thus in a unique position to establish itself as the international lead organization with respect to the systematic benchmarking of biomedical image analysis algorithms.

Assistance in challenge-related aspects: The SIG will assist MICCAI in handling challenge-related aspects, such as the review of applications for challenge organization or endorsement.