International Day Of Women in Statistics and Data Science

When initiating a statistical or machine learning analysis, one of the first and most consequential questions is: which method should be used for a given dataset? Traditional approaches for comparing methods include theoretical derivations and data simulation studies, which provide insight into method performance under controlled conditions. However, these approaches may not fully reflect the complexity of real-world data. Benchmarking—systematic comparison of methods across many real datasets—offers a complementary approach that can improve generalizability and provide practical guidance. Despite its common use in computer science, benchmarking remains underutilized in statistical methodology as new methods continue to emerge.

In this talk, I will discuss benchmarking in the context of statistical and machine learning research and contrast it with theory and simulation. I will outline key principles for conducting rigorous benchmarking studies and illustrate them using two case studies: benchmarking random forest variable selection methods for categorical and continuous outcomes and comparing methods for time-to-event data using the mlr3 framework. Together, these examples demonstrate how benchmarking can enhance scientific rigor, provide practical guidance for method selection, and support more transparent and reproducible methodological research.