EPJ C Highlight - A cautionary tale of machine learning uncertainty

Underestimating machine learning uncertainty

By decorrelating the performance of machine learning algorithms with imperfections in the simulations used to train them, researchers could be estimating uncertainties that are lower than their true values.

The Standard Model of particle physics offers a robust theoretical picture of the fundamental particles, and most fundamental forces which compose the universe. All the same, there are several aspects of the universe: from the existence of dark matter, to the oscillating nature of neutrinos, which the model can’t explain – suggesting that the mathematical descriptions it provides are incomplete. While experiments so far have been unable to identify significant deviations from the Standard Model, physicists hope that these gaps could start to appear as experimental techniques become increasingly sensitive.

A key element of these improvements is the use of machine learning algorithms, which can automatically improve upon classical techniques by using higher-dimensional inputs, and extracting patterns from many training examples. Yet in new analysis published in EPJ C, Aishik Ghosh at the University of California, Irvine, and Benjamin Nachman at the Lawrence Berkeley National Laboratory, USA, show that researchers using machine learning methods could risk underestimating uncertainties in their final results.

In this context, machine learning algorithms can be trained to identify particles and forces within the data collected by experiments such as high-energy collisions within particle accelerators – and to identify new particles, which don’t match up with the theoretical predictions of the Standard Model. To train machine learning algorithms, physicists typically use simulations of experimental data, which are based on advanced theoretical calculations. Afterwards, the algorithms can then classify particles in real experimental data.

These training simulations may be incredibly accurate, but even so, they can only provide an approximation of what would really be observed in a real experiment. As a result, researchers need to estimate the possible differences between their simulations and true nature – giving rise to theoretical uncertainties. In turn, these differences can weaken or even bias a classifier algorithm’s ability to identify fundamental particles.

Recently, physicists have increasingly begun to consider how machine learning approaches could be developed which are insensitive to these estimated theoretical uncertainties. The idea here is to decorrelate the performance of these algorithms from imperfections in the simulations. If this could be done effectively, it would allow for algorithms whose uncertainties are far lower than traditional classifiers trained on the same simulations. But as Ghosh and Nachman argue, the estimation of theoretical uncertainties essentially involves well-motivated guesswork – making it crucial for researchers to be cautious about this insensitivity.

In particular, the duo argues there is a real danger that these techniques will simply deceive the unsuspecting researcher by reducing only the estimate of the uncertainty, rather than the true uncertainty. A machine learning procedure that is insensitive to the estimated theory uncertainty may not be insensitive to the actual difference between nature, and the approximations used to simulate the training data. This in turn could lead physicists to artificially underestimate their theory uncertainties if they aren’t careful. In high-energy particle collisions, for example, it may cause a classifier to incorrectly confirm the presence of certain fundamental particles.

In presenting this ‘cautionary tale’, Ghosh and Nachman hope that future assessments of the Standard Model which use machine learning will not be caught out by incorrectly shrinking uncertainty estimates. This could enable physicists to better ensure reliability in their results, even as experimental techniques become ever more sensitive. In turn, it could pave the way for experiments which finally reveal long-awaited gaps in the Standard Model’s predictions.

Ghosh, A., Nachman, B. A cautionary tale of decorrelating theory uncertainties. Eur. Phys. J. C 82, 46 (2022).

This was our first experience of publishing with EPJ Web of Conferences. We contacted the publisher in the middle of September, just one month prior to the Conference, but everything went through smoothly. We have had published MNPS Proceedings with different publishers in the past, and would like to tell that the EPJ Web of Conferences team was probably the best, very quick, helpful and interactive. Typically, we were getting responses from EPJ Web of Conferences team within less than an hour and have had help at every production stage.
We are very thankful to Solange Guenot, Web of Conferences Publishing Editor, and Isabelle Houlbert, Web of Conferences Production Editor, for their support. These ladies are top-level professionals, who made a great contribution to the success of this issue. We are fully satisfied with the publication of the Conference Proceedings and are looking forward to further cooperation. The publication was very fast, easy and of high quality. My colleagues and I strongly recommend EPJ Web of Conferences to anyone, who is interested in quick high-quality publication of conference proceedings.

On behalf of the Organizing and Program Committees and Editorial Team of MNPS-2019, Dr. Alexey B. Nadykto, Moscow State Technological University “STANKIN”, Moscow, Russia. EPJ Web of Conferences vol. 224 (2019)

ISSN: 2100-014X (Electronic Edition)

© EDP Sciences