Uncertainty estimation for time series classification

Exploring predictive uncertainty in transformer-based models for variable stars

¹ Department of Computer Science, Universidad de Concepción, Edmundo Larenas 219, Concepción, Chile
² John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA
³ Center for Data and Artificial Intelligence, Universidad de Concepción, Edmundo Larenas 310, Concepción, Chile
⁴ Millennium Institute of Astrophysics (MAS), Nuncio Monseñor Sotero Sanz 100, Of. 104, Providencia, Santiago, Chile
⁵ Millennium Nucleus on Young Exoplanets and their Moons (YEMS), Chile
⁶ Heidelberg Institute for Theoretical Studies, Heidelberg, Baden-Württemberg, Germany

^⋆ Corresponding authors: mcadiz2018@inf.udec.cl;guillecabrera@inf.udec.cl

Received: 11 December 2024
Accepted: 30 April 2025

Abstract

Context. Classifying variable stars is key to understanding stellar evolution and galactic dynamics. With the demands of large astronomical surveys, machine learning models, especially attention-based neural networks, have become the state of the art. While achieving high accuracy is crucial, improving model interpretability and uncertainty estimation is equally important to ensuring that insights are both reliable and comprehensible.

Aims. We aim to enhance transformer-based models for classifying astronomical light curves by incorporating uncertainty estimation techniques to detect misclassified instances. We tested our methods on labeled datasets from MACHO, OGLE-III, and ATLAS, introducing a framework that significantly improves the reliability of automated classification for next-generation surveys.

Methods. We used Astromer, a transformer-based encoder designed to capture representations of single-band light curves. We enhanced its capabilities by applying three methods for quantifying uncertainty: Monte Carlo dropout (MC Dropout), hierarchical stochastic attention, and a novel hybrid method that combines the two approaches (HA-MC Dropout). We compared these methods against a baseline of deep ensembles. To estimate uncertainty scores for the misclassification task, we used the following uncertainty estimates: the sampled maximum probability, probability variance (PV), and Bayesian active learning by disagreement.

Results. In predictive performance tests, HA-MC Dropout outperforms the baseline, achieving macro F1-scores of 79.8 ± 0.5 on OGLE, 84 ± 1.3 on ATLAS, and 76.6 ± 1.8 on MACHO. When comparing the PV score values, the quality of uncertainty estimation by HA-MC Dropout surpasses that of all other methods, with improvements of 2.5 ± 2.3 for MACHO, 3.3 ± 2.1 for ATLAS, and 8.5 ± 1.6 for OGLE-III.