© The Authors 2026

1. Introduction

Astronomy’s abundance of data requires appropriately scalable, automated frameworks to be mined effectively. This demand has already driven the adoption of advanced machine-learning and deep-learning techniques for a variety of purposes (see for example Domínguez Sánchez et al. 2018; Cheng et al. 2020, 2021; Vega-Ferrero et al. 2021; Walmsley et al. 2022a, 2023; Huertas-Company et al. 2024, or see Huertas-Company & Lanusse 2023; Smith & Geach 2023 for general reviews). However, in most prior work a small subset of labelled examples have been used to train supervised models. This means that only a small fraction of available data is used, and that the trained models are very sensitive to ‘domain drift’ (Domínguez Sánchez et al. 2019), thus limiting their full deployment for scientific analysis (Huertas-Company & Lanusse 2023).

Recent works have demonstrated the transformative impact of models pre-trained on large amounts of unlabelled data (that is, ‘foundation models’), which can then be fine-tuned to perform a variety of tasks. This is particularly evident for language processing, in which the best-performing ‘frontier’ models now contain many billions of parameters and require ‘web-scale’ data of the order of terabytes to petabytes to be sufficiently pre-trained (e.g. Grattafiori et al. 2024). Furthermore, the recent rediscovery of neural scaling laws – empirical relationships that demonstrate how model performance improves predictably with increases in model size, dataset size, or computational resources – has led to great growth in the size of deep-learning models (Cortes et al. 1993; Kaplan et al. 2020; Henighan et al. 2020).

Astronomical datasets are also growing rapidly in size and complexity, creating the need for models that can scale with dataset size and perform multiple tasks effectively (Smith et al. 2024; Pan et al. 2024; Walmsley et al. 2024; Merz et al. 2025; The Multimodal Universe Collaboration 2024). The advent of the next generation of large-scale sky surveys such as Euclid (Laureijs et al. 2011), the Dark Energy Spectroscopic Instrument Legacy Survey (DESI; DESI Collaboration 2022, 2024a), and the Vera C. Rubin Legacy Surveys of Space and Time (LSST; Ivezić et al. 2019) opens new opportunities for developing a new generation of astronomical foundation models. These datasets offer multi-wavelength imaging and spectroscopy of millions of galaxies with relatively little labelled data, and thus provide an ideal sandbox for developing the next generation of machine-learning models. Euclid’s mission is especially valuable for its combination of high-resolution imaging with spectroscopy over a wide field of view, which creates a unique opportunity for advancing the next generation of large-scale machine-learning models. This synergy between cutting-edge surveys and artificial intelligence-driven methods will be instrumental in accelerating the adoption of machine learning in astrophysics (Euclid Collaboration: Humphrey et al. 2023; Euclid Collaboration: Leuzzi et al. 2024; Euclid Collaboration: Aussel et al. 2024; Euclid Collaboration: Enia et al. 2024; Euclid Collaboration: Mellier et al. 2025).

Several works have started exploring large pre-trained models in astronomy (e.g. Vega-Ferrero et al. 2024). Hayat et al. (2021) and Sarmiento et al. (2021) first demonstrated the quality of the embedding spaces on Sloan Digital Sky Survey (SDSS; Gunn et al. 1998, 2006) and Mapping Nearby Galaxies at APO (MaNGA; Bundy et al. 2015) data using contrastive learning, a self-supervised technique whereby a model is trained to bring similar samples closer in an embedding space while pushing dissimilar ones apart (Chen et al. 2020; Radford et al. 2021). Smith et al. (2024) showed that autoregressive galaxy imaging models follow similar scaling laws to the ones reported for text. Complementing these imaging methods, AstroCLIP (Parker et al. 2024) pioneered a multi-modal strategy by aligning spectra and imaging via contrastive learning. Following a similar approach, the PAPERCLIP framework (Mishra-Sharma et al. 2024) explores the intersection of astronomical imaging and natural language by fine-tuning a contrastive language–image pre-training (CLIP; Radford et al. 2021) CLIP model to align observations (e.g. galaxy images) and textual descriptions. This allows users to search for images based on descriptive language, and vice versa. Zhang et al. (2024) and Rizhko & Bloom (2025) present the first attempts to incorporate time-series data with CLIP, constructing an astronomical multi-modal dataset from time-series photometry data, spectra, and astrophysical metadata. These advancements illustrate the expanding frontier of multi-modal approaches, which combine data modalities to provide a more comprehensive understanding of astronomical phenomena. As just one example of a downstream use-case, researchers can use these models to gain insights into both the structural properties and the dynamical evolution of galaxies, reaching a state-of-the-art performance on classification and redshift estimation without any explicit optimisations towards these tasks.

Following these developments, in this work we pre-train and use a multi-modal autoregressive foundation model (AstroPT) to explore the optical and infrared imaging components and the spectral energy distributions (SEDs) of Euclid Q1 galaxies (Euclid Quick Release Q1 2025). Multi-modal models are capable of integrating and learning from different types of data (such as images, spectra, and text) simultaneously, while autoregressive models predict data sequentially by learning dependencies across inputs, such as pixel sequences in images or wavelength features in spectra or SEDs. Furthermore, autoregressive models can operate on the union of available observations without being restricted solely to cross-matched datasets, which typically represent only a small fraction of the available data. These properties make models like AstroPT wellsuited to analysing the diverse, high-dimensional data produced by modern surveys. In Fig. 1 we illustrate the rapid growth of foundation models in astronomy, demonstrating an increasing trend in model size as astronomical datasets expand. This trend is motivated by model scaling laws – empirical observations that model performance improves predictably with increasing data, compute, or parameter count, provided the other two are sufficient (Kaplan et al. 2020; Hoffmann et al. 2022). While recent work has started to explore potential environmental and economic limitations to these laws (e.g. Muennighoff et al. 2023; Diaz & Madaio 2023; Sengupta et al. 2025), they remain a useful guide for forecasting model capabilities, and thus continue to drive the state of the art in language modelling. We define the number of tokens as subword units for text (GPT), image pairs for contrastive models, or image patches for the AstroPT model. With each new Euclid data release – assuming that galaxy scaling laws hold in the same way as was previously found in textual models – models are projected to scale closer to the size of GPT-2 and GPT-3, large-scale transformer-based language models developed by OpenAI, representing significant advancements in natural language processing (Brown et al. 2020). Additionally, incorporating spectral data into multi-modal models significantly enhances their complexity, potentially requiring a further increase in model size.

Fig. 1. Scaling laws for foundation models in astronomy compared to the GPT series. The number of tokens is defined as subword units in text (GPT models); pairs of augmented images (contrastive learning models); or a number of patches (AstroPT model). The projected scaling of future models incorporating Euclid data in upcoming data releases are marked with green stars. Our current model, trained on Q1 Euclid data, is represented by a red star. The models from the literature are referenced as follows: AH21 (Hayat et al. 2021); S21 (Sarmiento et al. 2021), GS21 (Stein et al. 2021); GMS24 (Martínez-Solaeche et al. 2024); P24 (Parker et al. 2024); and MS24 (Smith et al. 2024).

We evaluate the performance of an AstroPT model on a variety of downstream tasks – specific, practical applications of a pre-trained model – such as galaxy morphology classification, redshift prediction, or stellar mass estimations. By comparing these results to those achieved by a baseline fully supervised model, we assess the advantages of pre-training and multi-modal learning for astronomical data analysis. Unsupervised and self-supervised learning methods are particularly appealing for downstream tasks because they can generate semantically meaningful representations without requiring extensive labelled datasets. Such methods were successfully applied in astronomy for galaxy morphology studies (e.g. Hocking et al. 2018; Slijepcevic et al. 2024), estimations of photometric redshift (photo-z; e.g. Merz et al. 2025; Pattnaik et al. 2025), and physical properties (e.g. Euclid Collaboration: Bisigello et al. 2023; Chu et al. 2024; Euclid Collaboration: Enia et al. 2024; Gai et al. 2024), as well as for anomaly detection (see Fotopoulou 2024, for a review), and similarity searches (e.g. Stein et al. 2021; Walmsley et al. 2022b; Parker et al. 2024). Recent advancements in self-supervised learning have significantly closed the performance gap between unsupervised and supervised learning (e.g. Cheng et al. 2020; Mohale & Lochner 2024), with self-supervised models now achieving comparable accuracy on downstream tasks such as classification and regression.

This paper is structured as follows. In Sect. 2, we introduce the Euclid mission and the Q1 sample selection. The AstroPT framework is presented in Sect. 3, together with other approaches to predict galaxy properties. In Sect. 4 we discuss the performance of AstroPT on Euclid imaging data, in comparison to other methods. Section 5 summarises the main results. In Appendix A the hyperparameters of the model are given, Appendix B presents the embeddings for VIS, and NISP imaging. The star–galaxy separation is discussed in Appendix C, and physical properties in Appendix D. Finally, the accuracy of morphological classification, and predictions of stellar masses and photo-z is detailed in Appendix E.

2. Data

The Euclid mission, launched in 2023, is a mission of the European Space Agency (ESA) that is dedicated to investigating the nature of dark energy, and dark matter by measuring the accelerated expansion of the Universe. Euclid primarily focuses on gathering high-resolution imaging and spectroscopic data of around 10⁹ galaxies out to z ≈ 3, and millions of galaxies out to z ≈ 6 over 14 000 deg² (Euclid Collaboration: Mellier et al. 2025). Euclid provides broadband optical imaging through the visible imaging channel (VIS; Euclid Collaboration: McCracken et al. 2026), and multi-band near-infrared spectroscopy and imaging with the near-infrared spectrometer and photometer (NISP; Euclid Collaboration: Jahnke et al. 2025). The VIS instrument captures optical data in the visible range corresponding to r, i, and z filters (I_E covers λ = 0.53–0.92 μm; Euclid Collaboration: Cropper et al. 2025), with a spatial resolution of approximately 0$\stackrel{″}{.}$18 (Euclid Collaboration: Cropper et al. 2025). This high-resolution imaging provides an excellent foundation for morphological studies, allowing for the extraction of detailed structural properties of galaxies. The NISP captures infrared data (Y_E, J_E, and H_E cover λ = 0.95–2.02 μm; Euclid Collaboration: Schirmer et al. 2022) and is designed for photometric redshift (photo-z) determination and galaxy clustering measurements, which complement the VIS data by providing additional depth and accuracy for cosmic surveys.

2.1. Q1 sample

As part of the Euclid mission, Q1 represents the first major public data release. Q1 provides a snapshot of early observations taken during the initial phase of the mission, covering ∼29.7 million sources over 63 deg² of the Euclid Deep Fields (EDFs). These fields are the Euclid Deep Field North (EDF-N), the Euclid Deep Field South (EDF-S), and Euclid Deep Field Fornax (EDF-F). The Q1 release includes raw and processed images (calibrated VIS, NISP frames, and background-subtracted mosaics, and segmentation maps), along with the MER photometric catalogue (Euclid Collaboration: Romelli et al. 2026). A detailed description of the Q1 data release is presented in Euclid Collaboration: Aussel et al. (2026), Euclid Collaboration: McCracken et al. (2026), Euclid Collaboration: Polenta et al. (2026), and Euclid Collaboration: Romelli et al. (2026).

The MER catalogue includes multi-wavelength photometry, combining data from Euclid VIS and NISP photometry with external photometric surveys spanning the ultraviolet to the near-infrared wavelengths. In particular, the catalogue includes fluxes in the I_E, H_E, J_E, and Y_EEuclid bands, complemented by optical bands (u, g, r, i, and z) from the Dark Energy Survey (Dark Energy Survey Collaboration 2016), and the ultraviolet near-infrared optical northern survey (UNIONS; Gwyn et al. in prep.). UNIONS imaging includes CFHT u, r, HSC g, z, and Pan-STARRS i, z, and is an independent collaboration aimed at enhancing the scientific yield of large and deep surveys of the northern sky. To ensure robust cross-ground-based survey comparisons, the photometry is measured using A-PHOT (Merlin et al. 2019), employing fixed circular apertures with diameters of two full widths at half maximum (FWHMs) of the worstseeing band (typically around 2″), computed individually for each source based on PSF-matched images. The bands used to construct SEDs (i.e. the collection of photometric measurements), and their corresponding fields are listed in Table 1.

This dataset is further complemented by a {star, galaxy, quasar (QSO)} trinary classification, the photo-z, and galaxy physical properties estimated with the k-nearest-neighbour algorithm (NNPZ; Tanaka et al. 2018; Euclid Collaboration: Enia et al. 2024, 2026; Euclid Collaboration: Tucci et al. 2026), among others. While these estimates are useful for certain downstream tasks (see Sect. 4.2), spectroscopic redshifts (DESI Collaboration 2024b) and stellar masses (Siudek et al. 2024) from DESI are used for validation to ensure reliability. The DESI sample constitutes only a small fraction of the entire Q1 dataset, and does not cover the full panoply of Euclid Q1 galaxies (see Fig. 2).

Fig. 2. Stellar mass-redshift relation for the Q1 sample (in cyan), the sample used in this work (in navy blue), and our sample with DESI spectroscopic information (in red). The inset plots show the residuals of redshift and stellar mass differences between the Euclid pipeline and DESI, respectively.

2.2. Parent sample

In this work, we have selected sources with a segmentation area greater than 800 pixels in the VIS data, which corresponds to galaxies that are sufficiently large to have accurate morphological classifications (Euclid Collaboration: Walmsley et al. 2026a). The Euclid survey operates with a pixel scale of approximately 0$\stackrel{″}{.}$1 per pixel, corresponding to a physical resolution of approximately 0.5 kpc at z ≈ 0.5. This criterion ensures that the sample includes only well-resolved objects, allowing for accurate morphological characterisation. We additionally restricted our sample to galaxies with H_E < 22.5. This magnitude limit ensures the inclusion of objects that are bright enough to be detectable in the near-infrared bands, which is crucial for obtaining their spectra. Figure 3 shows the distribution of Euclid magnitudes for sources in our sample, broken down by the I_E, Y_E, J_E, and H_E bands. For the selected sample, the images are complemented with the SEDs of sources covering the optical through to infrared fluxes. An example SED constructed from Euclid and auxiliary survey data is shown in Figure 4. This work solely builds on the imaging and photometric data, while we leave the addition of the spectra for future work.

Fig. 3. Distribution of Euclid photometry for the sample used in this paper.

Fig. 4. Representative SEDs of galaxies from the Q1 EDF-N (top) and EDF-S fields (bottom). The SED includes multi-wavelength photometric data spanning the ultraviolet to the near-infrared listed in Table 1.

Our final set comprises approximately 330 000 galaxies spread across the Q1 fields, shown in blue in Fig. 2. We split our galaxy dataset using an 80/10/10 training-test-validation split. This selection introduces constraints in both redshift and stellar mass, which may influence the representation of our data and the performance of downstream tasks. In particular, our sample is mostly limited to z ≲ 1 and log₁₀(M_*/M_⊙) ≳ 7. These cuts ensure higher-quality images for our analysis but could introduce biases, particularly in studies of low-mass and high-redshift galaxies. The spectroscopic subset from DESI further refines the selection, potentially impacting our ability to characterise the most massive systems with precise redshift information.

3. Methodology

In this section, we outline the machine learning methodologies employed to analyse the galaxy dataset, starting with AstroPT in Sect. 3.1. In Sect. 3.2 we introduce three separate models used to evaluate the performance of different techniques in downstream tasks (Sect. 4.2).

3.1. AstroPT

AstroPT¹ is an autoregressive transformer designed to extract scientifically useful embeddings from high-resolution galaxy images. A visualisation of the AstroPT architecture is shown in the right panel in Fig. 5. Due to space constraints, we do not go into detail about the transformer architecture in this paper. We instead recommend Sect. 4(d) of Smith & Geach (2023) for an astronomy-friendly introduction of the transformer model, and Sects. 7 and 9 of the same publication for a further discussion of the ideas behind AstroPT and other foundation models.

Fig. 5. AstroPT’s training objective is predicting the next patch in a sequence of galaxy image (and other modalities’) patches. On the left we show the order of galaxy patches used in this work, and on the right we show how each patch is processed by AstroPT.

AstroPT is initially trained to predict the next image patch in a sequence of image patches. This training process incentivises the model to learn a compact representation of the data, which captures the essential patterns and relationships inherent in the astronomical observations it is pre-trained on. Our dataset consists of 224 × 224 pixel FITS cut-outs in the VIS and NISP bands fixed to the same angular size from mosaics, i.e. MER background-subtracted calibrated frames. The cut-out size is an arbitrary choice, corresponding to a commonly used input size for deep-learning models due to their compatibility with standard convolutional neural network architectures and AstroCLIP (Parker et al. 2024). The fixed size allows for efficient training on GPUs while maintaining a balance between spatial resolution and computational cost. However, this choice has implications for the representation of galaxy morphology: (i) smaller galaxies may be dominated by background noise or multiple objects might be in the frame; and (ii) larger galaxies might not fit entirely within the cut-out, leading to truncation of their outer regions. These effects introduce potential biases into the embeddings, as their descriptive power may depend on the relative size of the galaxy within the cut-out. A potential way to mitigate these biases would be to adopt a variable cut-out size based on the galaxy’s angular size (e.g. Petrosian radius or half-light radius), enabling better coverage of large galaxies while minimising background contamination in small ones. For images, the training process involves optimising a Huber loss (Huber 1964) objective, with the model attempting to predict the next 16 × 16 pixel patch in a sequence of galaxy image patches (as illustrated in the left panel of Fig. 5). In this study, we use the same model architecture as the 90 million parameter model described in Smith et al. (2024). To ensure consistency across inputs, each image patch is normalised by taking its zscore, defined as

$$\begin{array}{c}\hfill z=\frac{x-\mu }{\sigma },\end{array}$$(1)

where x is the pixel value, μ is the mean, and σ is the standard deviation of the pixel values across the patch. This normalisation centres the data around zero and scales it to have unit variance, improving stability during pre-training (e.g. El-Nouby et al. 2024). However, we acknowledge that global normalisation may have an impact on information, like a contrast reduction in faint structures, particularly in the outskirts of galaxies. The verification of alternative normalisation strategies, such as local normalisation or ‘min-max’ scaling, is left to future work. To incorporate SEDs, we used a simple token-chaining mechanism. This process involves concatenating the SED data as a token sequence to be fed in after the imaging tokens allowing the model to process both modalities (imaging, and SED information) simultaneously. The SEDs are divided into patches of a length of one, and the SEDs are normalised using the z-score over the entire SED sequence. While the zscore normalisation preserves the relative shape of SEDs, it could, in principle, reduce the distinctiveness of extreme SEDs (e.g. very blue or very red galaxies). Testing alternative approaches, such as converting fluxes to magnitudes before normalisation, is left for future work. We used separate encoder and decoder layers for the images and SEDs. The encoders take as input the raw image or SED patch data, and process these data so that they produce embedding patches that are of a fixed size. These embedding patches are then passed into the model ‘trunk’, shown on the right of Fig. 5. After the model trunk processes these patches, the output is then projected back into a prediction of the next image or SED patch by the decoder head, which consists of the final layers of the model.

In our case, AstroPT was pre-trained three times: firstly on solely optical (VIS) images; secondly on optical and NISP (VIS + NISP) images; and thirdly on a chained modality stream of optical and NISP images and pre-calculated SEDs (VIS + NISP + SED). A 90-million parameter model² was used in all cases due to its efficiency, as larger architectures did not show significant performance improvements. We state our full hyperparameters in Appendix A.

3.2. Models and evaluation metrics

We investigated three methodologies for predicting galaxy properties, utilising supervised and self-supervised techniques. These approaches leverage the same dataset, and are applied to both regression tasks (such as photo-z and stellar mass estimation) and classification tasks (such as morphological classification of early- and late-type galaxies).

1. Supervised convolutional neural network (CNN): This approach involves training a CNN (Fukushima 1980) using the same cut-outs as are used for AstroPT, ensuring consistency in the input data. The implementation of SEDs is beyond the scope of this paper, and so is left for future work. Our CNN consists of convolutional layers followed by dense layers, which are trained end-to-end using the mean squared error loss (MSE) function for regression tasks, and the binary cross-entropy (BCE) loss function for classification tasks. This method is a fully supervised approach whereby the network relies entirely on labelled datasets to learn the mapping from input features to output predictions.

2. Linear model: Here we evaluate the quality of our representations derived from the AstroPT framework via a simple linear or logistic regression model. This is a minimalistic and interpretable approach, whereby we trained our regressor directly on the embeddings generated by the AstroPT framework. This approach leverages the representations learned by the self-supervised model and evaluates their utility without additional modifications to the embeddings. For regression tasks, the linear regression model outputs continuous values, while for classification tasks, the logistic regression model outputs class probabilities.

3. Multi-layer perceptron (MLP) model: This method also uses self-supervised embeddings from the AstroPT model as fixed input features. An MLP (Rosenblatt 1958) is then trained on labelled data for either regression or classification tasks. The MLP consists of four fully connected layers. To ensure stable, efficient training, batch normalisation is applied after each layer and dropout regularisation (with a 30% probability) is used to prevent overfitting. We use the leaky rectified linear unit (ReLU; Nair & Hinton 2010; Xu et al. 2015) as our activation function. The model is trained using the AdamW optimiser (Loshchilov & Hutter 2017) with weight decay, and the Huber (Huber 1964) or BCE loss is used as the loss function for regression or classification tasks, respectively.

All source code and full implementation details for our downstream tasks are available in the AstroPT GitHub repository³, which provides further insights into the underlying methods. Table 2 summarises the key details about the methodologies used for downstream tasks.

The morphological labels for all our sample come from the morphology catalogue (GZ; Euclid Collaboration: Walmsley et al. 2026b), for ≈5% of our sample the stellar mass labels, and the spectroscopic redshift are taken from DESI Early Data Release (DESI Collaboration 2024b), and associated Value-Added Catalogue of physical properties of DESI galaxies (Siudek et al. 2024). The GZ catalogue was constructed by fine-tuning Zoobot foundation models (Walmsley et al. 2024) using annotations from a one-month Galaxy Zoo campaign, where ∼10 000 volunteers classified 114 000 Euclid Wide Survey galaxies. A logistic regression ‘head’ was trained on frozen Zoobot features to predict morphology labels, allowing the pipeline model to classify Q1 galaxies even though it was not trained on Q1 data. The DESI Value-Added Catalogue provides stellar masses and star formation rates for over 1.3 million galaxies across redshifts z ∼ 0 − 6, derived via CIGALE SED fitting using optical (g, r, and z) and mid-infrared (WISE) photometry.

To assess the robustness of the models, a Monte Carlo simulation was conducted with 50 (reduced to 10 for supervised CNN due to time limitations) runs for different percentages of data used for training, each time evaluating key metrics given below. To evaluate our models’ performance, we used different sets of metrics depending on whether the tasks were regression or classification.

To predict continuous values, such as photo-z, or stellar mass estimates, we relied on the following metrics commonly used in regression tasks

• Bias (Δx): Indicates systematic offsets in predictions. We applied the normalisation to account for the scale dependence of redshift errors that scale with 1 + x_true. We also kept it for stellar mass estimates:

  $$\begin{array}{c}\hfill \mathrm{\Delta }x=\frac{x_{\text{pred}}-x_{\text{true}}}{1+x_{\text{true}}},\end{array}$$(2)

  where x_true and x_pred correspond to the true and predicted values, respectively.

• Normalised median absolute deviation (NMAD): A robust statistic to measure scatter that is less sensitive to outliers. It is defined as

  $$\begin{array}{c}\hfill \text{NMAD}=1.4826\times \text{median}\left(|\mathrm{\Delta }x-\text{median}(\mathrm{\Delta }x)|\right).\end{array}$$(3)

• Outlier fraction: The fraction of predictions with |Δx| exceeding 0.15, and 0.25 for photo-z, and stellar mass, respectively, quantifying catastrophic prediction errors:

  $$\begin{array}{c}\hfill \text{Outlier fraction}=\frac{\text{Number of galaxies with}|\mathrm{\Delta }x|>0.15(0.25)}{N}·\end{array}$$(4)

  The choice of the threshold of 0.15 for photo-z is commonly used to define catastrophic outliers. For stellar mass, while no strict convention exists, a threshold of 0.25 corresponds to the standard stellar mass error (Siudek et al. 2024).

For binary classification tasks such as morphological classification (such as early-type versus late-type galaxies), we used the following metrics.

• Accuracy: The proportion of correct predictions (both true positives and true negatives) among all predictions:

  $$\begin{array}{c}\hfill \text{Accuracy}=\frac{N_{\mathrm{TP}}+N_{\mathrm{TN}}}{N_{\mathrm{TP}}+N_{\mathrm{FP}}+N_{\mathrm{TN}}+N_{\mathrm{FN}}},\end{array}$$(5)

  where N_TP, N_TN, N_FP, and N_FN are the number of true positives, true negatives, false positives, and false negatives, respectively.

• Precision: The proportion of true positives among all instances predicted as positive:

  $$\begin{array}{c}\hfill \text{Precision}=\frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FP}}}·\end{array}$$(6)

• Recall (true positive rate, TPR): The proportion of true positives among all actual positive instances:

  $$\begin{array}{c}\hfill \text{Recall}=\frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FN}}}·\end{array}$$(7)

• False positive rate (FPR): The proportion of false positives among all actual negatives:

  $$\begin{array}{c}\hfill \text{FPR}=\frac{N_{\mathrm{FP}}}{N_{\mathrm{FP}}+N_{\mathrm{TN}}}·\end{array}$$(8)

• F1 score: The harmonic mean of precision and recall, balancing the trade-off between them:

  $$\begin{array}{c}\hfill F1=2\times \frac{\text{precision}\times \text{recall}}{\text{precision}+\text{recall}}·\end{array}$$(9)

4. Results

In this work, we explore the latent representations of galaxies derived from our pre-trained AstroPT models applied to three distinct datasets. The complementary nature of VIS, VIS + NISP, and VIS + NISP + SED data facilitates a systematic evaluation of the information content in the latent spaces derived from our models. This comparison also allows us to assess how well each wavelength constrains a galaxy’s physical parameters and to understand the limitations of relying solely on optical imaging.

4.1. AstroPT: Quality of self-supervised embeddings

In this section, we assess the quality and interpretability of self-supervised embeddings obtained from the Euclid VIS, VIS + NISP, and VIS + NISP + SED data via AstroPT. We do this by analysing their correlation with source properties. In all downstream tasks (including anomaly and similarity search, see Sect. 4.2), we used the raw 768-dimensional neural network embeddings as input. For visualisation purposes, we projected this high-dimensional embedding space onto two dimensions using uniform manifold approximation and projection (UMAP; McInnes et al. 2018). We initialised UMAP with a consistent set of hyperparameters for all visualisations (n_neighbours = 15, min_dist = 0.1, n_components = 2, random_state = 42) to ensure consistency across tasks. We validated different UMAP parametrisations, to: (i) determine whether the randomness in UMAP initialisation impacts the visualisations; (ii) to emphasize local structures that can potentially create tighter clusters; and (iii) to highlight the global relationships. While the shapes of the projections varied slightly across these settings, the trends related to galaxy morphological and physical properties remained consistent. Thus, we stuck to the default values that provide a good trade-off between the local, and global structures. The UMAP projection facilitates an intuitive exploration of clustering patterns and structures within the data. UMAP is particularly effective in preserving local and global structures, often outperforming methods such as principal component analysis (PCA) in revealing underlying relationships in the data (e.g. McInnes et al. 2018). However, we note that the dimensionality reduction algorithm ultimately influences the patterns and separations observed in these visualisations. While the UMAP projections suggest that relevant morphological and physical information is encoded in the AstroPT embeddings, the degree of separation achieved depends on the algorithm’s ability to preserve the global structure.

To more directly evaluate the utility of these embeddings, we also examined their performance in downstream tasks through approaches like the linear and MLP models (see Sect. 4.2). These analyses provide a complementary and quantitative assessment of the embeddings’ capacity to capture source properties, independent of the visualisation technique used.

A visualisation of the self-supervised embeddings trained on the VIS + NISP + SEDs images with example cut-outs and SEDs is shown in Fig. 6. The figure was created by binning the embedding space into a 50 × 50 grid and randomly selecting a sample from each cell, plotting its corresponding colour-mapped galaxy image at that location. The images were colour-mapped based on the VIS and NISP band images. The RGB image was generated only for visualisation purposes by averaging the Y_E-, J_E-, and H_E-band images to form an NISP channel, combining the VIS channel with this NISP channel to form a composite, and then stacking these three channels: VIS, NISP, and the composite. To preserve luminance, the luminosity channel of the resulting RGB image was replaced with the VIS channel, maintaining the brightness information while allowing for distinct colour contrasts. Zoom-in panels around the edges show that galaxies with similar morphologies tend to be clustered in different areas allowing for a separation of edge-on (left zoom-in panel), elliptical (top zoom-in panel), and spiral (bottom zoom-in panel) galaxies, while stars and low-quality images are separated in distinct cluster (right zoom-in panel). The zoom-in panels of spiral, and elliptical galaxies are selected within a photo-z range of 0.5 to 0.7, ensuring a fair comparison of their SEDs. While no striking differences are observed, the SEDs appear to broadly reflect morphological trends. Specifically, edge-on disc galaxies and ellipticals tend to be redder. However, this distinction is subtle and subject to interpretation. Additionally, the SEDs of stars and sources with lower data quality show spurious features. A similar plot, but generated using the AstroPT embeddings trained on VIS, and VIS + NISP data, is provided in Appendix B.

Fig. 6. UMAP visualisations of the embeddings from AstroPT trained on VIS + NISP + SEDs with example cut-outs and SEDs, see Sect. 4.1 for more details.

The visual similarity of sources located close to each other in the representative space is also confirmed by their properties. The outlier regions are clearly separated from the main representations and are mostly attributed to bad-quality imaging and stars. Figure 7 shows the relationship between the embeddings trained on VIS images only, on VIS + NISP images, and VIS + NISP + SED, and the detection quality flag. The detection quality flag (DET_QUALITY_FLAG) is a flag indicating specific issues with sources, such as contamination, blending, or proximity to bright stars (Merlin et al. 2015, 2016, 2019; Kümmel et al. 2022). The UMAP projections reveal significant structure within the embedding space, reflecting the quality of the data: sources with high-quality images (DET_QUALITY_FLAG < 4) cluster tightly, whereas sources with contamination (e.g. blending or proximity to bright stars) form distinct outlier clusters. Including NISP data enhances the separation of poorly flagged sources, suggesting that NIR data provides complementary morphological information compared to VIS data. This improvement may arise because NIR observations are less affected by dust extinction and better trace the underlying stellar mass distribution, which can help distinguish extended or blended sources that are challenging to classify in VIS alone. In particular, embeddings trained on VIS + NISP images reveal two primary branches: one dominated by sources flagged as unknown (i.e. not fitting galaxy, star, or QSO templates); and another branch associated with sources of lower quality. However, the separation between galaxies, stars, and QSOs remains less defined (see details in Appendix C). When SED information is incorporated into the training (VIS + NISP + SED), the embeddings show a bimodal structure, which is driven by the spatial distribution: the EDF-N and EDF-S are targeted by different telescopes (see Sect. 2). Besides these two main branches, stars are also more distinctly separated, benefiting from the additional spectral information provided by SEDs. Furthermore, the unknown sources form a more coherent and distinguishable cluster, indicating that the combination of imaging and SED data aids in isolating these objects from other classes. This highlights the potential of SED data to complement imaging in disentangling complex or ambiguous cases, such as blended sources or objects with uncertain classifications.

Fig. 7. Top: UMAP visualisations of the self-supervised embeddings from AstroPT trained on VIS (left), VIS + NISP (centre), and VIS + NISP + SED (right) images coloured by the detection quality flag. This flag allows one to identify sources with contaminants, e.g. proximity star (DET_QUALITY_FLAG ≥4). Bottom: Example VIS + NISP cut-outs of the outliers in the representative space.

We also examined the correlation of the self-supervised embeddings with morphological parameters derived from the GZ catalogue (Euclid Collaboration: Walmsley et al. 2026b). Early-type galaxies typically exhibit smooth profiles, while late-type galaxies tend to show distinct features such as spiral arms or a bulging core, contributing to a more complex, less smooth morphology. Although not identical, the GZ classification between smooth and featured galaxies provided with the Euclid Q1 release (Euclid Collaboration: Walmsley et al. 2026b) closely approximates an early- and late-type separation. The UMAP visualisations reveal distinct clustering patterns that correspond to an early and late-type separation and the presence of spiral arms as shown in Fig. 8. Across all embeddings – whether AstroPT is trained on VIS, VIS + NISP, or VIS + NISP + SED – specific regions of the latent space are consistently associated with early-type galaxies, which also exhibit a low probability of hosting spiral arms. Conversely, other areas correspond to late-type galaxies with pronounced spiral structures. While the gradients within VIS + NISP + SED embeddings are clear, the separations between morphological types become harder to identify. This suggests that the embeddings prioritise spectral diversity over morphological clarity when SED data are included, highlighting a trade-off between the abstraction of spectral information and the preservation of morphological distinctions. This highlights the ability of automatic separation of elliptical and star-forming galaxy populations.

Fig. 8. UMAP visualisations of the self-supervised embeddings from AstroPT trained on VIS (left), VIS + NISP (centre), and VIS + NISP + SED (right) coloured by morphological parameters derived from the GZ catalogue (Euclid Collaboration: Walmsley et al. 2026b). Representations show clear clustering of smooth, non-spiral, and non-smooth, spiral galaxies.

Interestingly, edge-on galaxies form a clearly identifiable region in the embeddings trained on VIS images alone, highlighting the capability of AstroPT to capture these specific morphological features (see the left panel in Fig. 9). This morphological distinctiveness aligns with their SED properties: a rising flux towards longer wavelengths, indicative of more dust, supports expectations for edge-on star-forming disc galaxies. However, as additional data from the NISP bands, and SEDs are incorporated, this distinct separation becomes less pronounced in the VIS + NISP and VIS + NISP + SED embeddings, as seen in the middle, and right panels of Fig. 9. While the inclusion of NISP and SED data enriches the latent space by providing additional spectral information, it does not necessarily enhance the clarity of morphological separations. Instead, the embeddings appear to encode a broader range of physical and morphological characteristics, becoming more abstract as additional data are integrated. The reduction in morphological separation, particularly for edge-on galaxies, may also be attributed to the lower resolution of NISP images relative to VIS, which limits the preservation of finer structural details. This does not imply that AstroPT’s performance degrades with lower-resolution data but instead highlights the trade-offs in representation fidelity when combining datasets of varying resolutions. These findings demonstrate the adaptability of AstroPT to encode rich, multidimensional information, albeit at the cost of losing some morphological specificity in certain contexts.

Fig. 9. UMAP visualisations of the self-supervised embeddings from AstroPt coloured by morphological parameters derived from the GZ catalogue (Euclid Collaboration: Walmsley et al. 2026b). Left: Representations from VIS images alone, showing a distinct region for edge-on galaxies. Middle: Representations incorporating both VIS and NISP data, where the separation of edge-on galaxies becomes less distinct. Right: Representations incorporating VIS, NISP, and SED data. Bottom: Example cut-outs and SEDs of the edge-on galaxies.

Similarly, we find that physical properties are correlated with the latent space and morphological parameters. Namely, the region associated with spiral galaxies is characterised by lower stellar masses, while the space occupied by elliptical galaxies tends to have higher masses. There is though no trend with the Euclid spectroscopic redshift and only a subtle trend of higher redshift for more massive galaxies using photo-zs. More details are given in Appendix D.

These visualisations confirm that the self-supervised embeddings capture meaningful physical and morphological properties, with improved performance when both VIS and NISP data are utilised, except for the shape elongation (i.e. edge-on, ellipticity) which are caught by VIS embeddings but blended when using VIS + NISP imaging. The latent space’s ability to distinguish between low- and high-quality data, as well as between source types, underscores the robustness of the AstroPT framework for image analysis.

4.2. Downstream tasks

In our work, we evaluate the utility of self-supervised representations for different downstream tasks: morphology classification (see Sect. 4.2.1), photo-z estimations (see Sect. 4.2.2), and stellar mass predictions (see Sect. 4.2.3).

4.2.1. Morphological properties

The performance of three different methods – a supervised CNN, a linear model, and an MLP model (see Sect. 3.2) –is evaluated for their ability to classify galaxy as early-type or late-type based on the morphological catalogue (see Euclid Collaboration: Walmsley et al. 2026b). We define galaxies as early-type if the probability of being classified as smooth exceeds 0.5 and as late-type otherwise. However, this threshold-based classification may introduce inaccuracies, as it relies on predicted continuous probabilities rather than an inherent binary distinction. The accuracy, precision, and recall as a function of the percentage of labelled data are shown in Fig. 10. Table 3 shows a statistical summary for models trained on 100% of the labels, while Table E.1 extends this comparison across all percentages of labelled data. As expected, the results suggest that performance varies with the percentage of the labelled data used for training, but this dependence is neither uniform nor consistent across methods. In particular, the linear and MLP models present relatively stable performance, with only minor fluctuations, suggesting that even modest amounts of labelled data are sufficient for robust classification. However, the supervised CNN shows a more pronounced dependence on labelled data, with a clear decline in accuracy, precision, and recall as the labelled dataset size decreases. Interestingly, incorporating NISP imaging does not enhance classification performance; in some cases, it even slightly lowers accuracy, precision, and recall, possibly due to the NISP spatial resolution (see also discussion about edge-on galaxies in Sect. 4.1) or due to z-score normalisation (see Sect. 3). Similarly, adding SEDs does not improve classification, likely because the SEDs contain little direct morphological information. These findings suggest that while increasing the number of labelled examples provides some benefits, a self-supervised pre-training dramatically reduces the need for labelled samples. The self-supervised embeddings, trained on a relatively small labelled sample, already provide a robust foundation for morphological classification. Notably, the MLP model consistently achieves the highest performance across all metrics, particularly in distinguishing between early-type and late-type galaxies.

Fig. 10. Comparison of the accuracy, precision, and recall as functions of the percentage of labels used for training, for early-type/late-type classification. Results are shown for the VIS (left), VIS + NISP (centre), and VIS + NISP + SED configurations.

4.2.2. Photometric redshifts

We verified the performance of different methods in predicting photo-z using the metrics described in Sect. 3.2. The spec-zs from DESI, available for about 5% of our sample, were assumed to be the true redshifts. The accuracy of Euclid photo-z derived with NNPZ as part of the Q1 release is shown in the top panel in Fig. 11. Figure 12 shows the bias, NMAD, and outlier fraction as a function of the percentage of labels used for training for photo-z obtained through these methods. A statistical summary is provided in Table 4 for a 100% of labels used for training, for the full table see Table E.2. The results highlight the impact of adding NISP imaging data, and SED information, and the effectiveness of different training approaches. The inclusion of NISP imaging improves photo-z accuracy, reducing metrics to levels comparable to those achieved by the official Euclid pipeline coming from the NNPZ template fitting algorithm (Euclid Collaboration: Tucci et al. 2026). These improvements are consistent with previous findings, such as those by Salvato et al. (2019), who demonstrated the importance of including NIR bands to constrain galaxy redshifts, particularly at higher redshifts. The performance is further improved when the SED information is included in the training. The bottom panel in Fig. 11 illustrates the ability of the MLP model to predict photo-zs, showing that the VIS + NISP + SED configuration achieves tighter clustering of predictions around the ideal 1:1 line, while the NNPZ photo-zs exhibit a more significant offset. Notably, the MLP model that catches the non-linearity tends to outperform other methods across all metrics, particularly for low percentages of labelled data. This approach achieves lower NMAD, and outlier fractions demonstrating its ability to adaptively leverage the added NISP, and SED information. For instance, with 100% labelled data, the MLP model achieves an NMAD = 0.048 ± 0.002 and an outlier fraction of 2.13 ± 0.38%, for training on VIS + NISP + SEDs approaching the performance of the official Euclid photo-z pipeline (NMAD = 0.037 and an outlier fraction of 2.21%). Linear models, while simpler, deliver competitive results for higher percentages of labelled data, where they approach the performance of Euclid pipeline, and more complex models. We note that we observe a non-monotonic behaviour in model performance as a function of training sample size. Specifically, the NMAD and outlier fraction peak at 5% training data, while performance improves at both smaller (1%) and larger (20%) sample sizes. This behaviour suggests the combination of the presence of the double descent effect (e.g. Schaeffer et al. 2023) in our regression model, and sample size balance. The double descent phenomenon occurs when test error initially decreases with increasing data size (classical regime), then spikes at an interpolation threshold, and finally decreases again as overparameterisation allows for better generalisation. In our case, with larger sample sizes (> 10%), the model moves into the overparameterised regime, where it can better capture meaningful patterns. On the other hand, when training on extremely limited data, the model may underfit, but given the low parameter count in linear regression this underfitting is less pronounced compared to the interpolation threshold. The fact that 1% performs comparably to 10% suggests that at extremely low sample sizes, the model is unable to overfit noise significantly, leading to stable results. In comparison, supervised CNN models exhibit modest improvements with increased labelled data, but generally underperform compared to other methods.

Fig. 11. Comparison of photo-z from Euclid official NNPZ pipeline (top) and predictions from the MLP model trained on AstroPT embeddings (bottom) versus DESI spec-z. A solid black line represents the 1:1 relation, while dotted lines indicate the outlier thresholds of 0.15.

Fig. 12. Comparison of the bias, NMAD, and outlier fraction as functions of the percentage of labels used for training, for true redshift (DESI spec-z) versus Euclid photo-z predictions. Results are shown for the VIS (left), VIS + NISP (middle), and VIS + NISP + SED (right) configurations.

Overall, an MLP model trained with SED (VIS + NISP + SED data), not only achieves the best overall metrics, with improvements in bias and outlier fraction values in comparison to photo-zs from NNPZ for higher labelled data percentages, but also maintains superior results under low-data scenarios, demonstrating its robustness and adaptability for photo-z estimation tasks requiring only the SED information. These results underscore the effectiveness of leveraging advanced training strategies for accurate photometric redshift estimation.

4.2.3. Stellar masses

In this section we evaluated how AstroPT model is able to predict the stellar masses in comparison to more traditional approaches. As the ground truth, we assume the stellar masses of the DESI Value-Added Catalogue (Siudek et al. 2024), which provides a robust dataset derived from the SED fitting of optical and infrared photometry. First, we compare the stellar masses from Euclid official pipeline (Euclid Collaboration: Tucci et al. 2026) with the ones from DESI (see Fig. 13). The stellar masses are in agreement, as shown by the following 1:1 relation, but there is also a wide spread.

Fig. 13. Comparison of stellar mass from Euclid official NNPZ pipeline (top) and predictions from MLP model trained on AstroPT embeddings (bottom) versus DESI stellar masses from Siudek et al. (2024). A solid black line represents the 1:1 relation, while dotted lines indicate the outlier thresholds of 0.25 dex.

Similarly, as for the morphological parameters and photo-zs, we predict stellar mass using different methods and metrics introduced in Sect. 3.2. We compare this property for both VIS and VIS + NISP models, but do not compare for our VIS + NISP + SED model as the z-score normalisation across the SEDs removes the additional information needed to predict this property. The bias, NMAD, and outlier fraction as a function of the percentage of labelled data for the stellar mass are shown in Fig. 14. A statistical summary is given in Table 5. The MLP model, and linear model consistently achieve the lowest NMAD across, but only MLP is independent on the number of labels used for training, demonstrating its robustness even with limited labelled data. The linear method, similarly to the photo-z estimates, shows the non-homogenous behaviour due to sample size, and the double descent effect (see Sect. 4.2.2 for details). Nevertheless, for all labels used in training, both linear and MLP methods outperform estimates from Euclid official pipeline when the AstroPT is trained on both VIS and NISP data. In particular, stellar mass predictions with the linear model are statistically even closer to DESI stellar masses than official NNPZ estimates (with NMAD 0.023 ± 0.001, and 0.027, and outlier fraction 0.3 ± 0.1%, 2.34% for MLP, and NNPZ, respectively. The performance degrades when solely VIS images are used for training. This highlights the added value of NISP’s complementary wavelength coverage in constraining stellar mass. The distributions of the MLP stellar mass predictions is shown in the bottom panel in Fig. 13.

Fig. 14. Comparison of the NMAD and outlier fraction as functions of the percentage of labels used for training, for DESI stellar masses from Siudek et al. (2024) versus Euclid predictions. Results are shown for the VIS (left) and VIS + NISP (right) configurations.

4.2.4. Anomaly search

To identify peculiar objects within the learned feature space, we applied two unsupervised anomaly detection algorithms – isolation forest (IF; Liu et al. 2008) and local outlier factor (LOF; Breunig et al. 2000) – to the AstroPT embeddings. Both methods were implemented using the Scikit-learn Python package (Pedregosa et al. 2011). Before running the algorithms, we filtered out spurious detections by removing sources with DET_QUALITY_FLAG > 0. To ensure robustness against sensitivity to parameter choices, we executed each algorithm multiple times with varying hyperparameters. Specifically, IF was tested with different numbers of base estimators, ranging from 50 to 200 in increments of 50, and the maximum number of features for splitting is varied between 1 and 20 in steps of 5. Additionally, we set the contamination parameter to an estimated fraction of outliers in the dataset and do not use bootstrap sampling. For LOF, we explored the effect of different neighbourhood sizes by varying the number of neighbours from 5 to 50 in steps of 5. We used the ‘ball-tree’ (Dolatshah et al. 2015) algorithm for nearest neighbour searches and employed the cosine distance metric, as it has been shown to better capture similarities in high-dimensional spaces (Aggarwal et al. 2001; You 2025). The contamination parameter is set to reflect the expected proportion of outliers.

In the embeddings trained solely with VIS images, LOF predominantly detects anomalies often associated with spurious features, such as cosmic rays, bright stars in close proximity, or ghost artefacts (see upper panel in Fig. 15). We note that these objects were not flagged by standard quality-control metrics, indicating LOF’s ability to identify subtle irregularities that might otherwise be overlooked. Additionally, LOF identifies extended objects in the outskirts of the cut-out region, sometimes galaxies located far from image centres. These galaxies tend to be brighter and more extended than the central source, though such cases are relatively rare since most relevant features are concentrated near the image centre. In contrast, IF predominantly identifies large galaxies with rich morphological structures, including those with prominent rings indicative of recent merger events (see lower panel in Fig. 15). A key distinction between the two methods emerges in the spatial distribution of detected anomalies within the embedding space: LOF-identified outliers are scattered throughout, whereas IF-detected anomalies cluster closely together in the latent space (see Fig. 16). These differences highlight the sensitivity of each method to different types of anomalies, with LOF favouring local density variations, and IF is adept at detecting sources with complex morphological structures.

Fig. 15. Example VIS cut-outs of anomaly detections found with LOF (upper panel) and IF (bottom panel) on VIS embeddings.

Fig. 16. UMAP visualisations of the self-supervised embeddings trained on VIS images with anomalies detected with LOF, and IF algorithms.

When incorporating both VIS and NISP embeddings, the nature of the detected anomalies shifts. Both with IF and LOF algorithms, the VIS + NISP embedding highlights objects with multiple sources appearing within the field of view, potentially identifying interacting galaxies (see Fig. B.2). However, this embedding is also prone to contamination by non-interacting systems, as the algorithms do not inherently account for redshift differences and instead focus on structural features within the images. Including redshift information as an additional constraint within the anomaly detection process could help disentangle interacting systems from unrelated projections of nearby and distant galaxies. Alternatively, utilising smaller image cut-outs centred on each source might reduce confusion from neighbouring objects, improving both the accuracy and physical interpretability of detected anomalies. These refinements would enable a more reliable identification of astrophysically meaningful outliers within the embedding space.

When SED information is included, the nature of the detected anomalies changes once again. This exemplifies how complementary information can alter the structure of the embedding space and, consequently, the anomaly detection results. As more information is incorporated, anomalies become potentially more interesting. When using the IF algorithm, we detect objects with significant dust presence, predominantly in the H_E band (see Fig. 17). The inclusion of SED data provides a measure of the relative importance of emission across different wavelengths, encoding valuable physical information. Interestingly, although the LOF algorithm still tends to detect anomalies driven by spurious features, it is nonetheless able to identify the two most prominent anomalies found with IF.

Fig. 17. Anomalies detected with the IF algorithm in the VIS + NISP + SED embedding. For each galaxy we show VIS and NSIP images and the SED. Fluxes are normalised to the Euclid VIS value.

4.2.5. Similarity search

AstroPT projects objects with comparable features into the same regions of the embedding space, facilitating efficient similarity searches to identify objects of interest. This approach has shown great promise in various applications, particularly in identifying interacting galaxies. By using an interacting galaxy as a query, one can probe the embedding space to assess how many of its neighbouring galaxies also exhibit signs of interaction. This method allows for the rapid identification of potential interacting galaxies, contributing to large-scale surveys and studies of galaxy dynamics. In this work, we explore the potential of similarity search for identifying interacting galaxies and investigate how this approach can be adapted to astronomical datasets. As shown in Fig. 18, a significant fraction of the neighbouring galaxies to a query galaxy – chosen as an interacting galaxy – hereafter referred to as ‘similar objects’ – are indeed interacting. In particular, the distribution of the merging_merger parameter, which quantifies the probability of galaxies being in a merger process (Euclid Collaboration: Walmsley et al. 2026a), shows a higher fraction of merger candidates among the similar objects compared to the general galaxy population. This suggests that the similarity search effectively identifies galaxies involved in interactions. However, we also observe that the distribution of similar objects remains elongated, extending towards non-interacting galaxies as indicated by merging_merger ≈0, similar to the general distribution. This implies that, while the method is generally effective, it is not without contamination. In particular, the similarity search identifies some galaxies that are not truly interacting, highlighting the need for further refinement of the approach to reduce such false positives.

Fig. 18. Distribution of merging_merger parameter for similar objects computed with VIS embedding and general galaxy distribution.

The presence of non-interacting neighbours, or ‘false positives’, in the identified clusters can be explained by the way AstroPT was trained and operates. During the training process, a fixed image size was chosen to ensure that the largest galaxies were fully captured within the image frame. This approach works well for galaxies at lower redshifts, where the central regions of galaxies can be clearly separated. However, for higher-redshift galaxies, the fixed image size may include multiple objects or extended outskirts of galaxies within the same frame. As a result, the algorithm may erroneously prioritise similarities in the outskirts of these galaxies, where there may be multiple objects or extended features, rather than focusing solely on the central regions where interactions are more likely to be detected. This can lead to the identification of objects as interacting when they are not, especially if the objects in question lie at different redshifts or are not truly interacting.

Despite these limitations, the approach shows great potential in identifying interacting galaxies. The contamination in the results, while non-negligible, offers insight into the nuances of the similarity search method and highlights the need for further refinement, particularly in distinguishing objects that are at different redshifts or in disentangling the central regions of interacting galaxies from their outskirts. These findings contribute to the broader understanding of similarity search applications in astrophysics and suggest areas for improvement in future models to enhance the accuracy of interaction detection.

5. Summary

In this paper, we have explored AstroPT, an autoregressive foundation model pre-trained on approximately 300 000 optical and infrared Q1 imaging and SEDs for deriving galaxy properties. By learning a unified and informative representation from multi-modal data, AstroPT provides a scalable solution for analysing large, unlabelled datasets, reducing dependence on human-labelled training samples while maintaining high performance for a variety of downstream tasks (such as photometric redshift estimation, morphology classification, anomaly detection).

The paper’s highlights are as follows.

• Advancing foundation models in astronomy: AstroPT showcases the potential of multi-modal foundation models for a scalable and efficient resolution of many crucial astrophysical tasks, such as morphology classification, galaxy characterisation, and the search for peculiar sources. Unlike traditional task-specific models, AstroPT provides a generalisable feature space applicable across multiple domains, demonstrating the feasibility of self-supervised learning for astronomy at scale. Our results provide critical insight into the synergy between optical, infrared, and SED data, guiding future strategies for galaxy characterisation and cosmological analysis with Euclid.

• Multi-modal synergies: The integration of optical, and NIR photometry highlights the advantages of multi-modal approaches. Pre-trained embeddings capture the inherent diversity of galaxy morphologies (see Figs. 6 and 8), clearly isolating galaxy classes hardly distinguishable with standard methods, such as edge-on galaxies (see Fig. 9), and outliers, such as spurious objects and stars missed by standard techniques (see Figs. 7 and C.1).

• Unified representations for multiple tasks: Self-supervised embeddings provide a consistent representation for various downstream tasks, including photo-z and stellar mass estimates, anomalies, and similarity searches. This cross-task applicability underscores the flexibility of AstroPT’s learned representations, which remain effective across different astrophysical domains. By leveraging the task-agnostic nature of self-supervised embeddings, we reduce dependence on large labelled datasets while maintaining high performance. This scalability is crucial for addressing the challenges of analysing large multi-modal datasets in future surveys.

• Morphological classification: Using solely optical images, our models achieve robust performance in predicting galaxy types. Surprisingly, incorporating NIR data and SEDs does not improve classification results, indicating that the additional spectral coverage does not introduce significant morphological information. The MLP and linear models demonstrate consistent performance across different dataset sizes, maintaining higher robustness compared to traditional supervised approaches, even with a limited number of labels (see Fig. 10 and Table 3). This suggests that deep-learning-based embeddings effectively capture morphological characteristics using only optical imaging, potentially simplifying future survey strategies.

• Photo-z estimations: By integrating NIR, optical images, and SEDs, our models achieve competitive accuracy in photo-z estimates compared to the standard Euclid pipeline relying on traditional methods. Notably, the MLP model shows reduced bias and a lower fraction of catastrophic outliers, demonstrating that self-supervised embeddings can enhance photometric redshift predictions while being less reliant on large labelled training samples (see Fig. 12, and Table 4). This highlights the potential of deep-learning models adapted for standard tasks such as photo-z estimations, and the importance of multi-band photometry for capturing diverse galaxy properties and achieving higher precision.

• Stellar mass estimations: We explore the ability of AstroPT to reconstruct fundamental galaxy properties, such as stellar mass, showing that MLP-based models achieve high accuracy even in low-label regimes. The linear model performs comparably well, showcasing the flexibility of the learned representations (see Fig. 14 and Table 5). This result highlights the robustness of self-supervised learning in deriving astrophysical parameters, even when traditional template-fitting approaches require extensive labelled training sets.

• Outlier and similarity search: AstroPT’s anomaly and similarity search capabilities allow for the identification of rare astrophysical objects, including galaxies with rings, interacting systems, and other unusual morphologies. Including SEDs in training, we can also identify very dusty sources. This demonstrates the model’s utility in detecting previously overlooked populations in large-scale galaxy surveys, paving the way for new discoveries in Euclid and future astronomical datasets. This work underscores the transformative potential of combining multi-modal datasets, self-supervised learning, and foundation models for tackling large-scale challenges in galaxy analysis.

• Limitations, and future perspectives: While AstroPT demonstrates strong performance across multiple tasks, some limitations remain. The reliance on pre-training with Q1 data means that domain adaptation might be required for different survey depths or observational conditions. Additionally, the current classification thresholds, and sample selection, could introduce selection biases, necessitating further refinement. Future work should explore strategies to mitigate these biases and improve generalisation to different datasets.

Acknowledgments

The Euclid Consortium acknowledges the European Space Agency and a number of agencies and institutes that have supported the development of Euclid, in particular the Agenzia Spaziale Italiana, the Austrian Forschungsförderungsgesellschaft funded through BMK, the Belgian Science Policy, the Canadian Euclid Consortium, the Deutsches Zentrum für Luft- und Raumfahrt, the DTU Space and the Niels Bohr Institute in Denmark, the French Centre National d’Etudes Spatiales, the Fundação para a Ciência e a Tecnologia, the Hungarian Academy of Sciences, the Ministerio de Ciencia, Innovación y Universidades, the National Aeronautics and Space Administration, the National Astronomical Observatory of Japan, the Netherlandse Onderzoekschool Voor Astronomie, the Norwegian Space Agency, the Research Council of Finland, the Romanian Space Agency, the State Secretariat for Education, Research, and Innovation (SERI) at the Swiss Space Office (SSO), and the United Kingdom Space Agency. A complete and detailed list is available on the Euclid web site (www.euclid-ec.org). This work has made use of the Euclid Quick Release Q1 data from the Euclid mission of the European Space Agency (ESA), 2025, https://doi.org/10.57780/esa-2853f3b. Based on data from UNIONS, a scientific collaboration using three Hawaii-based telescopes: CFHT, Pan-STARRS, and Subaru www.skysurvey.cc. Based on data from the Dark Energy Camera (DECam) on the Blanco 4-m Telescope at CTIO in Chile https://www.darkenergysurvey.org. DESI construction and operations is managed by the Lawrence Berkeley National Laboratory. This research is supported by the U.S. Department of Energy, Office of Science, Office of High-Energy Physics, under Contract No. DE–AC02–05CH11231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract. Additional support for DESI is provided by the U.S. National Science Foundation, Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF’s National Optical-Infrared Astronomy Research Laboratory; the Science and Technology Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simons Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Science and Technology of Mexico (CONACYT); the Ministry of Science and Innovation of Spain, and by the DESI Member Institutions. The DESI collaboration is honored to be permitted to conduct astronomical research on Iolkam Du’ag (Kitt Peak), a mountain with particular significance to the Tohono O’odham Nation. MHC and MS acknowledge support from the State Research Agency (AEIMCINN) of the Spanish Ministry of Science and Innovation under the grants “Galaxy Evolution with Artificial Intelligence” with reference PGC2018-100852-A-I00 and “BASALT” with reference PID2021-126838NBI00. MS acknowledges also support by the Polish National Agency for Academic Exchange (Bekker grant BPN/BEK/2021/1/00298/DEC/1). HDS acknowledges financial support by RyC2022-030469-I grant, funded by MCIN/AEI/10.13039/501100011033 and FSE+.

References

Appendix A: Hyperparameters, computations, and carbon emissions

The training of deep-learning models requires considerable energy, which can contribute to carbon emissions. To counteract further emission from unnecessary retraining we follow the recommendations of (Strubell et al. 2019) and make available our model weights and inference code in the Hugging Face repository⁴. While this study made use of the 100% renewable energy-powered ITER Teide HPC cluster (Mampaso et al. 2022) for pre-training, for the sake of transparency we estimate and show our energy usage during the pre-training of AstroPT here: AstroPT-90M takes 2 hours to train on 4xA100-40G NVIDIA GPUs, which corresponds to 2 kWh of energy use according to the Machine Learning CO2 Impact Calculator (https://mlco2.github.io/impact/).

Appendix B: AstroPT embeddings for VIS and VIS+NISP imaging

Figure B.1 presents a visualisation of the AstroPT embeddings trained on the VIS (left) and combined VIS and NISP data (right). This plot demonstrates how the inclusion of both optical and infrared wavelengths influences the organisation and clustering of different galaxy types and stars in the embedding space.

Fig. B.1. UMAP visualisations of the embeddings from AstroPT trained on VIS (left) and VIS+NISP (right) images with example cut-outs, see text for more details.

In Fig. B.2, we show typical anomalies found with LOF (upper panel) and IF (bottom panel) for the VIS+NISP embedding. As can be observed, anomalies consist of images that include multiple objects within the cut-outs.

Fig. B.2. Example VIS and NISP cut-outs of anomaly detections found on VIS+NISP embeddings with IF (upper panel) and LOF (bottom panel).

Appendix C: Star-QSO-galaxy separation

Figure C.1 shows the correlation of the embeddings with the source type (unknown, galaxy, star, QSO) derived from the spectroscopic pipeline. Clearly, the outlier regions of embeddings are occupied by stars and unknown sources, but stars are also found over all regions (see Fig. C.1). Galaxies, QSOs, and stars tend to occupy a broad distribution. Adding NISP data sharpens the separation of unknown objects where a long tail gathers the invalid NISP data. Incorporating SEDs further enhance the separation of unknown sources from the main space.

Fig. C.1. UMAP visualisations of the self-supervised embeddings from AstroPT coloured by the source type (QSO, galaxy, star, or unknown). The left panel corresponds to VIS-only embeddings, the centre panel includes both VIS and NISP data, and the right panel includes VIS, NISP, and SED data.

Fig. C.2. UMAP visualisations of the self-supervised embeddings from AstroPT. Top: Coloured by the stars. Bottom: Coloured by QSO. Left panels correspond to VIS-only embeddings, centre panels include both VIS, and NISP data, and the right panels include VIS, NISP, and SED data.

Appendix D: Quality of the embeddings: Physical properties

Figures D.1, and D.2 highlight the relationship between the latent space and physical properties, such as stellar mass, and redshift. There is a clear connection between morphological trends (see Fig. 8) and stellar masses. Early-type galaxies, which are typically more massive, occupy regions of the UMAP space associated with higher stellar masses. In contrast, spiral galaxies, known for their lower stellar masses, cluster in distinct, separate regions. This differentiation is especially pronounced when embeddings are trained with VIS+NISP data, suggesting that the additional NISP features help to better constrain stellar mass distributions. This is further enhanced when the SEDs are included in the training. The Euclid photometric redshift shows no strong correlation with the embeddings, although there is a subtle trend of higher stellar masses being linked to slightly higher redshifts, especially for VIS+NIS+SED, consistent with the evolutionary connection between galaxy mass and formation epoch. The trend with the redshift is clearer when the spectroscopic redshift from DESI is used. These results underscore the utility of self-supervised representations in capturing key physical galaxy properties, particularly with multi-band data.

Fig. D.1. UMAP visualisation of self-supervised embeddings, highlighting correlations with stellar mass and photometric redshift. The left panels show the distribution of galaxies when AstroPT is trained on VIS images, while the centre panels represent the training with VIS+NISP images, and the right panels correspond to training on VIS, NISP images, and SEDs. The top UMAP visualisations are colour-coded by stellar mass from Euclid official pipeline (top). The bottom UMAP visualisations are colour-coded by Euclid official photometric redshifts.

Fig. D.2. UMAP visualisation of self-supervised embeddings, highlighting correlations with stellar mass and photometric redshift. The left panels show the distribution of galaxies when AstroPT is trained on VIS images, while the centre panels represent the training with VIS+NISP images, and the right panels correspond to training on VIS, NISP images, and SEDs. The left UMAP visualisations are colour-coded by stellar mass from DESI VAC (Siudek et al. 2024). The right UMAP visualisations are colour-coded by the DESI spectroscopic redshift (DESI Collaboration 2024b) available for about 5% of the sample.

Appendix E: Accuracy of predictions

Table E.1, E.2, and E.3 provide a detailed summary of the performance metrics for morphology classification, photo-z, and stellar mass predictions, respectively, across different models and data configurations (see Sect. 3.2). Metrics are reported for various percentages of labelled data used during training. These tables complements Figs. 10, 12, 14 by offering a more detailed view of the performance differences between the models. MLP model consistently achieves the best overall metrics, particularly when leveraging self-supervised embeddings with even a minimal set of labels. The linear model also demonstrates competitive performance, and serves as a robust baseline for comparison.

All Tables

All Figures

Fig. 1. Scaling laws for foundation models in astronomy compared to the GPT series. The number of tokens is defined as subword units in text (GPT models); pairs of augmented images (contrastive learning models); or a number of patches (AstroPT model). The projected scaling of future models incorporating Euclid data in upcoming data releases are marked with green stars. Our current model, trained on Q1 Euclid data, is represented by a red star. The models from the literature are referenced as follows: AH21 (Hayat et al. 2021); S21 (Sarmiento et al. 2021), GS21 (Stein et al. 2021); GMS24 (Martínez-Solaeche et al. 2024); P24 (Parker et al. 2024); and MS24 (Smith et al. 2024).

In the text

	Fig. 2. Stellar mass-redshift relation for the Q1 sample (in cyan), the sample used in this work (in navy blue), and our sample with DESI spectroscopic information (in red). The inset plots show the residuals of redshift and stellar mass differences between the Euclid pipeline and DESI, respectively.
In the text

	Fig. 3. Distribution of Euclid photometry for the sample used in this paper.
In the text

	Fig. 4. Representative SEDs of galaxies from the Q1 EDF-N (top) and EDF-S fields (bottom). The SED includes multi-wavelength photometric data spanning the ultraviolet to the near-infrared listed in Table 1.
In the text

	Fig. 5. AstroPT’s training objective is predicting the next patch in a sequence of galaxy image (and other modalities’) patches. On the left we show the order of galaxy patches used in this work, and on the right we show how each patch is processed by AstroPT.
In the text

	Fig. 6. UMAP visualisations of the embeddings from AstroPT trained on VIS + NISP + SEDs with example cut-outs and SEDs, see Sect. 4.1 for more details.
In the text

	Fig. 7. Top: UMAP visualisations of the self-supervised embeddings from AstroPT trained on VIS (left), VIS + NISP (centre), and VIS + NISP + SED (right) images coloured by the detection quality flag. This flag allows one to identify sources with contaminants, e.g. proximity star (DET_QUALITY_FLAG ≥4). Bottom: Example VIS + NISP cut-outs of the outliers in the representative space.
In the text

	Fig. 8. UMAP visualisations of the self-supervised embeddings from AstroPT trained on VIS (left), VIS + NISP (centre), and VIS + NISP + SED (right) coloured by morphological parameters derived from the GZ catalogue (Euclid Collaboration: Walmsley et al. 2026b). Representations show clear clustering of smooth, non-spiral, and non-smooth, spiral galaxies.
In the text

Fig. 9. UMAP visualisations of the self-supervised embeddings from AstroPt coloured by morphological parameters derived from the GZ catalogue (Euclid Collaboration: Walmsley et al. 2026b). Left: Representations from VIS images alone, showing a distinct region for edge-on galaxies. Middle: Representations incorporating both VIS and NISP data, where the separation of edge-on galaxies becomes less distinct. Right: Representations incorporating VIS, NISP, and SED data. Bottom: Example cut-outs and SEDs of the edge-on galaxies.

In the text

	Fig. 10. Comparison of the accuracy, precision, and recall as functions of the percentage of labels used for training, for early-type/late-type classification. Results are shown for the VIS (left), VIS + NISP (centre), and VIS + NISP + SED configurations.
In the text

	Fig. 11. Comparison of photo-z from Euclid official NNPZ pipeline (top) and predictions from the MLP model trained on AstroPT embeddings (bottom) versus DESI spec-z. A solid black line represents the 1:1 relation, while dotted lines indicate the outlier thresholds of 0.15.
In the text

	Fig. 12. Comparison of the bias, NMAD, and outlier fraction as functions of the percentage of labels used for training, for true redshift (DESI spec-z) versus Euclid photo-z predictions. Results are shown for the VIS (left), VIS + NISP (middle), and VIS + NISP + SED (right) configurations.
In the text

	Fig. 13. Comparison of stellar mass from Euclid official NNPZ pipeline (top) and predictions from MLP model trained on AstroPT embeddings (bottom) versus DESI stellar masses from Siudek et al. (2024). A solid black line represents the 1:1 relation, while dotted lines indicate the outlier thresholds of 0.25 dex.
In the text

	Fig. 14. Comparison of the NMAD and outlier fraction as functions of the percentage of labels used for training, for DESI stellar masses from Siudek et al. (2024) versus Euclid predictions. Results are shown for the VIS (left) and VIS + NISP (right) configurations.
In the text

	Fig. 15. Example VIS cut-outs of anomaly detections found with LOF (upper panel) and IF (bottom panel) on VIS embeddings.
In the text

	Fig. 16. UMAP visualisations of the self-supervised embeddings trained on VIS images with anomalies detected with LOF, and IF algorithms.
In the text

	Fig. 17. Anomalies detected with the IF algorithm in the VIS + NISP + SED embedding. For each galaxy we show VIS and NSIP images and the SED. Fluxes are normalised to the Euclid VIS value.
In the text

	Fig. 18. Distribution of merging_merger parameter for similar objects computed with VIS embedding and general galaxy distribution.
In the text

	Fig. B.1. UMAP visualisations of the embeddings from AstroPT trained on VIS (left) and VIS+NISP (right) images with example cut-outs, see text for more details.
In the text

	Fig. B.2. Example VIS and NISP cut-outs of anomaly detections found on VIS+NISP embeddings with IF (upper panel) and LOF (bottom panel).
In the text

	Fig. C.1. UMAP visualisations of the self-supervised embeddings from AstroPT coloured by the source type (QSO, galaxy, star, or unknown). The left panel corresponds to VIS-only embeddings, the centre panel includes both VIS and NISP data, and the right panel includes VIS, NISP, and SED data.
In the text

	Fig. C.2. UMAP visualisations of the self-supervised embeddings from AstroPT. Top: Coloured by the stars. Bottom: Coloured by QSO. Left panels correspond to VIS-only embeddings, centre panels include both VIS, and NISP data, and the right panels include VIS, NISP, and SED data.
In the text

Fig. D.1. UMAP visualisation of self-supervised embeddings, highlighting correlations with stellar mass and photometric redshift. The left panels show the distribution of galaxies when AstroPT is trained on VIS images, while the centre panels represent the training with VIS+NISP images, and the right panels correspond to training on VIS, NISP images, and SEDs. The top UMAP visualisations are colour-coded by stellar mass from Euclid official pipeline (top). The bottom UMAP visualisations are colour-coded by Euclid official photometric redshifts.

In the text

Fig. D.2. UMAP visualisation of self-supervised embeddings, highlighting correlations with stellar mass and photometric redshift. The left panels show the distribution of galaxies when AstroPT is trained on VIS images, while the centre panels represent the training with VIS+NISP images, and the right panels correspond to training on VIS, NISP images, and SEDs. The left UMAP visualisations are colour-coded by stellar mass from DESI VAC (Siudek et al. 2024). The right UMAP visualisations are colour-coded by the DESI spectroscopic redshift (DESI Collaboration 2024b) available for about 5% of the sample.

In the text