© The Authors 2023

1 Introduction

Over the last few decades, multi-band wide imaging surveys have been driving discoveries, demonstrating the power of large datasets to enable precision cosmology. Obtaining precise photometric redshifts is crucial for exploiting large galaxy imaging surveys (Salvato et al. 2019), and they are a limiting factor in the accuracy of cosmology measurements that use galaxies (Knox et al. 2006). Current and upcoming imaging surveys such as the Dark Energy Survey (DES; The Dark Energy Survey Collaboration 2005), the Kilo-Degree Survey (KiDS; de Jong et al. 2013), Euclid (Laureijs et al. 2011), and the Rubin Observatory Legacy Survey of Space and Time (LSST; LSST Science Collaboration 2009) critically depend on robust redshift estimates to obtain reliable science results (Blake & Bridle 2005).

With larger imaging surveys (as the quality and number of photometric observations increase), the photo-z performance requirements, both in terms of bias and precision, have become increasingly stringent in response to a need to reduce the uncertainties in the science measurements. As an example, the analysis of the first year of DES data (DES Y1) had a photo-z precision requirement ${\sigma }_{z_{\text{p}}}{{}_{-}}_{z_{\text{s}}}<0.12$ (Sánchez et al. 2014), with ${\sigma }_{z_{\text{p}}}{{}_{-}}_{z_{\text{s}}}$ being the standard deviation of the residuals between the photometric redshift, z_p, and the spectroscopic redshift, z_s (as a proxy of the true redshift). In order to exploit the constraining power of LSST, it is required that the mean fractional photo-z bias ∣〈∆z〉∣ < 0.003, with ∆z := (z_p – z_s)/(1 + z_s), and the scaled photo-z scatter σ_∆z < 0.02 (Schmidt et al. 2020), which corresponds to photo-zs that are around three times more precise than in DES Y1. Similarly, for Euclid, the scaled photo-z bias is required to be below 0.002 and σ_∆z < 0.05 (Laureijs et al. 2011).

The increasingly stringent requirements on the photo-z measurements have triggered extensive investigation efforts dedicated to improving photo-z estimation methodologies. Therefore, there are many different photo-z codes, which can be classified into two main approaches: the so-called template-fitting methods (e.g. LePhare: Arnouts & Ilbert 2011; BPZ: Benítez 2011; and ZEBRA: Feldmann et al. 2006) and data-driven (machine-learning) methods (e.g. ANNz: Collister & Lahav 2004; ANNz2: Sadeh et al. 2016; tpz: Carrasco Kind & Brunner 2013, Skynet: Bonnett 2015, and spiderZ: Jones & Singal 2017). These methods commonly only use the measured photometry to produce photo-z estimates. Furthermore, there is a wealth of techniques for improving the photo-z performance, such as including galaxy morphology (Soo et al. 2018), using Gaussian processes (Gomes et al. 2018; Soo et al. 2021), implementing ‘pseudo-labelling’ semi-supervised approaches to determine the underlying structure of the data (Humphrey et al., in prep.), and directly predicting the photo-z from astronomical images (Pasquet-Itam & Pasquet 2018; Pasquet et al. 2019; Chong & Yang 2019).

The broadband photo-z performance is limited by the resolution and the wavelength coverage provided by the photometric filters. Narrow-band photometric surveys are in between spectroscopy and broadband photometry (Benitez et al. 2014; Martí et al. 2014; Eriksen et al. 2019). They are imaging surveys with a higher wavelength resolution than broadband surveys, but they typically cover smaller sky areas due to the increased telescope time needed to cover the same wavelength range. In this paper we use multi-task learning (MTL; Caruana 1997) and narrowband data to improve broadband photo-z estimates. Multi-task learning is a machine-learning methodology in which the model benefits from predicting multiple related tasks together, for example a network that predicts the animal type (e.g. elephant, dog, dolphin, or unicorn) and its weight. In this example, the network learns the correlations between each animal class and how heavy they are (e.g. an elephant is heavier than a dog), and such correlations are used to improve the final predictions in both tasks.

In astronomy, data that could be helpful for improving the photo-z performance often exist, for example photometry in several bands. However, such data are not always available for the complete wide field, preventing us from using it. With MTL, we can utilise these data to improve the photo-z predictions without explicitly providing them as input. Particularly, we implemented an MTL neural network that predicts the photo-z and the narrowband photometry of a galaxy from its broadband photometry. The narrow-band data are used to provide ground-truth labels to train the auxiliary task of reconstructing the narrow-band photometry (Liebel & Körner 2018). Therefore, we only need it to train the network, and we can evaluate the photo-z of any galaxy with only its broadband photometry. In this way, the data available in certain fields can be exploited to improve the photo-z estimations in other fields.

We tested the method with data from the Physics of the Accelerating Universe Survey (PAUS). It is a narrow-band imaging survey carried out with the PAUCam instrument (Castander et al. 2012; Padilla et al. 2016, 2019), a camera with 40 narrow bands that cover the optical spectrum (Casas et al. 2016). The method could also be applied to other narrow-band surveys such as the Javalambre Physics of the Accelerating Universe Survey (J-PAS; Benitez et al. 2014).

The paper is structured as follows. In Sect. 2 we present the data used throughout the paper. Section 3 introduces MTL and the method developed and tested in this work. In Sect. 4 we show the performance of the photo-z method in the COSMOS field, including bias, scatter, outliers, and the photo-z distributions. The performance on a deeper galaxy sample is tested in Sect. 5 using simulated galaxies. Finally, we use self-organising maps (SOMs) to explore the photo-z distribution of COSMOS galaxies in colour space (Sect. 6) and to gain a better understanding of the underlying mechanism of our method (Sect. 7).

2 Data

In this section we present the PAUS data (Sect. 2.1) and the photometric redshift galaxy sample (Sect. 2.2). The broadband data and the spectroscopic sample are introduced in Sect. 2.3 and Sect. 2.4, respectively, while Sect. 2.5 shows the galaxy simulations used in the paper.

2.1 PAUS data

PAUS data are taken at the William Herschel Telescope (WHT), at the Observatorio del Roque de los Muchachos in La Palma (Canary Islands). Images are taken with the PAUCam instrument (Castander et al. 2012; Padilla et al. 2019), an optical camera equipped with 40 narrow bands covering a wavelength range from 4500 to 8500 Å (Casas et al. 2016). The narrow-band filters have a 130 Å full width at half maximum and a separation between consecutive bands of 100 Å. They are mounted in five trays with eight filters per tray that can be exchanged and placed in front of the CCDs. The narrow-band filter set effectively provides a high-resolution photometric spectrum (R ~ 50). This allows PAUS to measure high-precision photo-zs to faint magnitudes (i_AB < 23) while covering a large sky area (Martí et al. 2014). In this work we use the full pass-band filter information¹.

With a template-fitting algorithm, PAUS reaches a photo-z precision σ_z/(1 + z) = 0.0035 for the best 50% of the sample (Eriksen et al. 2019). Similar precision is obtained with Delight (Soo et al. 2021), a hybrid template-machine-learning photometric redshift algorithm that uses Gaussian processes. The PAUS photo-z precision was improved further with a deep-learning algorithm that reduces the scatter by 50% compared to the template-fitting method in Eriksen et al. (2020). Furthermore, with a combination of PAUS narrow bands and 26 broad and intermediate bands covering the UV, visible, and near infrared spectral range, Alarcon et al. (2021) presented an unprecedented precise photo-z catalogue for COSMOS (Scoville et al. 2007) with σ_z/(1 + z) = 0.0049 for galaxies with i_AB < 23. The excellent PAUS photo-z precision enables studies of intrinsic galaxy alignments and three-dimensional galaxy clustering (Johnston et al. 2021a), as well as determining galaxy properties (Tortorelli et al. 2021) and measuring the D4000Å spectral break (Renard et al., in prep.).

PAUS has been observing since the 2015B semester, and as of 2021B, PAUS has taken data during 160 nights. It partially covers the Canada-France-Hawaii Telescope Legacy Survey (CFHTLS) fields² W1, W2, and W3, as well as the full COSMOS field³. In the W2 field, so far PAUS has observed in the overlapping region with the GAMA 9-hour field⁴ (G09). Currently, PAUS data have a 40 narrow-band coverage of 10 deg² in each of W1 and G09, 20 deg² in W3, and 2 deg² in COSMOS. The PAUS data are stored at the Port d’Informació Científica (PIC), where the data are processed and distributed (Tonello et al. 2019). This paper uses data from the COSMOS field (Scoville et al. 2007), which were specifically taken in the semesters 2015B, 2016A, 2016B, and 2017B. The complete PAUS photometric catalogue in COSMOS comprises 64 476 galaxies to i_AB< 23 in 40 narrow-band filters. This corresponds to approximately 12.5 million galaxy observations (5 observations per galaxy and narrow-band filter).

Two methods for extracting the galaxy photometry have been developed for PAUS: a forced aperture algorithm (MEMBA) and a deep-learning-based pipeline (Lumos; Cabayol-Garcia et al. 2020; Cabayol et al. 2021). In this study we have found that the resulting photo-z performance with both photometric approaches is very similar. In the COSMOS field, the parent detection catalogue is provided by Laigle et al. (2016) and the photometry calibration is relative to the Sloan Digital Sky Survey (SDSS) stars (Castander et al., in prep.). A brief description of the photometric calibration can be found in Eriksen et al. (2019).

2.2 tometric redshift sample

Throughout the paper we also use the high-precision photometric redshifts from Alarcon et al. (2021, PAUS+COSMOS hereafter). They were estimated with a template-fitting method modelling the spectral energy distributions (SEDs) as a linear combination of emission line and continuum templates to then compute the Bayes evidence by integrating over the linear combinations. In addition to the PAUS narrow bands, the PAUS+COSMOS catalogue uses 26 broad and intermediate bands covering the UV, visible, and near-infrared spectrum (see Sect. 2 in Alarcon et al. 2021, for more details). The PAUS+COSMOS photo-zs reach a precision of σ_z/(1 + z) = 0.0036 and σ_z/(1 + z) = 0.0049 for galaxies at i_AB 21 and i_AB < 23, respectively. These photo-zs are more precise and less biased than those from Laigle et al. (2016), which use a combination of 30 broad-, intermediate-, and narrow-band filters.

2.3 Broadband data

The broadband data used in this paper are from Laigle et al. (2016, COSMOS2015 hereafter), which includes the u-band from the Canada-France-Hawaii Telescope (CFHT)/MegaCam and the Subaru BVriz filters. We carry out a spatial matching of COSMOS2015 and PAUS galaxies within 1′′. Then, we apply a cut on magnitude i_AB < 23 and on redshift z < 1.5, which results in a catalogue with around 33 000 galaxies of which approximately 9000 have spectroscopic redshifts. The redshift cut is prompted by the photo-z distribution in the PAUS+COSMOS catalogue, with very few galaxies with z > 1.5 (Fig. 1).

Fig. 1 Redshift distributions for the COSMOS spectroscopic sample (red line) and the full (spectroscopic and photo-z) COSMOS sample.

2.4 Spectroscopic galaxy sample

To train the neural network, one needs a galaxy catalogue with known redshifts. We used the zCOSMOS Data Release (DR) 3 bright spectroscopic data (Lilly et al. 2007), which cover 1.7 deg² of the COSMOS field. The catalogue covers a magnitude range of 15 < i_AB < 23 and a redshift range of 0.1 < z < 1.2. We only keep redshifts with a confidence class (conƒ) of 3 < conƒ < 5, which leads to a catalogue with ~9400 galaxies. We extended the spectroscopic sample with a compilation of 2693 redshifts from Alarcon et al. (2021). This compilation includes redshifts from C3R2 DR1 and DR2 (Masters et al. 2017, 2019), 2dF (Colless et al. 2001), DEIMOS (Hasinger et al. 2018), FMOS (Kashino et al. 2019), LRIS (Lee et al. 2018), MOSFIRE (Kriek et al. 2015), MUSE (Urrutia et al. 2019), Magellan (Calabrò et al. 2018), and VIS3COS (Paulino-Afonso et al. 2018), with a quality cut to keep only those objects with a reliable measurement.

Figure 1 shows the redshift distribution of the COSMOS spectroscopic sample (red) and the full PAUS sample in the COSMOS field (blue), where the redshift is defined as the spectroscopic redshift if this is available and as the PAUS+COSMOS photo-z otherwise. Including the PAUS+COSMOS photo-z is particularly relevant for galaxies with z > 1 , where there are very few spectroscopic measurements.

2.5 Galaxy mocks

In Sect. 5 we also use the Flagship galaxy simulations described in Castander et al. (in prep.). The Flagship galaxy catalogue has been developed to study the performance of the Euclid mission. The mock catalogue populates the halos detected in the Euclid Flagship N-body simulation (Potter et al. 2017), which is a large two trillion particles simulation on a box of 3780 h⁻¹Mpc, and a mass resolution of m_p = 2.4 × 10⁹ h⁻¹ M_⊙. The N-body simulation uses a cosmological model with parameters similar to the Planck 2015 cosmology (Planck Collaboration XIII 2016). Halos are identified with the ROCKSTAR halo finder (Behroozi et al. 2013). Galaxies are assigned to the halos using a hybrid halo occupation distribution and abundance matching technique similar to the one used for the MICE catalogues described in Carretero et al. (2014). Galaxies are divided into central and satellites. Each halo contains a central and a number of satellites given by their halo occupation. Galaxies are also tagged in three colour types: blue, green, and red. The relative abundance of central and satellites as a function of colour type and absolute magnitude is constrained by the observed colour-magnitude distribution and the clustering as a function of colour at low redshift. At higher redshift only observed colour distributions are used. Each galaxy is assigned a SED, including its extinction, from the COSMOS SED library (e.g. Ilbert et al. 2013), which includes SED templates from Polletta et al. (2007) and additional blue templates from Bruzual & Charlot (2003). In order to have a more continuous distribution of galaxy magnitudes and colours, the SED assigned to each galaxy is a linear combination of two consecutive templates in the COSMOS template library. Emission lines are then added to the SED of each galaxy. The Hα flux is computed from the rest-frame ultra-violet flux of each galaxy template following Kennicutt (1998). The Hα fluxes are then re-adjusted to make them follow the Pozzetti et al. (2016) model 1 and model 3 distributions. The Hß flux is computed from the Hα flux assuming case B recombination (Osterbrock & Ferland 2006). The other emission line fluxes ([OII], [OIII], [NII], and [SII]) are computed following relations obtained from observed distributions. The emission line fluxes are added to the continuum assuming a Gaussian distribution of width given by the galaxy magnitude and the Faber-Jackson or Tully-Fisher relation. Finally, the SEDs containing the emission lines are convolved with the filter transmission curves to produce the expected observed fluxes. This prescription is followed to generate both broad- and narrow-band photometry. The Flagship catalogue is a property of the Euclid Consortium and is available at CosmoHub⁵ (Carretero et al. 2017; Tallada et al. 2020), a web application based on Hadoop to interactively distribute and explore massive cosmological datasets.

3 Multi-task neural network to improve broadband photo-zs

In this section we describe MTL (Sect. 3.1) and present the networks and training procedures used throughout the paper (Sect. 3.2).

3.1 Multi-task learning

Deep-learning algorithms consist of training a single or an ensemble of models to accurately perform a single task, for example predicting the redshift. Multi-task learning is a training methodology that aims to improve the performance on a single task by training the model on multiple related tasks simultaneously (Caruana 1997). One can think of MTL as a form of inductive transfer, where the knowledge that the network acquires from one task introduces an inductive bias to the model, making it prefer certain hypotheses over others. A simple pedagogical example is a network to classify cats and dogs. If we include a secondary task to classify the shape of the ears in, for example spiky or rounded, the network will make correlations between the ear shapes and the animal class, in such a way that the predicted ears shape will also affect the cat-dog classification. This kind of network has already been successfully applied in other fields, such as video processing (Song et al. 2020) and medical imaging (Moeskops et al. 2017), where in the latter case a single network is trained to segment six tissues in brain images, the pectoral muscle in breast images, and the coronary arteries. There are also successful implementations in astrophysics. Examples include, for example, Parks et al. (2018), which characterises the strong HI Lyα absorption in quasar spectra simultaneously predicting the presence of strong HI absorption and the corresponding redshift z_abs and the HI column density. Also, Cunha & Humphrey (2022) describe SHEEP, a machine-learning pipeline for the classification of galaxies, quasi-stellar objects, and stars from photometric data. Broadly speaking, there are two types of MTL-network architectures, called soft- and hard-parameter sharing (Zhang & Yang 2021). In the former, each task has its parameters, which are regularised to be similar amongst tasks. For the latter, the hidden layers of the network are shared between tasks, while keeping task-specific layers separate. Hard-parameter sharing is the most common MTL architecture and it is the one used in this paper.

Fig. 2 Top: baseline network architecture. The input contains five colours that propagate through six fully connected layers. Each layer is followed by a dropout layer, which is represented by a yellow-crossed circle. Bottom: MTL network. This builds on the baseline network and adds an extra output layer for the additional task of predicting the narrow-band photometry.

3.2 Model architecture and training procedures

In our analysis we used mixture density networks (MDNs) to predict the photo-z probability distribution as a linear combination of N independent Gaussians (D'Isanto & Polsterer 2018; Eriksen et al. 2020). The network predicts the mean and the standard deviation of N distributions, together with N additional mixing coefficients (α) weighting the relative importance of each Gaussian component to the combined probability distribution, so that ${\displaystyle {\sum }_{i=0}^{i=N}{\alpha }_{\text{i}}=1}$.

Figure 2 shows the two MDNs used in this paper, both of them predicting the photo-z probability distribution p(z) as the combination of three independent Gaussian distributions. The top panel presents the baseline network, a single-task network mapping the broadband photometry to the photometric redshifts. It concatenates six fully connected layers with parameters 5:300:500:1000:500:300:9, where the numbers correspond to the number of nodes in the layers. Therefore, the first contains five nodes, corresponding to the uBVriz broadband colours. The last layer consists of nine output parameters corresponding to the mean (z), the standard deviation (σ_i), and the mixing coefficients, α, of the three Gaussians building the p(z). Each layer is followed by a 2% dropout layer (Srivastava et al. 2014), a regularisation method in which several nodes are randomly ignored during the training phase.

The bottom panel in Fig. 2 represents the MTL network introduced in this paper, which includes the additional task of predicting the PAUS narrow-band photometry using a hard parameter-sharing architecture (Fig. 2). The core architecture is the same as that of the baseline network (upper panel) but this network contains an extra output layer for the additional task of predicting the narrow-band photometry.

The photo-z loss function of both networks is the negative log-likelihood: $${ℒ}_{\text{Z}}:={\displaystyle \sum _{i=1}^N\left[\log \left({\alpha }_i\right)-\frac{{\left(z_i-z_{\text{S}}\right)}^2}{{\sigma }_i^2}-2\log \left({\sigma }_i\right)\right]}.$$(1)

The ground-truth redshift labels are the spectroscopic redshifts (z_s) as defined in Sect. 2.4 and the summation is over the Gaussian components. For some training configurations, we also used high-precision photo-zs (Sect. 2.2) as ground-truth labels to extend the photo-z training sample beyond the spectroscopic sample.

The MTL network enables including information from the galaxy SED, while extending the training sample to galaxies without spectroscopic redshift but with narrow-band photometry. The two tasks share internal representations when predicting the photo-z and the narrow-band photometry simultaneously; thus, the non-spectroscopic galaxies indirectly affect the training of the photo-z prediction.

The training of the narrow-band is addressed with a least absolute deviation loss function, $${ℒ}_{\text{NB}}:=\frac{{\text{Σ}}_i\left|{\text{NB}}_i^{\text{pred}}-{\text{NB}}_i^{\text{obs}}\right|}{N-1},$$(2)

where ${\text{NB}}_i^{\text{pred}}$ and ${\text{NB}}_i^{\text{obs}}$ are the predicted and observed narrowband colours in the ith filter, respectively, and N is the number of narrow bands. We also tested other alternatives, for example the mean-squared error, but this was hindering the network’s convergence and we decided on the absolute-mean error. Another alternative was to predict the probability distribution of the narrow-band fluxes using a MDN as well, but this did not resulted in better photo-z estimations.

Consequently, there are the following two training methodologies. The first is z_s: This is the usual training that maps the broadband photometry to photo-z using spectroscopic redshifts as ground-truth redshifts and a negative log-likelihood loss function (Eq. (1)).

The second is z_s+NB: This methodology includes MTL. It maps the broadband photometry to photo-z and narrow-band photometry, and therefore the loss function is the mean of the combined negative log-likelihood loss (Eq. (1)) and narrow-band reconstruction (Eq. (2)) tasks for all galaxies (N) for which the loss is computed as $${ℒ}_{{\text{NB+z}}_{\text{s}}}:=\frac{1}{N}{\displaystyle \sum _{j=1}^N\left[{ℒ}_z^j+{ℒ}_{\text{NB}}^j\right]}.$$(3)

We only used galaxies with spectroscopic redshift to train the photo-z predictions, while all galaxies with narrow-band observations trained the narrow-band reconstruction. In general, one can also weight the two terms in the loss functions. Testing different values, we found the photo-z scatter to have a minimum in a wide range of values around equal weighting.

Furthermore, we considered two variants in the training procedure to explore the possibility of using high-precision photometric redshifts (Sect. 2.2) to train the networks: the first is z_s + z_PAUS: This is a variation of the z_s method. The training sample extends to galaxies having a high-precision photo-z estimate in the PAUS+COSMOS catalogue. For galaxies with spectroscopy, we use the spectroscopic redshift as ground-truth while for the rest of the training sample, the PAUS+COSMOS photo-z is used to train the network.

The second is z_s +NB+z_PAUS: This is a variation of the z_s+NB method, and it also extends the training sample with galaxies with a high-precision photo-z estimate in the PAUS+COSMOS catalogue. In contrast to the z_s+NB method, here all galaxies are used to train the photo-z prediction and the narrow-band photometry reconstruction. The ground-truth redshift labels are the spectroscopic redshifts if available and otherwise, the PAUS+COSMOS photo-z.

The networks are implemented in PyTorch (Paszke et al. 2017). All the training procedures use an Adam optimiser (Kingma & Ba 2015) for 100 epochs with an initial learning rate of 10⁻³ that reduces by a factor of ten every 50 epochs.

4 Photo-z performance in the COSMOS field

In this section we show the photo-z performance of our method on galaxies with i_AB < 23 and z < 1.5 in the COSMOS field. We study the effect that MTL has on the dispersion (Sect. 4.2) and the bias (Sect. 4.3) of the predicted photo-zs.

4.1 Photo-z performance metrics

To evaluate the accuracy and precision of the photo-z estimates, we define $$\text{Δ}z:=\left(z_{\text{p}}-z_{\text{t}}\right)/\left(1+z_{\text{t}}\right),$$(4)

where z_p and z_t are the mean predicted photo-z and the ground-truth redshift, respectively. The bias and the dispersion are defined as the median and σ₆₈ of ∆z, respectively, where we define σ₆₈ as $${\sigma }_{68}:=\frac{1}{2}\left[Q_{84}\left(\text{Δ}z\right)-Q_{16}\left(\text{Δ}z\right)\right],$$(5)

and Q₁₆(∆z), Q₈₄(∆z) are the 16th and 84th percentiles of the ∆z distribution. We also include the metric $${\sigma }_{\text{NMAD}}:=1.4826\times \text{median}\left[\left|\text{Δz}-\text{median}\left(\text{Δz}\right)\right|\right]$$(6)

used in the Euclid photo-z challenge paper (Desprez et al. 2020).

To evaluate the performance on the full COSMOS catalogue, we define the ground-truth redshift as the spectroscopic redshift if available and otherwise, as the PAUS+COSMOS photo-z (Sect. 2.2)⁶. If it is not specified by the method, our networks are trained with spectroscopic redshifts only. For the performance evaluation, however, the PAUS+COSMOS photo-zs are also used, but only to evaluate the photo-z of galaxies from the full COSMOS catalogue that do not have a spectroscopic redshift estimate. The predicted photo-zs are defined as the mean of the redshift probability distribution provided by the network (Sect. 3.2).

In order to estimate the photo-zs of the complete COSMOS catalogue, the networks are trained independently ten times with ~11 000 spectroscopic galaxies in each iteration, which roughly corresponds to 90% of the sample. Each network is used to evaluate the corresponding 10% of excluded galaxies in such a way that the ensemble of networks evaluates the full COSMOS catalogue.

Including MTL extends the training sample to about 40 000 galaxies, which corresponds approximately 3.5 times more galaxies than in the spectroscopic sample. In order to evaluate the full COSMOS sample, we trained the network seven independent times with 85% of the spectroscopic galaxies and 85% of the non-spectroscopic sample. This corresponds to around 11 000 galaxies with spectroscopy and 25 000 without. We ensured that the fraction of galaxies with spectroscopic redshifts in each iteration is similar by sampling without replacement the same number of spectroscopic galaxies in each iteration.

4.2 Photo-z dispersion

Table 1 presents the photo-z precision for the COSMOS spectroscopic sample and the complete COSMOS sample using the four different training procedures presented in Sect. 3.2. These results are presented in more detail in Fig. 3, which shows the photo-z dispersion in equally populated magnitude and redshift bins with the same four methodologies. The solid black line corresponds to the baseline network mapping broadband photometry to photo-z (method z_s in Sect. 3.2). This method is trained on the spectroscopic sample and provides a σ₆₈ = 0.020 for the full sample. These are quite precise and accurate redshifts compared to other broadband redshift estimates in the same field. In Hildebrandt et al. (2009), redshifts in the D2 CFHT deep field (Coupon et al. 2009), which overlaps with COSMOS, were estimated with the template-fitting code BPz (Benítez 2011) using the CFHT ugriz filter set. Their photo-z precision is σ₆₈ = 0.0498, while for the same galaxy sub-sample our network provides σ₆₈ = 0.0187. Here neither the methodology nor the input data are the same, but having these CFHT photo-z estimates as a reference improves our photo-z baseline network performance. Here neither the methodology nor the input data are the same, but the CFHT photo-zs are a reference to compare the performance of our baseline network with.

In Fig. 3 we show the MTL training (method z_s+NB in Sect. 3.2) that uses all galaxies with PAUS photometry to train the narrow-band reconstruction and only those with spectroscopy to train the photo-z prediction. This extends the training sample of the shared layers (see the bottom panel of Fig. 2) from around 12 000–30 000 galaxies. This method provides a precision of σ₆₈ = 0.0176, corresponding to a 13% improvement with respect to the baseline methodology (solid black line). Moreover, the additional PAUS galaxies for the narrow-band reconstruction loss includes a more homogeneous colour-space coverage in the training sample. In Sect. 6 we discuss the underlying mechanism that causes MTL with PAUS to improve the photo-zs.

The blue dotted line in Fig. 3 also corresponds to a direct mapping of the broadband photometry to photo-zs. However, in contrast to the solid black line, this case is trained on an extended sample including galaxies without spectroscopic redshifts (method z_s + z_PAUS in Sect. 3.2), for which the PAUS+COSMOS photo-z measurement is used as a ground-truth redshift label in the training. It shows a precision of σ₆₈ = 0.0168, which corresponds to a 18% improvement with respect to the baseline training.

The best photo-z performance is achieved combining MTL and photo-z data augmentation with PAUS+COSMOS data (method z_s+NB+z_PAUS in Sect. 3.2), which corresponds to the dotted green line in Fig. 3. This method gives a 22% improvement with respect to the baseline network, with a precision of σ₆₈ = 0.0163.

In addition to uncertainties due to limited sample size (sample variance), our findings could also be affected by the intrinsic galaxy distribution being different at different parts of the sky (cosmic variance). To ensure that our results are not due to imprinting cosmic variance from the training to the test field, we tested our methods on two independent and spatially separated fields. These two fields are ~2 deg² and contain galaxies from the Flagship simulations to i < 23 (Sect. 2.5). All the networks have been trained with 30 000 galaxies from the train field, and later evaluated on 20 000 galaxies different galaxies from the train and test fields (making sure that there is no overlap between the training and test galaxies in the train field). We estimated the sample variance of each of these fields by making 1000 bootstrap realisations and it is a sub-percent error. With the baseline z_s method, we obtain a 2% change in the photo-z precision between the train and test fields. Repeating the same test with the z_s+z_PAUS+NB method, we obtain a 3% change in the photo-z precision between fields, similar to the baseline case. These changes are much lower than the photo-z improvement we obtain with the MTL implementations (e.g. 22% for the z_s+z_PAUS+NB method), suggesting that such improvements are not caused by cosmic variance.

Fig. 3 Photo-z dispersion in equally populated magnitude differential bins to iAB < 23 (top) and equally spaced redshift bins to z < 1.5 (bottom). Each line corresponds to a different training procedure (see Sect. 3.2). While the black line corresponds to a baseline training, the other coloured lines include MTL (red and green lines) and data augmentation with photo-zs from the PAUS+COSMOS catalogue as ground-truth redshifts (blue and green lines).

Fig. 4 Photo-z bias in equally populated redshift bins (left) and equally populated i-band magnitude bins (right). The grey area corresponds to the Euclid photo-z bias requirement of ∆z = 0.002.

4.3 Photo-z bias and outlier rate

In this subsection we show the bias and the outlier rate for the photo-z predictions with the MTL networks and the baseline broadband network. The left panel in Fig. 4 shows the photo-z bias in equally populated redshift bins in the redshift range 0.1 < z_t < 1.5. We excluded the first redshift bin from the analysis since there are almost no galaxies with z_t < 0.07, which caused a bias at very low redshift⁷. The shaded area corresponds to the Euclid photo-z bias requirement <0.002 (Laureijs et al. 2011). Overall, for z_t < 1 the four methods presented in Sect. 3.2 are unbiased at the level of < 0.002. However, the z_s and the z_s+NB are still showing a trend within the 0.2% bias range, where low-redshift galaxies tend to be biased positive and the high-redshift ones, biased negative. In contrast, the z_s + z_PAUS and the z_s + z_PAUS+NB methods display a flatter bias with redshift.

At higher redshifts (z_t > 1), the baseline network photo-zs show a ~2% bias. Implementing MTL without increasing the photo-z training sample (solid red line) moderately improves the bias, but it is still far from the Euclid requirement. On the other hand, increasing the training sample with PAUS+COSMOS photo-zs produces a strong bias reduction (blue and green lines), decreasing the bias to ~l% for the highest-redshift galaxies. Figure 1 suggests that this is likely to be caused by a lack of training examples with spectroscopy at z_t > 1. The training sample at high-redshift is increased with the PAUS+COSMOS photo-zs.

The right panel of Fig. 4 shows the photo-z bias in equally populated i-band magnitude bins. Comparing to the right panel in the same figure, the bias binning in i-band magnitude is lower than that binning in redshift. For instance, binning in redshift the largest bias that the (z_s) method obtains is a ~2.5% for the highest-redshift galaxy bin. In contrast, binning in magnitude, galaxies in the faintest bin reach a 0.8% bias with the same method. This is partly because binning in magnitude, positive and negative biases in redshift cancel each other out. The photo-zs of galaxies with i < 22 are unbiased with the four methods. For galaxies with i > 22, the z_s method displays the largest bias, which is already reduced with the MTL method without data augmentation (z_s+NB). The methods extending the sample using the PAUS+COSMOS photo-zs (green and blue lines) reduce the bias of the z_s and the z_s+NB methods.

In this paper, we consider a galaxy to be an outlier if $$\left|z_{\text{p}}-z_{\text{t}}\right|/\left(1+z_{\text{t}}\right)>\mathrm{0.15.}$$(7)

In the spectroscopic sample, the baseline network yields 0.6% outliers, which reduces to 0.5% with the MTL using PAUS photometry, the training sample extension with PAUS+COSMOS photo-z, and the combination of both. The fraction of outliers in the PAUS sample in COSMOS is 1.1% for the baseline network and for the training sample extension with PAUS+COSMOS photo-zs (z_s + z_PAUS) The methodologies including MTL reduce the outlier fraction to 0.8% (z_s+NB) and 0.6% (z_s + z_PAUS+NB). While in the spectroscopic sample extending the training sample and including MTL have a similar effect on the outlier fraction, in the full PAUS sample in COSMOS MTL has a stronger impact. The MTL methodologies are particularly reducing the number of high-redshift photo-z outliers.

In order to validate the predicted photo-z probability distributions p(z), we use the probability integral transform (PIT; Dawid 1984; Gneiting et al. 2005; Bordoloi et al. 2010), which is defined as $$\text{PIT}\equiv {\displaystyle {\int }_{-\infty }^{z_{\text{true}}}\text{d}zp\left(z\right)},$$(8)

where z_true is the true redshift. When the p(z) faithfully represents the true redshift, the PIT distribution is the uniform distribution U[0,l]. Contrary, PIT histograms with peaks at the edges (i.e. around zero and unity) indicate the presence of outlier measurements. Also, PIT histograms more populated at the centres than on the edges denote over-dispersed probability distributions, while valleys at the centre of the histogram correspond to under-dispersed ones.

We measure the PIT distribution for the complete COSMOS sample using a combination of spectroscopic redshifts and high-precision photo-z as true redshift. Figure 5 shows the PIT distributions for the p(z) measured with the baseline z_s method (black line), the MTL method (dashed red line), extending the training sample with high-precision photo-z (dotted blue line), and combining the training sample augmentations and MTL (green line). In all cases, the PIT distribution is approximately a U[0,l] distribution, indicating that our networks predict robust probability distributions with reliable uncertainty measurements. The baseline and the z_s+NB methods display peaks on the edges of the distribution corresponding to outliers in the probability distributions. These peaks are reduced with the two methods using PAUS+COSMOS photo-zs in the training sample.

Fig. 5 PIT distribution for the COSMOS photo-zs predicted with the baseline zs method (black), the zs+NB method (red), the zs+zPAUS method (blue), and the zs+zPAUS+NB method (green). Including the PAUS+COSMOS photo-zs in the training reduces the number of outliers on the edges of the distribution.

5 Photo-z performance on deeper galaxy simulations

So far, all the networks have been trained and evaluated on samples within the same magnitude range i_AB < 23 (see Sect. 4). However, if the MTL network developed in this paper aims to improve the photo-z estimates of future deeper broadband surveys such as Euclid or LSST, the photo-z improvement it provides must hold for fainter galaxies. In the case of Euclid, observations will reach a limiting magnitude of 24.5 for the VIS instrument (Cropper et al. 2012; Amiaux et al. 2012) with 10 σ depth for extended sources, which corresponds to a similar depth in the i-band filter. Rubin will observe to a single exposure depth of i_AB ~ 24.5 and a co-added survey depth of i_AB ~ 27.5 (Ivezić et al. 2019), where the depth in the r band and the i band are also similar.

Currently, there are no PAUS measurements beyond i_AB = 23, thus limiting the magnitude range of the MTL training sample. Although observing deeper with PAUS is technically feasible, it would require considerably more observing time. Therefore, the MTL network must provide reliable photo-z predictions for deep data samples, while it is trained on a shallower data sample. Nevertheless, we note that this problem is not exclusive to our MTL network, but it affects all photo-z machine-learning algorithms. These are usually trained on relatively shallow spectroscopic samples and used to predict the photo-zs for much deeper data samples (Masters et al. 2017).

In this section we explore how the MTL network performs for deep samples (i_AB < 25), while the training is limited to galaxies with i_AB < 23 using Flagship simulated galaxy mocks (see Sect. 2.5). The broad bands used for this test are the CFHT u band, the griz bands from DECam (Honscheid & DePoy 2008), and the Euclid Near-infrared spectrometer and photometer (NISP) near-infrared bands H_E, J_E and Y_E (Euclid Collaboration 2022)⁸. These are not the same bands that were used in the tests of the COSMOS field (see Sects. 2.3 and 4), but these bands were chosen to demonstrate the potential benefits for the Euclid photo-z estimation.

We trained the four methods presented in Sect. 3.2 on a sample with 10 000 spectroscopic galaxies, which are augmented to 30 000 with PAUS-like galaxies without spectroscopic redshifts and limited to i_AB < 23. These numbers were chosen to approximately match the number of spectroscopic and PAUS-like galaxies in the COSMOS field (see Sect. 4). To simulate the performance of the approaches that extend the training sample with high-precision photo-zs (methods z_s+z_PAUS and z_s+NB+z_PAUS in Sect. 3.1), we added a scatter to the true redshifts of the PAUS-like simulated galaxies, so that the precision resembles that of the PAUS+COSMOS photo-zs.

The left panels in Fig. 6 show the photo-z bias of 30 000 simulated test galaxies to magnitude i_AB < 25 in equally populated magnitude bins (top) and in equally spaced redshift bins (bottom). The shaded areas correspond to the Euclid photo-z requirement of ∆z < 0.002. We obtain a larger bias than the Euclid requirement with all methods, although those including MTL reduce the bias of fainter galaxies. Although we are not meeting the Euclid bias requirement, our aim is to advance the usage of machine-learning photo-z developing novel methodology, rather than providing the final pipeline. We hope the improvement and ideas seen in this paper can be helpful for further development of Euclid machine-learning algorithms.

The right panels in Fig. 6 show the photo-z precision for the same 30 000 simulated test galaxies to magnitude i_AB < 25 in magnitude (top) and redshift (bottom) bins. The baseline network (black thick line) achieves an overall precision of σ₆₈ = 0.076, which increases to σ₆₈ = 0.085 for galaxies with i_AB > 23. Training using photo-zs but without MTL (z_s+z_PAUS, dotted blue line) improves the precision to σ₆₈ = 0.0654 and σ₆₈ = 0.080 for galaxies with i_AB > 23. With z_s+NB, the overall precision is σ₆₈ = 0.067, which degrades to σ₆₈ = 0.082 for galaxies with i_AB > 23. Finally, combining MTL and the photo-z data augmentation (z_s+NB+z_PAUS, solid green line) provides the best photo-z performance with σ₆₈ = 0.065 for the full sample, which increases to σ₆₈ = 0.079 for galaxies with i_AB > 23 The best performance in terms of bias and precision is obtained with the z_s+z_PAUS+NB method, which provides 16% more precise photo-zs than the baseline network for galaxies with i_AB < 25, which increases to 20% for i_AB < 24.

Fig. 6 Top: photo-z bias (left) and precision (right) in equally populated magnitude bins. Bottom: photo-z bias (left) and dispersion (right) in equally spaced spectroscopic redshift bins. The shaded grey areas indicate ∆z > 0.002, corresponding to the Euclid requirement for the photo-z bias. All plots are for 30 000 Flagship test galaxies with magnitudes iAB < 24.5 for the methods presented in Sect. 3.2. The training sample contains around 15 000 spectroscopic galaxies, extended to 30 000 with PAUS-like galaxies without spectroscopy, all of them to iAB < 23.

6 Photo-z in colour space

While the effect of increasing the training sample in machine-learning algorithms has been extensively studied, we still need to understand why MTL with narrow-band photometry improves the photo-z estimates. In this section we use SOMs to explore the COSMOS photo-z performance in colour space (Sect. 6.1). Furthermore, in Sects. 6.2 and 6.3 we identify colour-space regions with strong emission lines where the broadband photo-zs precision is lower.

6.1 MTL photo-z in colour space

A SOM (Kohonen 1982)⁹ is an unsupervised machine-learning algorithm trained to produce a low-dimensional (typically two-dimensional) representation of a multi-dimensional space. A two-dimensional SOM contains (N_x, N_y) cells, each of them with an associated vector of attributes, in our case colour vectors. Initially, each cell is represented with random colours, which during the training phase are optimised to represent the colour space of the training sample. The SOM training also groups together cells representing similar colours, creating a colour-space map. Once trained, each galaxy is assigned to its closest cell in colour space. Moreover, since the SOM clusters galaxies with similar galaxy colours it also clusters galaxies with similar redshifts (Masters et al. 2015; Buchs et al. 2019). The appendices contain a more detailed explanation of SOM algorithms. Self-organising maps have already been used in different astronomical applications, such as the correction for systematic effects in angular galaxy clustering measurements (Johnston et al. 2021b) and for estimation and calibration of photometric redshifts (Carrasco Kind & Brunner 2014; Wright et al. 2020a,b; Hildebrandt et al. 2021).

To show the MTL performance in colour space we trained a 60 × 70 SOM on the uBVriz photometry from the COSMOS2015 catalogue (see Sect. 2.3), and subsequently assigned a SOM cell to each galaxy in the catalogue. The choice of SOM dimension is based on previous works, where 60 × 70 cells was found to give a good balance between resolution in colour space and the number of galaxies per cell. Figure 7 shows the predicted photo-zs in colour space, with each column corresponding to a photo-z estimation method described in Sect. 3.2. The first row shows the photo-z distribution, where each cell is coloured with the median photo-z of the galaxies it contains. The leftmost panel (z_s, panel A) displays the photo-zs with the baseline network (z_s method), and the second (B) and third (C) panels include MTL in the training (i.e. z_s+NB and NB+z_s+z_PAUS methods, respectively; bottom panel of Fig. 2). The rightmost panel shows the ground-truth redshift distribution.

The three methods show a photo-z distribution in colour space that is similar to that of the ground-truth redshifts. However, some differences can be seen in the plots in the second row (panels D, E, and F), which show the differences between the predicted and true-redshift colour maps (e.g. panel D = panel A − z_t). The network trained with only broad bands (panel D) exhibits two regions with less accurate photo-zs. These regions are centred around coordinates (5, 35) and (55, 25), and the redshift accuracy improves when MTL (panel E) or z_PAUS+MTL (panel F) are included in the training.

These regions are also spotted in the third row of Fig. 7, which shows the photo-z precision (σ₆₈, Eq. (5)). Comparing panels G and D, we note that the photo-z precision worsens in the same regions where photo-zs are less accurate, but this moderately improves with MTL (z_s+NB, panel H) and including the PAUS+COSMOS photo-zs (NB+z_s + z_PAUS, panel F). Finally, the fourth row shows the dispersion of the redshift distribution (i.e. the width of the N(z) within SOM cells. This quantity is also higher for the clusters pointed out in panels D and G. However, contrary to the previous panels, the z_s+NB training (panel K), or the z_s+NB+z_PAUS (panel L) do not narrow the redshift distributions.

The fact that the photo-z accuracy and precision improve with MTL, while the width of the redshift distribution does not, suggests that galaxies from different populations, that is, galaxies with different redshifts, are assigned to these cells. Figure 8 supports this hypothesis by showing that the PAUS+COSMOS photo-zs also exhibit a higher redshift dispersion (right panel) in the SOM cells within the problematic regions, while the PAUS+COSMOS photo-z accuracy is smooth across colour space (left panel). Therefore, there are galaxies with different redshifts clustered together in broadband colour space.

Fig. 7 SOMs showing the photo-z performance in the COSMOS field. The first row exhibits the median predicted photo-z in colour space for the baseline network (first panel), including MTL training (second panel), with MTL and data augmentation with PAUS+COSMOS photo-zs (third panel), and the ground-truth redshift (fourth panel). The second row shows the bias in the photo-z predictions for the three training methods of the first row (first three panels). The third row follows the same scheme as the second but displays the photo-z precision. Finally, the fourth row shows the photo-z cell dispersion also following the same scheme. White cells correspond to empty cells, that is, cells without any galaxy.

Fig. 8 Bias (left) and precision (right) of the PAUS+COSMOS photo-zs in the COSMOS spectroscopic sample.

6.2 Broadband degeneracies in colour space

Self-organising map cells that contain different galaxy populations can be the result of colour-redshift degeneracies in the broadband photometry. Such broadband degeneracies also cause the worse photo-z performance of the baseline network in the problematic colour-space regions. The photo-z performance improves with the MTL training (panel E in Fig. 7).

The inaccurate photo-z cluster in Fig. 7 is adjacent to an empty colour-space region, which shows up as a blank stripe separating two neighbouring galaxy populations. To understand which galaxies populate cells next to empty regions, we trained a SOM on a simulated galaxy sample (see Sect. 2.5 for details on the mock) using the uBVriz broadband photometry. The top panel in Fig. 9 shows the median distance among the SOM vectors characterising each cell and its directly neighbouring cells (within a 3 × 3 square). Compared with the bottom panel in the same figure (where we have assigned each galaxy in the mock to a SOM cell), one can visually see that regions showing larger distances in the upper plot coincide with empty regions (blank stripes) in the bottom ones. Therefore, cells neighbouring empty colour-space regions represent noisier or outlier galaxies, whose colours differ from the rest of the galaxy sample.

To directly see the effect of noise in the SOM, the bottom row in Fig. 9 shows the colour-space redshift distribution for the noisy (left) and noiseless (right) colours of the same galaxies. Comparing the two panels demonstrates that the blank region between galaxy populations is broader in the noiseless case. When noise is included, cells on the edges of the empty regions in the right panel are populated. This, together with such cells being located further from the other cells in colour space (top panel), indicates that cells neighbouring empty spaces describe a colour-space region that is not representative of the majority of the galaxy sample (e.g. very noisy galaxies or outliers), which can potentially cause broadband colour-redshift degeneracies.

Fig. 9 SOM trained on a galaxy simulated mock with the uBVriz broad bands. Top: distance between every SOM cell vector and its 3 × 3 neighbours. Bottom left: median photo-z in each SOM cell for noisy simulated galaxies. Bottom right: median photo-z in each SOM cell for noiseless simulated galaxies.

Fig. 10 Photo-z scatter for galaxies in three independent SOM cells. The galaxies in each cell are represented with a different marker (stars, crosses, and circles).

6.3 Emission-line confusions

The SOM in Fig. 8 shows a region in colour space that contains different galaxy populations, which indicates the potential presence of colour-redshift degeneracies. Figure 10 shows the photo-zs of the galaxies assigned to three different cells within such a colour-space region. There are four different redshift populations assigned to the region: z ~ 1.4, z ~ 0.4, and z ~ 1.2, which is many times confused with galaxies at z ~ 0.8. For the three cells (each of them represented with a different style marker), we plotted the predicted photo-z (z_p) and the true one (z_t) with the baseline network (z_s, blue), the network including MTL (z_s+NB, red), and that including MTL and photo-z data augmentation (z_s+NB+z_PAUS, orange).

The first cell (marked with stars) contains galaxies with z_t ~ 0.4 and the three networks predict the correct redshift. The second cell (marked with crosses) contains galaxies with z_true ~ 0.8 and z_true ~ l·2. In general, the MTL network improve the photo-z prediction of these galaxies. Lastly, the third cell (marked with dots) contains galaxies with redshifts z_t ~ 1.4. The baseline network predicts these photo-zs around z_p ~ 0.8, and again the z_s+NB and the MTL+z_PAUS training approaches are able to improve the photo-zs. Photo-z confusions from z_t ~ 0.8 to z_t ~ 1.2 and from z_t ~ 1.45 to z_t ~ 1.25 are recurrent, showing up at several SOM cells within the low photo-z performance cluster.

Figure 11 explores the mean Hα, Hβ, [O II], and [O III] emission-line luminosity in colour space. The emission-line luminosity is estimated as $$L_{\text{el}}:=4\pi F_{\text{el}}{D}_{\text{L}}^2,$$(9)

where F_el is the emission-line flux and D_L is the luminosity distance, which is estimated assuming Planck 2020 cosmology (Planck Collaboration VI 2020). Emission-line fluxes are taken from the photometry catalogue used for the PAUS+COSMOS photo-z (Alarcon et al. 2021), which were estimated by fitting the galaxy photometry to a template that modelled the emission-line fluxes as a 10 Å wide Gaussian distribution.

Figure 11 shows strong emission lines at the low photo-z performance colour-space regions, for example the regions centred at (5, 30) and (55, 25). These results, together with the redshift confusions seen in Fig. 10, suggest that emission lines are likely to cause degeneracies in broadband data.

Since a high ratio of [O III] to Hβ lines may indicate the presence of active galactic nuclei, we first verified that our galaxies do not host a Seyfert nucleus. The distribution of our sample on the ‘blue’ emission-line diagnostic diagram (Lamareille 2010) classify our sources as star-forming galaxies. Looking at the correlation of star-formation rates and stellar masses, often called the main sequence (Whitaker et al. 2012), galaxies showing a photo-z mismatch from z_t ~ 0.8 to z_p ~ 1.2 occupy the starburst region (i.e. galaxies with enhanced star formation, Rodighiero et al. 2011). Furthermore, these two emission lines overlap at wavelengths between the i- and z-broadband filters, which makes the emission line harder to detect.

Our findings suggest that some photometric features cause the photo-z mismatches. Emission lines have proven helpful to break colour-colour degeneracies and to improve the photo-z estimation (Csörnyei et al. 2021). Despite this, in some regions of colour parameter space emission-line confusion is a potential cause for colour-redshift degeneracies.

Fig. 11 Emission-line luminosity in colour space for Hα, Hß, [O II], and [O III], as indicated in the title.

7 Understanding the MTL underlying mechanism

In this section we aim to understand the underlying mechanism of MTL that improves the photo-z estimation. In Sect. 7.1, we use a variation of our fiducial network to encode the galaxy photometry in a 2-dimensional space similar to a SOM, while in Sect. 7.2 we study the impact of using other auxiliary tasks (other than predicting the narrow-band photometry) in the MTL network.

7.1 Underlying data representation in colour space with MTL

For this test, we modify the fiducial network architecture (see Sect. 3.2 and Fig. 2). In the modified network, we reduce the input dimension to two features, which are used to predict the photo-z and reconstruct the narrow-band colours. Encoding the galaxy information in a two-dimensional feature space simplifies its visualisation and brings it closer to the SOM colour-space representation, which we have already studied (Sect. 6).

The galaxy representations in the two-dimensional feature space must encode all the information needed to make the photo-z prediction. Furthermore, in the MTL network, those two numbers are also used to reconstruct the narrow-band photometry and thus must also encode the relevant information for this task. Therefore, comparing the feature space representation of the baseline network (decoding only to the photo-z) and the MTL network (also predicting the narrow-band photometry) helps us to better understand why the MTL improves the photo-z estimates.

As the network’s feature space is not constrained, the network can encode the same galaxy differently in several independent trainings. Consequently, the coordinates assigned to each galaxy do not contain any valuable information by themselves and distances from different feature-space maps (e.g. the feature map of the z_s network and that of the z_s+NB) cannot be directly compared. However, the overlap of different redshift populations in feature-space indicate potential degeneracies.

In Fig. 12 we plot the 50% , 68%, and 95% contours of the feature-space coordinates for the z_s (left panel) and the z_s+NB (right panel) methods. These are drawn using a test set of 70 000 Flagship galaxies (Sect. 2.5) to i_AB < 25, while the methods train on galaxies to i_AB < 23 (Sect. 5). We draw the contours for a selection the redshift bins to show the separation of high-redshift galaxies (red, blue, and green contours), where the MTL method significantly reduces the photo-z scatter with respect to the z_s method (Fig. 6, top and bottom left panels). We have also plotted the contours of a distant redshift population (purple contours) to show that this is further in feature space than the others.

There is a significant overlap amongst high-redshift populations in the z_s case (left panel). Particularly, the core of the green and blue contours overlap with the red-contour galaxies. We expect some overlap since the three contours are consecutive in redshift; however, the z_s+NB method shows a cleaner separation between the three redshift populations. This indicates that the z_s+NB has a better internal representation of the galaxies, where different redshift populations are further in feature space. The narrow-band reconstruction loss (Eq. (2)) adds the low-resolution SED information to the training, which can potentially lead to an improved internal representation of galaxies in the two-dimensional feature space. Furthermore, MTL methods also include this information for galaxies without spectroscopic redshift, which effectively acts as a data augmentation technique. This is particularly important for high-redshift galaxies, for which we have very few examples in the spectroscopic sample (Fig. 1).

Fig. 12 Contours of the two-dimensional feature space coordinates for the zs (left) and zs + NB (right) methods. The features from each of the methods are from independent training and cannot be compared. We can only compare the overlap of the different populations.

Fig. 13 Photo-z precision in the COSMOS field when the auxiliary task of predicting the galaxy SED is included in the training. The galaxy SED prediction is addressed as a classification, where the true SED is a class between 1 and 47.

7.2 MTL with other galaxy parameters

So far in this paper, we explored how photo-z predictions benefit from MTL predicting PAUS narrow-band fluxes as an auxiliary task. However, MTL is a more general technique that could be exploited beyond narrow-band photometry reconstructions. While a conventional neural-network training searches for the function (ϕ) that best predicts the photo-z (z) given the broadband photometry (ƒ), namely ϕ(z|f), with MTL the optimisation is extended to the function that best predicts the photo-z together with other related parameters (x_i), $$\varphi \left(z,x_1,\dots ,x_{\text{N}}|\theta \right),$$(10)

where x_i could be any galaxy parameter that correlates with the galaxy photo-z such as the galaxy type.

Template-fitting photo-z methods predict the joint probability distribution p(z, t|f) of the redshift (z) and the galaxy type (t) and marginalise over the templates (Benítez 2011). In principle, this is closely related to what MTL does when it is required to predict both quantities at the same time. The network looks for the function that better generalises the prediction of both parameters (e.g. type and redshift), but makes independent predictions in which it ‘marginalises’ over the parameter it is not predicting.

Figure 13 shows the photo-z precision of data in the COSMOS field when the galaxy type is included as an MTL auxiliary task. The SED template is encoded as a discrete number between 1 and 47 as described in the COSMOS2015 catalogue. These correspond to 31 unique SEDs and 16 SEDs with different extinction laws. Including the SED template (dotted blue line) reduces the photo-z scatter with respect to the baseline network (solid black line). However, MTL using PAUS narrow bands (dashed red line) still provides better photo-z estimates. This result suggests that while the SED helps produce a better representation of the data in colour space (see Sect. 7.1), PAUS narrow-band photometry contains information about the SED, as well as the emission lines or the extinction.

Figure 13 also shows the photo-z performance when both the SED and the narrow-band data are used as auxiliary tasks (green dashed-dotted line). We find that this degrades the photo-z performance with respect to using the SED or the narrowband photometry solely. In theory, using both the narrow-band photometry and the SED number should benefit the network. However, the information available in these two tasks is highly correlated, which can hinder the predictions. Understanding this better is ongoing research and further study is deferred to future work.

We also explored MTL predicting galaxy parameters such as the star-formation rate, the galaxy mass, and the E(B – V) extinction parameter as auxiliary tasks (not shown). However, none of these parameters improved the predicted photo-zs. Furthermore, including the near-infrared photometry did not improve the photo-zs either.

Fig. 14 Photo-z precision as a function of number of bands in the predicted photometry for zs +NB (dotted blue line) and zs + zPAUS +NB (dashed red line). The horizontal line corresponds to the zs (dashed-dotted blue line) and zs + zPAUS (solid red line), where MTL is not enabled.

7.3 Effect of narrow-band resolution

The improved photo-z from predicting the narrow-band photometry can potentially result from a better internal description of the galaxy SED type. We test this hypothesis by evaluating the performance of the networks using MTL for different resolutions of the output predicted photometry.

Figure 14 shows the photo-z precision of the MTL methods as a function of the number of predicted narrow bands (i.e. the output photometry resolution). Assuming the MTL networks use the narrow-band photometry to improve the internal representation of galaxies, increasing the output photometry resolution effectively corresponds to turning on this mechanism. To obtain lower-resolution photometries, we take the mean of groups of consecutive narrow bands (e.g. 2, 4, and 10). Then, we train the z_s+NB and z_s + z_PAUS+NB methods several times to predict the photo-z and the narrow-band photometry with a different resolution in every training.

The horizontal flat lines in Fig. 14 indicate the photo-z precision for the methods without MTL; z_s (dashed-dotted blue line) and z_s + z_PAUS (solid red line). The dotted blue line and the dashed red line show the z_s+NB and z_s + z_PAUS+NB performance for the different output photometry resolutions, respectively. As the output photometry resolution increases, the photometric redshift precision improves. This suggests that the MTL networks are using the narrow-band photometry prediction to improve the internal representation of the SED, and consequently the SED internal fitting, which has a direct impact in the photo-z prediction. The narrow-band photometry contains important additional information about the SED type and galaxy parameters, which are useful when predicting the redshift.

The z_s+NB MTL recovering two-band photometry leads to predictions above the z_s line, which is the result without MTL. In this limit adding the photometry loss degrade the photo-z results. We trained this network several times to ensure the result was correct, obtaining the same degrading in all cases.

8 Discussion and conclusions

Photometric redshifts are crucial for exploiting ongoing and future large galaxy broadband imaging surveys. While covering large sky areas, the broadband spectral resolution limits the redshift performance through colour-redshift degeneracies. The PAUS is a narrow-band imaging survey that can provide very precise photo-z measurements for a combination of wide and deep fields. In this paper we have introduced a new method for improving broadband photo-z estimates, using deep-learning techniques on PAUS narrow-band data.

Multi-task learning is a machine-learning training methodology that aims to improve the performance and generalisation power of a network by training it on several related tasks simultaneously. This forces the model to share representations among related tasks, exploiting their commonalities and enabling the network to generalise better on the original task. We implemented an MTL network that simultaneously predicts the photometric redshift and infers the narrow-band photometry from the broadband photometry (see Sect. 3). The photo-z network is therefore forced to share parameters that are also used to predict the narrow-band photometry, which improves the internal colour-space representation of the data.

In the COSMOS field for galaxies to i_AB < 23, our method reduces the photo-z scatter by approximately 20% (see Sect. 4.2) and the number of photo-z outliers by from ~1.1 to ~0.6% (see Sect. 4.3). We also tested the potential of the method for fainter galaxies using Euclid-like galaxy simulations. For this, we trained the network on a magnitude-limited sample with i_AB < 23 and evaluated it on a sample with i_AB < 25. The MTL predicts up to 16% more precise photo-z_s for galaxies with i_AB < 25 than the baseline network (see Sect. 5).

We used SOMs to study the photo-z performance in different colour-space regions, detecting a region that contains galaxies with degenerate photometry-redshift mappings. This region has a larger photo-z variation within the SOM cells, suggesting that more than one galaxy population is assigned to the same colour-space location (see the left panel in Fig. 8). This correlation results in a photo-z mismatch between two galaxy populations, which affects broadband photo-z estimates. Our MTL network improves the photo-zs in the degenerated colour-space regions using PAUS narrow-band data to learn the underlying colour-space distribution of galaxies.

This paper explores how to exploit data from narrow-band photometric surveys such as PAUS to improve broadband photo-z estimates using machine learning. The key point of using MTL instead of, for example, just using the narrow-band photometry to obtain more precise photo-zs is that it only requires narrow-band photometry for the training galaxies and the photo-z of any galaxy can be evaluated with only the broadband data. This enables us to exploit fields where we have narrowband data to obtain better photo-zs in other fields where these are not available. PAUS photometry in the COSMOS field is publicly available, so current and future weak lensing surveys, such as Euclid or the LSST, can readily benefit from this methodology to improve their photo-z estimates. Moreover, MTL is a general machine-learning mechanism that enables fields with different types of photometry to be exploited in order to improve photo-z predictions. While PAUS narrow-band photometry is a clear candidate, other surveys such as J-PAS (Benitez et al. 2014) or ALHAMBRA (Moles et al. 2008) provide more fields with interesting data to exploit for the benefit of photo-z estimations.

Acknowledgements

The PAU Survey is partially supported by MINECO under grants CSD2007-00060, AYA2015-71825, ESP2017-89838, PGC2018-094773, PGC2018-102021, SEV-2016-0588, SEV-2016-0597, MDM-2015-0509, PID2019-Ш317GB-C31 and Juan de la Cierva fellowship and LACEGAL and EWC Marie Sklodowska-Curie grant No 734374 and no.776247 with ERDF funds from the EU Horizon 2020 Programme, some of which include ERDF funds from the European Union. IEEC and IFAE are partially funded by the CERCA and Beatriu de Pinos program of the Generalitat de Catalunya. Funding for PAUS has also been provided by Durham University (via the ERC StG DEGAS-259586), ETH Zurich, Leiden University (via ERC StG ADULT-279396 and Netherlands Organisation for Scientific Research (NWO) Vici grant 639.043.512), Bochum University (via a Heisenberg grant of the Deutsche Forschungsgemeinschaft (Hi 1495/5-1) as well as an ERC Consolidator Grant (No. 770935)), University College London, Portsmouth support through the Royal Society Wolfson fellowship and from the European Union’s Horizon 2020 research and innovation programme under the grant agreement No 776247 EWC. The results published were also funded by the Polish National Agency for Academic Exchange (Bekker grant BPN/BEK/2021/1/00298/DEC/1), the European Union’s Horizon 2020 research and innovation programme under the Maria Skłodowska-Curie (grant agreement No 754510) and by the Spanish Ministry of Science and Innovation through Juan de la Cierva-formacion program (reference FJC2018-038792-I). The PAU data centre is hosted by the Port d’Informació Científica (PIC), maintained through a collaboration of CIEMAT and IFAE, with additional support from Universitat Autònoma de Barcelona and ERDF. We acknowledge the PIC services department team for their support and fruitful discussions. CosmoHub has been developed by the Port d’Informació Científica (PIC), maintained through a collaboration of the Institut de Física d’Altes Energies (IFAE) and the Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas (CIEMAT) and the Institute of Space Sciences (CSIC&IEEC), and was partially funded by the “Plan Estatal de Investigación Científica y Técnica y de Innovación” program of the Spanish government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research. The Euclid Consortium acknowledges the European Space Agency and a number of agencies and institutes that have supported the development of Euclid, in particular the Academy of Finland, the Agenzia Spaziale Italiana, the Belgian Science Policy, the Canadian Euclid Consortium, the French Centre National d’Etudes Spatiales, the Deutsches Zentrum für Luft- und Raumfahrt, the Danish Space Research Institute, the Fundação para a Ciência e a Tecnologia, the Ministerio de Economía y Competitividad, the National Aeronautics and Space Administration, the National Astronomical Observatory of Japan, the Netherlandse Onderzoekschool Voor Astronomie, the Norwegian Space Agency, the Romanian Space Agency, the State Secretariat for Education, Research and Innovation (SERI) at the Swiss Space Office (SSO), and the United Kingdom Space Agency. A complete and detailed list is available on the Euclid web site (http://www.euclid-ec.org). Data availability: The PAUS raw data are publicly available through the ING group. A few reduced images are publicly available at https://www.pausurvey.org. The Flagship catalogue is a property of the Euclid Consortium.

Appendix A Self-organising maps

A SOM (Kohonen 1982) is an unsupervised machine-learning algorithm trained to produce a low-dimensional (typically two-dimensional) representation of a multi-dimensional space. A two-dimensional SOM contains N_x × N_y cells with an associated vector of attributes (w^k), where N_x(N_y) is the dimension of the SOM on the x(y)-axis, and k corresponds to the kth SOM cell. Each of these vectors has the same length as the input data.

The SOM training phase is an iterative process during which the SOM cells compete amongst themselves to represent the training data. Initially, the cell vectors (w^k) are randomly sampled from a uniform distribution, and these are updated after each iteration step (t). In every training iteration, each galaxy vector of measured attributes x (e.g. in our case the galaxy colours), is compared to all the SOM cells' vectors via a χ² expression, $${\chi }^2\left(w^k\left(t\right),x\right)={{\displaystyle \sum _i\left[\frac{x_i-w_i^k\left(t\right)}{{\sigma }_i}\right]}}^2,$$(A.1)

where i sums over galaxy attributes and σ_i is the uncertainty associated with x_i. The evaluated galaxy is assigned to the cell with the lowest χ², which updates its associated vector of attributes w^k (t) according to the matched galaxy features.

Furthermore, in the SOM training procedure, the vector of features from cells neighbouring the best matching cell are also updated, clustering together galaxies with similar attributes. This is implemented with a neighbouring function H(t, d), which depends on the distance (d) between the best matching cell and the updated one. The neighbouring function is commonly implemented as a Gaussian kernel with an iteration-dependent variance ${\sigma }_{\text{kernel}}^2\left(t\right)$. Therefore, the vector of attributes for a particular cell k after iteration t + 1 is $$w^k\left(t+1\right)=w^k\left(t\right)+\alpha \left(t\right)H\left(t,\left|w-x\right|\right)\left(x-w^k\left(t\right)\right),$$(A.2)

where α(t) is the learning rate.

After a few iterations over the training sample, the result is a map of (N_x × N_y) vectors in a two-dimensional space grouping together cells with similar features while preserving the topology of the multi-dimensional space.

Appendix B Redshift distributions, N(z), and scatter plots

Unbiased redshift distributions, N(z), are crucial for a variety of science applications, with the most stringent requirements being in weak lensing (e.g. Hildebrandt et al. 2012; Hoyle et al. 2018). broadband photo-zs commonly suffer from biases due to degeneracies between colours and redshift, (e.g. Newman et al. 2015; Masters et al. 2017).

Figure B.1 shows N(z) in tomographic redshift bins for 0 < z_t < 1.5 spaced by 0.2. The last tomographic bin is defined from 1.2 < z_t < 1.5 so that the number of galaxies in the bin is increased. The ground-truth redshift defining the tomographic bins (z_t) is a combination of the spectroscopic redshift (when it is available) and the PAUS+COSMOS photo-z elsewhere. The vertical solid grey line indicates the ground-truth median redshift of the tomographic bin, while the dashed coloured lines represent the median redshifts of the predicted photo-zs assigned to the bins.

Multi-task learning with photo-z data augmentation (z_s + z_PAUS+NB) always provides equal or more accurate N(z) than the baseline network (z_s, black line). As expected from Fig. 4, the N(z) values exhibiting the largest bias are those with z_t > 1.2, particularly the bin at z_t > 1.2. In this bin, MTL together with the photo-z data augmentation (z_s+NB+z_PAUS, green line), significantly shifts the median of the N(z) towards the PAUS+COSMOS result.

Commonly, redshift distributions require a bias correction to reach the accuracy requirements of cosmological measurements. Techniques such as clustering redshifts are applied to correct such biases (Ménard et al. 2013; Schmidt et al. 2013; Gatti et al. 2018; van den Busch et al. 2020; Hildebrandt et al. 2021). MTL reduces the bias of the N(z) already at the photo-z prediction stage. Even if the MTL photo-zs still require some correction, the final redshift distributions would benefit from initially having less biased redshift distributions (if these redshift distributions are used to fit the clustering-z data points).

Figure B.2 shows the density scatter between the predicted photo-zs and the true redshift. Here, we are plotting the complete COSMOS sample, and we therefore use a combination of spectroscopic redshift and PAUS+COSMOS photo-zs as true redshift. The top left panel corresponds to baseline network (z_s method) and clearly shows higher photo-z scatter and more outliers in the high-redshift region with respect to the other methods. The MTL method (top right, z_s+NB) already reduces photo-z scatter and number of high-redshift outliers. The methods including additional PAUS+COSMOS photo-zs in the training sample (bottom panels) further improve the photo-z performance.

Fig. B.1 N(z) estimates of the full COSMOS sample divided into seven tomographic bins over the redshift range 0 < z < 1.5. Tomographic bins are defined using the spectroscopic redshifts and the PAUS+COSMOS high-precision photo-zs for galaxies without spectroscopy. The vertical solid black lines indicate the median ground-truth redshift, while the other vertical lines indicate the median redshifts of the N(z) estimates. Unseen lines are hidden by other overlapping lines.

Fig. B.2 Scatter plot of the 1:1 relation between the predicted photo-z and the true redshift, which is a combination of spectroscopic redshift and PAUS+COSMOS photo-zs, in the complete COSMOS sample for the four methods in Sect. 3.

Appendix C Effect of training with photo-zs as ground-truth targets

In this work we have implemented and tested two training methodologies that rely on narrow-band photo-z estimates as ground-truth targets (see methods z_s + z_PAUS and NB+z_s + z_PAUS in Sect. 3.2). Even if such photo-zs are overall very accurate, its implementation in the training could harm the photo-z performance since these are less precise than spectroscopic redshifts and could potentially include outliers. In this section we explore the effect that less precise redshift labels (Sect. C.1) and the presence of outliers (Sect. C.2) have on the photo-z performance.

Appendix C.1 Effect of the dispersion in the ground-truth photo-z

Figure C.1 shows the photo-z precision of a set of 1000 spectroscopic galaxies for four independent broadband networks (simply mapping colours to redshift), each of them trained with different ground-truth redshifts. The redshifts used for training are the spectroscopic redshifts (see Sect. 2.4), the PAUS+COSMOS photo-zs (see Sect. 2.2), the COSMOS30 photo-zs (Laigle et al. 2016), which combine 30 photometric filters and estimates the photo-z with LePhare (Arnouts & Ilbert 2011), and a set of CFHT photo-zs from Hildebrandt et al. (2012) combining six broad bands (ugriz) with photo-z estimated with BPZ (Benítez 2011). The input data are, in all cases, the CFHT u band and the BVriz Subaru broadband filters from COSMOS2015.

The red points in Fig. C.1 show the redshift dispersion using a training sample of galaxies with spectroscopic redshift. We always keep the same training sample (which contains around 6000 galaxies) and change the labelled true redshifts in each independent training (spectroscopic catalogue, PAUS data, COSMOS30, and the CFHT catalogue). Using spectroscopic redshifts as ground-truth redshifts results in a dispersion of σ₆₈ = 0.016. Replacing the spectroscopic redshift with the photo-z from PAUS+COSMOS, COSMOS30, or CFHT yields σ₆₈ = 0.017, σ₆₈ = 0.018, and σ₆₈ = 0.046, respectively. As the ground-truth redshifts become less precise, the machine-learning photo-z performance degrades.

To obtain the green points (Fig. C.1), we extended the training sample to all galaxies in the COSMOS sample with a photo-z estimate, which results in approximately 15 000 galaxies when the four catalogues are merged. Then, three independent networks are trained using the PAUS+COSMOS, the COSMOS-30, and the CFHT photo-zs as true redshifts (the spectroscopic redshift is not used even if it is available). This provides a precision of σ₆₈ = 0.016, σ₆₈ = 0.017, and σ₆₈ = 0.045 for the PAUS+COSMOS, the COSMOS30, and the CFHT photo-zs, respectively. The three networks improve the photo-z precision with respect to training with spectroscopic redshifts only. Indeed, with the PAUS+COSMOS photo-z labels we already reach the photo-z precision with spectroscopic labels.

Fig. C.1 Photo-z performance as a function of the ground-truth redshift precision used for training the networks. The training redshifts are the spectroscopic redshifts, the PAUS+COSMOS photo-zs, COSMOS30, and a set of CFHT photo-zs in COSMOS. Red points correspond to training on the spectroscopic sample (around 6000 galaxies). The green and red points show the training sample extended to COSMOS galaxies with photo-zs (around 15 000 galaxies). The blue lines show the expected photo-z performance as a function of target redshift precision. The true redshifts, the spectroscopic redshift in the COSMOS2015 catalogue (solid blue line), and the simulated redshift in the PAUS mock (blue dashed line) are scattered with precision in 0.001 bins. The top inset zooms into the framed area in the main plot (lower-left corner)

Finally, the blue points in the figure correspond to the networks trained with the same 15 000 photo-z galaxies as in the green points, but combining spectroscopic redshifts (if available) and photo-zs as ground-truth training redshifts. Combining spectroscopic redshifts with PAUS+COSMOS photo-zs yields σ₆₈ = 0.015, which improves upon the precision obtained with spectroscopic redshifts only.

The light blue lines in Fig. C.1 show the expected performance as a function of the ground-truth redshift precision. The solid line uses the COSMOS2015 uBVriz broad bands and the dashed one uses simulated data from the PAUS mock described in Sect. 5. In both cases, true redshifts (spectroscopic or simulated) are scattered with the corresponding dispersion in the abscissa.

Both networks (solid and dashed lines) are trained with 15000 galaxies to have a direct comparison with the previous results. We always use the scattered redshifts as ground-truth targets, in such a way that the lines should be compared with the green points since these are trained using only photometric redshifts. The results obtained with the PAUS+COSMOS and COSMOS30 match the expectation curves, but there is a significant mismatch with the CFHT photo-zs. This is potentially triggered by systematic errors or outliers in the CFHT photo-z not represented in the blue curves.

Appendix C.2 Effect of photo-z outliers in the training redshifts

In Fig. C.1 showed a mismatch between the expected (solid blue curve) and photo-z performance training a network with the CFHT photo-zs as ground-truth redshifts (rightmost red point). However, the expectation assumes that the CFHT photo-zs are not affected by other effects such as systematic errors or catastrophic outliers.

Figure C.2 shows the effect of outliers in the ground-truth targets of the training sample. The network is trained 20 independent times with 5000 COSMOS2015 spectroscopic galaxies including a fraction (monotonically increasing in each iteration) of labelled photo-z outliers. This procedure is repeated for the spectroscopic redshifts (black line), the PAUS+COSMOS photo-zs (red line), the COSMOS30 photo-zs (blue line), and the CFHT photo-zs (green line).

In the left panel, the artificial outlier redshifts are swapped with a random value sampled from a uniform distribution U(0, 1.5) to simulate catastrophic outliers. The predicted photo-z precision degrades as the fraction of target redshift outliers increases. This also affects the predicted p(z), which become noisier and broader (not shown). However, and unexpectedly, the network can provide reasonable photo-z estimates with up to 80% of catastrophic outliers in the training sample. Furthermore, the network is able to make reliable photo-z predictions of galaxies that have been used in the training sample with wrong target redshift values. This result holds when either spectroscopic redshifts or any of the photo-zs are used for training.

The middle panel shows the effect of a systematic multiplicative shift in the training sample redshifts, where the selected targets are shifted to 20% higher redshifts. In this scenario, the predicted photo-z precision degrades faster than when outliers are random (left panel) but the network does never completely break. For an outlier fraction higher than 60%, the precision settles at σ₆₈ = 0.03, but the bias rapidly increases. Finally, the rightmost panel presents the effect of a systematic shifting the redshift (z_mod) so that O III is confused with Hα in the training redshifts: $$z_{\mathrm{mod}}={\lambda }_{\text{O\hspace{0.17em}III}}/{\lambda }_{\text{H}\alpha }\left(1+z_{\text{t}}\right)-1,$$(C.1)

where z_t is the galaxy redshift.

The training degrades and breaks much faster than in the two previous cases, where with around 40% of wrong target redshifts the network is not able to provide reliable predictions. As the fraction of affected target redshifts increases, the predicted p(z) become more doubly peaked. Moreover, a plot of photo-z versus spec-z scatter displays two clear lines, one with the correct mapping and another shifted upwards (not shown), which is triggered by the training objects with the photo-z artificially shifted to confuse the emission lines. Again, the effect of outliers is similar regardless of the redshifts used for training (spectroscopic or different-precision photo-z).

Contrary to expectations, the left panel of Fig. C.2 indicates that the network can learn the mapping between the galaxy photometry and redshifts with up to 80% of catastrophic outliers in the training sample. Given that the training sample is composed of 5000 galaxies, this means that the network can effectively learn the colour-redshift relations from 1000 galaxies, learning to ignore the remaining 4000 spurious galaxies.

Figure C.3 shows the cost function evolution of a network trained with wrong target redshifts for half of the training while keeping the rest to the correct redshift values. The cost function is split in two; one for those objects with correct redshift (red) and another for those with wrong redshifts (blue). In the left panel, the modified target redshifts are switched to a random value from a uniform distribution U(0, 1.5), as in the left panel of Fig. C.2. The cost function of galaxies with the correct target redshift decreases, which indicates that the network is learning from them. In contrast, the cost function of incorrectly labelled galaxies remains constant along the training, showing that the network is not learning anything from them. Therefore, the network is effectively only learning from galaxies with correct target redshifts. Randomly swapping redshifts to different values breaks any correlation between the photometry and the redshifts. Hence, the network is only learning the colour-redshift mapping from galaxies with the correct target redshift. Nevertheless, having a large fraction of wrong labels adds noise to the training, broadening the predicted p(z).

Fig. C.2 Effect of outliers and systematic errors on the ground-truth redshift sample used during training. The training sample consists of 5000 spectroscopic galaxies with photometry from COSMOS2015. Each coloured line uses a different sample of redshifts as true redshifts, i.e. spectroscopic redshifts (black), PAUS+COSMOS photo-zs (red), COSMOSЗO photo-zs (blue), and CFHT photo-zs (green). The ground-truth redshift of the selected fraction of training galaxies is replaced by a random redshift value sampled from U(O, 1.5) (left), a 20% higher redshift (centre), and redshifts modified with Eq. (C.1) right).

Fig. C.3 Training loss function for galaxies with a wrong (blue) and a corrected (red) target redshift. The training sample consists of 5000 spectroscopic galaxies with photometry from COSMOS2015. In the left panel, the modified target redshifts are randomly switched to a value drawn from U(0, 1.5), while in the right panel the wrong redshift labels are generated with Eq. (C.1).

The right panel of Fig. C.3 shows the loss function for the correct and the wrongly labelled training galaxies separately when the incorrect redshift labels are generated with Eq. (C.1). This introduces a new colour-redshift relation that forces the network to learn both from galaxies with wrong and correct target redshifts. This can also be noted in the p(z) behaviour, which presents a double-peaked distribution (not shown). Hence, Figs. C.2 and C.3 indicate that having catastrophic outliers in the training sample labels effectively adds noise to the photo-z predictions. In contrast, a systematic bias in the training sample targets produces a bias in such predictions.

Appendix D Robustness of the methods to outliers in the target redshifts

In this appendix we study the robustness of our training methodologies to outliers in the redshifts used as ground-truth to train the network. In Sect. 5, in order to simulate the PAUS+COSMOS photo-zs used to train the z_s+z_PAUS and the z_s+z_PAUS+NB methods, we scatter the true redshift from the simulations to a similar precision of PAUS+COSMOS photometric redshifts. This process assumes that photometric redshift errors are purely Gaussian; however, real data also have non-Gaussian errors and photo-z outliers, which can be caused by, for example, noisy photometry, emission-line confusions, and other artefacts in the data.

Systematic outliers in the ground-truth target redshifts have a much stronger impact than random catastrophic outliers (Figs. C.3 and C.2). Therefore, we assume the most adverse scenario where all outliers are systematically shifted according to Eq. (C.1).

Figure D.1 shows the impact that outliers in the target redshifts have in the performance of the methods. We studied four different cases: a sample without outliers in the training sample (black solid line), with a 5% of outliers in the spectroscopic redshifts and 1% of outliers in the photo-zs (red dashed line), 5% of spectroscopic redshift outliers and 5% photo-zs outliers (blue dashed line), and 5% of spectroscopic redshift outliers and 10% photo-zs outliers (green dashed line).

Adding the 5% of spectroscopic redshift outliers already has an impact on the predicted photo-z bias of the baseline method (zs, top left panel). The effect of spectroscopic redshift outliers in the training sample is mitigated by the MTL network (top right panel), which is not affected by training-photo-z outliers since these are not used during the training (method z_s + NB, Sect. 3.2). We also observe that training with photo-z samples with up to ~5% of outliers also mitigates the effect of outliers in the spectroscopic sample. This is expected since adding more training data reduces the relative importance of an outlier in the training sample. However, the bottom plots also show that training samples with more than 10% of systematic photo-z outliers degrade the photo-z performance.

We also studied the robustness of the dispersion and the outlier rate to the target-redshift outliers and these two metrics are much less affected by the presence of target-redshift outliers.

Fig. D.1 Effect on the photo-z predictions of different outlier rates in the target spectroscopic and high-precision photometric redshifts used as ground-truth targets to train the methods in Sect. 3. In all cases, outliers have been included following Eq. (C.1).

Appendix E Further studies of multi-task training

In this section we aim to give a more technical view of the network functioning. In Fig. E.1 we show the evolution of the photo-z prediction loss function (Eq. 1) with time for the baseline method (zs, black line) and the MTL method (z_S+NB, red line). In both cases, the solid line corresponds to the training loss, while the dashed line is the validation loss. The networks have been trained for 100 epochs with an initial learning rate of 10⁻³, which decreases to 10⁻⁴ after 50 epochs. In this test, unlike previous tests, we initialise the two networks with the same weights. Still we observe training with the two different losses leads to a lower photo-z loss (ℒ_z). We also find adding the narrow-band loss stabilises the photo-z loss in the validation sample, meaning the network better generalise with the additional narrow-band loss.

Fig. E.1 Training (solid lines) and validation (dashed lines) loss for the zs (black) and zs+NB (red) methods. All methods are trained for 100 epochs with an initial learning rate of 10−3 and the same initial conditions.

References

All Tables

All Figures

	Fig. 1 Redshift distributions for the COSMOS spectroscopic sample (red line) and the full (spectroscopic and photo-z) COSMOS sample.
In the text

	Fig. 2 Top: baseline network architecture. The input contains five colours that propagate through six fully connected layers. Each layer is followed by a dropout layer, which is represented by a yellow-crossed circle. Bottom: MTL network. This builds on the baseline network and adds an extra output layer for the additional task of predicting the narrow-band photometry.
In the text

Fig. 3 Photo-z dispersion in equally populated magnitude differential bins to iAB < 23 (top) and equally spaced redshift bins to z < 1.5 (bottom). Each line corresponds to a different training procedure (see Sect. 3.2). While the black line corresponds to a baseline training, the other coloured lines include MTL (red and green lines) and data augmentation with photo-zs from the PAUS+COSMOS catalogue as ground-truth redshifts (blue and green lines).

In the text

	Fig. 4 Photo-z bias in equally populated redshift bins (left) and equally populated i-band magnitude bins (right). The grey area corresponds to the Euclid photo-z bias requirement of ∆z = 0.002.
In the text

	Fig. 5 PIT distribution for the COSMOS photo-zs predicted with the baseline zs method (black), the zs+NB method (red), the zs+zPAUS method (blue), and the zs+zPAUS+NB method (green). Including the PAUS+COSMOS photo-zs in the training reduces the number of outliers on the edges of the distribution.
In the text

Fig. 6 Top: photo-z bias (left) and precision (right) in equally populated magnitude bins. Bottom: photo-z bias (left) and dispersion (right) in equally spaced spectroscopic redshift bins. The shaded grey areas indicate ∆z > 0.002, corresponding to the Euclid requirement for the photo-z bias. All plots are for 30 000 Flagship test galaxies with magnitudes iAB < 24.5 for the methods presented in Sect. 3.2. The training sample contains around 15 000 spectroscopic galaxies, extended to 30 000 with PAUS-like galaxies without spectroscopy, all of them to iAB < 23.

In the text

Fig. 7 SOMs showing the photo-z performance in the COSMOS field. The first row exhibits the median predicted photo-z in colour space for the baseline network (first panel), including MTL training (second panel), with MTL and data augmentation with PAUS+COSMOS photo-zs (third panel), and the ground-truth redshift (fourth panel). The second row shows the bias in the photo-z predictions for the three training methods of the first row (first three panels). The third row follows the same scheme as the second but displays the photo-z precision. Finally, the fourth row shows the photo-z cell dispersion also following the same scheme. White cells correspond to empty cells, that is, cells without any galaxy.

In the text

	Fig. 8 Bias (left) and precision (right) of the PAUS+COSMOS photo-zs in the COSMOS spectroscopic sample.
In the text

	Fig. 9 SOM trained on a galaxy simulated mock with the uBVriz broad bands. Top: distance between every SOM cell vector and its 3 × 3 neighbours. Bottom left: median photo-z in each SOM cell for noisy simulated galaxies. Bottom right: median photo-z in each SOM cell for noiseless simulated galaxies.
In the text

	Fig. 10 Photo-z scatter for galaxies in three independent SOM cells. The galaxies in each cell are represented with a different marker (stars, crosses, and circles).
In the text

	Fig. 11 Emission-line luminosity in colour space for Hα, Hß, [O II], and [O III], as indicated in the title.
In the text

	Fig. 12 Contours of the two-dimensional feature space coordinates for the zs (left) and zs + NB (right) methods. The features from each of the methods are from independent training and cannot be compared. We can only compare the overlap of the different populations.
In the text

	Fig. 13 Photo-z precision in the COSMOS field when the auxiliary task of predicting the galaxy SED is included in the training. The galaxy SED prediction is addressed as a classification, where the true SED is a class between 1 and 47.
In the text

	Fig. 14 Photo-z precision as a function of number of bands in the predicted photometry for zs +NB (dotted blue line) and zs + zPAUS +NB (dashed red line). The horizontal line corresponds to the zs (dashed-dotted blue line) and zs + zPAUS (solid red line), where MTL is not enabled.
In the text

Fig. B.1 N(z) estimates of the full COSMOS sample divided into seven tomographic bins over the redshift range 0 < z < 1.5. Tomographic bins are defined using the spectroscopic redshifts and the PAUS+COSMOS high-precision photo-zs for galaxies without spectroscopy. The vertical solid black lines indicate the median ground-truth redshift, while the other vertical lines indicate the median redshifts of the N(z) estimates. Unseen lines are hidden by other overlapping lines.

In the text

	Fig. B.2 Scatter plot of the 1:1 relation between the predicted photo-z and the true redshift, which is a combination of spectroscopic redshift and PAUS+COSMOS photo-zs, in the complete COSMOS sample for the four methods in Sect. 3.
In the text

Fig. C.1 Photo-z performance as a function of the ground-truth redshift precision used for training the networks. The training redshifts are the spectroscopic redshifts, the PAUS+COSMOS photo-zs, COSMOS30, and a set of CFHT photo-zs in COSMOS. Red points correspond to training on the spectroscopic sample (around 6000 galaxies). The green and red points show the training sample extended to COSMOS galaxies with photo-zs (around 15 000 galaxies). The blue lines show the expected photo-z performance as a function of target redshift precision. The true redshifts, the spectroscopic redshift in the COSMOS2015 catalogue (solid blue line), and the simulated redshift in the PAUS mock (blue dashed line) are scattered with precision in 0.001 bins. The top inset zooms into the framed area in the main plot (lower-left corner)

In the text

Fig. C.2 Effect of outliers and systematic errors on the ground-truth redshift sample used during training. The training sample consists of 5000 spectroscopic galaxies with photometry from COSMOS2015. Each coloured line uses a different sample of redshifts as true redshifts, i.e. spectroscopic redshifts (black), PAUS+COSMOS photo-zs (red), COSMOSЗO photo-zs (blue), and CFHT photo-zs (green). The ground-truth redshift of the selected fraction of training galaxies is replaced by a random redshift value sampled from U(O, 1.5) (left), a 20% higher redshift (centre), and redshifts modified with Eq. (C.1) right).

In the text

	Fig. C.3 Training loss function for galaxies with a wrong (blue) and a corrected (red) target redshift. The training sample consists of 5000 spectroscopic galaxies with photometry from COSMOS2015. In the left panel, the modified target redshifts are randomly switched to a value drawn from U(0, 1.5), while in the right panel the wrong redshift labels are generated with Eq. (C.1).
In the text

	Fig. D.1 Effect on the photo-z predictions of different outlier rates in the target spectroscopic and high-precision photometric redshifts used as ground-truth targets to train the methods in Sect. 3. In all cases, outliers have been included following Eq. (C.1).
In the text

	Fig. E.1 Training (solid lines) and validation (dashed lines) loss for the zs (black) and zs+NB (red) methods. All methods are trained for 100 epochs with an initial learning rate of 10−3 and the same initial conditions.
In the text