© The Authors 2024

1 Introduction

The accelerated expansion of the Universe is now a well-established fact, corroborated by a large amount of observational evidence. However, the origin of this acceleration is still uncertain. It could either be a cosmological constant in the equation of general relativity, or a mysterious dark energy suggestive of physics beyond the standard model of particle physics (Planck Collaboration VI 2020).

To better understand the origin of the Universe’s accelerated expansion, the Euclid space mission was proposed and accepted in 2011 by the European Space Agency (ESA) as a medium-class mission. The spacecraft consists of a 1.2m telescope to observe 15000 deg² of extragalactic sky comprising the Euclid Wide Survey (Euclid Collaboration: 2022b) and a 50 deg² Euclid Deep Survey. It is equipped with two instruments to perform single broadband optical imaging with very high resolution in the VISible instrument (VIS), and simultaneous near-infrared (NIR) slitless spectroscopy and imaging with the Near Infrared Spectrometer and Photometer (NISP). These two instruments are part of the innovative design of Euclid that will acquire measurements of the shapes of 2 billion galaxies up to a redshift of z ~ 2.3 and 30 million spectroscopic redshifts (Laureijs et al. 2011) out to a redshift of z ~ 2. These independent measurements will enable us to reconstruct the expansion history of the Universe using probes in the core science areas of weak gravitational lensing (WL) and galaxy clustering (GC). Moreover, cosmological forecasts for Euclid predict an increase in the dark energy figure of merit by at least a factor of three when performing a combined analysis using cross-correlations of WL and GC measurements (Euclid Collaboration: 2020).

In order to reach its scientific goals, the following mission level requirements have been proposed (Laureijs et al. 2011): the density of galaxies brighter than I_E = 24.5 mag and detected with a 10σ confidence must be at least 30 galaxy arcmin⁻². Their corresponding median redshift should be z > 0.8 with an error on the mean redshift per bin below 0.002. The density of galaxies with spectroscopic redshift and an Hα flux > 2 × 10⁻¹⁶ erg cm⁻² s⁻¹ is required to be at least 3500 deg⁻². The control of systematic errors will be one of the most challenging aspects of the mission. For VIS, the point spread function (PSF) ellipticity must be known to an accuracy better than 2 × 10⁻³, and the stray light should remain less than 20% of the ecliptic zodiacal light.

Moreover, we aim for the contrast ratio of ghost images – images produced by multiple reflections of the optical system – to be below 10⁻⁴ (Cropper et al. 2014). For NIR spectroscopy, we require the purity of the spectroscopic sample (0.7 < z < 1.8 and Hα > 2 × 10⁻¹⁶ erg cm⁻² s⁻¹) to be above 80%, the completeness higher than 45%, and the spectroscopic resolution R ≥ 380 for the red grisms and R ≥ 260 for the blue grism. Finally, integration of external (EXT) survey data at the pixel level is essential for Euclid to reach the required photometric redshift precision per object: σ(z)/(1 + z) ≤ 0.05. These requirements are particularly important, especially in space experiments such as the Euclid mission, due to their elevated cost and the impossibility of performing repairs once the spacecraft is launched.

Given these tight requirements, a forward modelling approach has been adopted by the Science Ground Segment (SGS) as the main method to aid the development of the data-processing and science analysis pipeline (Laureijs et al. 2011). Within SGS, the Organisation Unit for Simulations (OU-SIM) is tasked with designing a simulation pipeline with added flexibility to deliver expeditiously realistic pixel-level images for the Euclid mission, following the release of a new instrument model. Our image simulations have two main objectives. The first is to serve as test data for the development of the SGS data-processing pipeline and the computing infrastructure before the arrival of real Euclid observations. Second, they are to be used to validate the stringent requirements that the SGS has in terms of performance and data quality. In this respect, several end-to-end Science Performance Verification (SPV) tests have been performed throughout the mission preparation, which allowed for the reproduction of certain instrumental issues to assess their impact and guide decision-making. Pixel-level simulations have been essential in the discovery of alternative solutions to critical problems, such as the non-conformity of one of the three red grisms for the NISP instrument (Euclid Collaboration: 2022b).

The production of simulations is generally planned in advance in orchestrated tests that we call Scientific Challenges (SC; Frailis et al. 2019). These exercises have been essential for the development of the SGS pipeline and the validation of science and technical requirements. Each challenge has leveraged new algorithms and infrastructure as well as improved instrument models derived during the construction of the spacecraft. Moreover, the outcomes of each SC have proven to be an effective indicator of the status and readiness of the SGS in addition to the various ESA reviews.

In this paper, we describe our simulation framework developed within OU-SIM with a focus on Scientific Challenge 8 (SC8). SC8 included for the first time the complete ground segment processing chain, enabling a full end-to-end simulation of the so-called level-1 raw images, level-2 detrended and calibrated data products, and level-3 core science-ready measurements (e.g. the cosmic shear and the galaxy two-point correlation function measurements, etc.). A majority of the data-processing was operated entirely by the SGS System Team without manual intervention from the pipeline development teams. The main goals of SC8 were (1) to simulate a large area of the Euclid Wide Survey along with the corresponding ground-based and calibration data for processing through the entire SGS pipeline; (2) to assess the capability of the SGS infrastructure to process a large set of data within a reasonable timescale; (3) to validate SGS operations under nominal conditions; (4) to validate the applicable requirements for the Ground Segment Implementation Review (GSIR); and (5) to benchmark the data quality reference for the third SPV test (SPV03).

All Euclid photometric imaging and spectroscopic channels were simulated, as well as seven EXT Euclid-supporting photometric imaging surveys spanning the northern and southern hemispheres. This extensive and challenging simulation effort resulted in 31 TB of raw output images for ~165 deg², which were subsequently processed and analysed with the Euclid SGS pipeline.

This paper is organised as follows. In Sec. 2, we begin with an overview of the drivers behind the development of our framework; namely, the components of the Euclid satellite and the organisation of the SGS. Our description of the OU-SIM workflow begins in Sec. 3, in which we explain our methods of creating synthetic galaxy and stellar catalogues, the inputs into the simulations. Sec. 4 follows with an introduction to the mission database, which contains the instrument and survey parameters, and thus provides the specifications for launching a coordinated production of simulations for different surveys (space- and ground-based). In Sect. 5, we describe in detail the instrumental features simulated in SC8. We outline our workflow for large-scale simulation productions in Sec. 6.1. The application and performance of our end-to-end framework in SC8 are detailed in Sec. 6. In Sec. 7, we summarise ongoing work and the latest improvements to the simulation pipeline. Finally, in Sect. 8, we present our conclusions.

2 The satellite and the Science Ground Segment

The Euclid satellite is composed of the Service Module (SVM) and the Payload Module (PLM). The SVM¹ includes the sunshield, the star trackers and gyros, the thrusters, the micro-motions and slews control systems, hydrazine and cold gas tanks, the Attitude and Orbital Control System (AOCS), the solar panel and electric power system, the thermal regulation system, and the downlink communication system. The AOCS jitter will meet the high-quality imaging requirements with a pointing dispersion smaller than 35 mas in each exposure, providing a stable attitude through the whole integration time.

The PLM (Gaspar Venancio et al. 2014) comprises the telescope, the PLM thermal control system, the Fine Guidance Sensor (FGS), and the VIS and NISP instruments (see Sec. 2.1 and Sec. 2.2, respectively, for a more detailed description). The design of the optical system is illustrated in Fig. 8 of Racca et al. (2016). The telescope follows a Korsch design with a 1.2-m diameter primary mirror providing an optically corrected and unvignetted field of view of 0.79 × 1.16 deg². The incoming light beam is split with a dichroic plate, reflecting the bluer part into the optical VIS instrument and transmitting the NIR light towards the NISP instrument. A more detailed description of the dichroic optical element is described in Euclid Collaboration: (2022c). This particular design feature allows simultaneous observations to be performed with both instruments.

Launch occurred in June 2023 on a SpaceX Falcon9 launch vehicle, which departed from Cape Canaveral. The mission will carry out a nominal survey that is expected to be completed in six years. The spacecraft will orbit the Second Sun-Earth Lagrangian point (L2) in a wide halo orbit. In addition to the 15 000 deg² Euclid Wide Survey, Euclid will also perform a Euclid Deep Survey with additional observations near both ecliptic poles for a total area of ~50 deg². The Euclid Wide Survey will avoid the Galactic and Ecliptic planes, with high dust extinction and high background, respectively. The VIS instrument will measure extended sources of m_AB = 24.5 in the visible band, I_E, with a signal-to-noise ratio of at least 10. For the NISP instrument’s NIR bands Y_E, J_E, and H_E, point-like sources will have a signal-to-noise of 5 or greater at a magnitude m_AB = 24. The Euclid Deep Survey will provide 40–53 times more observations (depending on the background level at each field location), reaching 2 magnitudes deeper than the Euclid Wide Survey. Further details are described in Euclid Collaboration: (2022b).

2.1 The VISible instrument

The VIS instrument is an optical imager featuring a mosaic of 36 (6 × 6), 4096 × 4132 pixel Teledyne e2v CCD detectors, which have been specially optimised for this mission. It operates in the single I_E broadband from 550 to 900 nm (equivalent to a combined r+i+z band) shaped by the reflection of the coating on the optical elements (the dichroic plate and fold mirrors, where FM1, FM2 have a hybrid metal-dielectric coating, and FM3 is silver coated) and the quantum efficiency of the CCD detectors (Cropper et al. 2014). The VIS focal plane covers a field of view of 0.57 deg² with a central pixel scale of $0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}1$ per pixel, resulting in an undersampled VIS PSF which limits the maximum optical resolution of the opto-electrical system. However, the exquisite image quality and exceptional temporal stability required of the VIS instrument and the telescope will allow measurements of galaxy shapes with enough accuracy to estimate the gravitational-lensing effect caused by the large-scale structures in the Universe on distant background galaxies.

2.2 The Near Infrared Spectrometer and Photometer

The NISP instrument was designed to provide broadband photometry (NISP-P) in three bands as well as low-resolution NIR spectra (NISP-S) divided into two wavelength ranges (Maciaszek et al. 2016, 2022). The NISP focal plane contains a mosaic of 16 (4 × 4), 2048 × 2048 pixel infrared detectors from Teledyne, with a field of view of 0.53 deg² and a pixel scale of $0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}3$. The NISP-P channel is equipped with three broadband filters (i.e. Y_E, J_E, and H_E) that span the range of 950 to 2021 nm (Euclid Collaboration: 2022c). Figure 1 illustrates the broadband transmissions for these filters along with the VIS photometric band. In combination with the VIS band and ground-based optical photometry, the three NISP broadbands will provide accurate photometric redshifts for billions of galaxies.

The low-resolution spectra acquired by NISP-S are obtained with grisms that disperse all the light without any slit, a technique called slit-less spectroscopy. The advantage of this method is that the design is simpler and more robust than a slit mechanism, requiring no target selection. However, slit-less spectroscopy presents two disadvantages: a higher background that reduces the signal-to-noise ratio and contamination of spectra caused by overlapping sources and intrinsic spatial extent. To optimise NISP-S, the spectroscopic channel contains in total four grisms providing a spectral resolution of R = 480 for a $0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}5$ diameter source. However, only three of the four grisms will be operational due to the non-conformity of RGS270 (Euclid Collaboration: 2022b). Two grisms (RGS000 and RGS180) cover the redder portions of the spectrum from 1206 to 1892 nm for the Euclid Wide Survey. The third is the “blue” grism (BGS000) with a wavelength range between 926 and 1266 nm, which will only be used during the Euclid Deep Survey. The two red grisms cover the same wavelength range with a rotated dispersion orientation (0° and 180°). In order to facilitate the decontamination of overlapping slit-less spectra, they will operate at their nominal positions as well as at orientations of −4° and 184°, respectively (Maciaszek et al. 2022). With the NISP-S low-resolution spectra, we can obtain spectroscopic redshifts enabling precise measurements of the distribution and clustering of galaxies.

2.3 The Euclid Science Ground Segment

The Euclid SGS is one half of the so-called Euclid Ground Segment with the Operations Ground Segment (OGS) comprising the other half (Racca et al. 2016). The SGS and OGS have different functions. The OGS is in charge of operating the spacecraft, analysing the telemetry, and performing the downlink transmissions of the data. As stated previously, the SGS is responsible for carrying out the entire data-processing and will perform key cosmological measurements, ultimately delivering the main scientific results.

There are ten organisation units (OUs) comprising the SGS data-processing pipeline². Each OU is responsible to define, design, and validate a specific analysis of the SGS workflow. Processing the massive data volumes within the Euclid SGS takes place in a distributed system across the nine Science Data Centres (SDC) in nine countries (Finland, France, Germany, Italy, Netherlands, Spain, Switzerland, United Kingdom, and the United States). While most of the SDCs have a High Throughput Computing (HTC) design, the underlying infrastructure varies across each SDC. The Euclid SGS has established common tools and guidelines to ensure homogeneity in terms of development, storage, and computing, independent of location (Frailis et al. 2019). We summarise the main elements of the common infrastructure system under which we develop the OU-SIM pipeline to operate within the SGS: (1) The Euclid Archive System (EAS) is a database containing all data products and metadata processed for the Euclid mission; this includes images, catalogues, processing status, etc. It also manages the transfer of data among the local storage facilities at each SDC³ (additional information about the EAS is provided in Williams et al. 2019). (2) A Common Data Model defines the format of the data and associated metadata to ensure that interfaces between pipelines and with the archive are stable. (3) An Infrastructure Abstraction Layer (IAL) orchestrates the data-processing and adds a common layer allowing jobs to run independently of the underlying IT infrastructure. It defines the Pipeline Processing Order (PPO) which describes the elements, configuration, and inputs of a specific processing task. (4) A common Euclid Development ENvironment (EDEN) establishes the set of libraries and associated versions to be used by any of the Euclid software. This prevents inconsistencies or changes in the functionality of different libraries between development and production. (5) The COllaborative DEvelopment ENvironment (CODEEN) is a continuous integration and continuous delivery (CI/CD) platform that automates the building, unit testing and distribution of all the scientific software in the SGS. Source code is extracted from a Version Control System (i.e. Gitlab) and run through a CI/CD pipeline to be finally deployed on a distributed file system (CernVM-FS) available on all SDCs. This system design allows SGS operations to be performed smoothly and efficiently across the nine SDCs, providing the extra advantage of increased computing power and storage capacity.

2.4 The simulation framework

The strict requirements of this mission demand extremely precise simulations with exquisite detail, capturing all known instrumental and environmental models. This is to ensure that all critical data-processing components are tested and validated prior to the launch. Producing image simulations at this high level of quality not only depends on active engagement with the instrument teams, it also requires leveraging our interfaces with all OUs. Each processing function (PF) has its own respective requirements that the simulations must satisfy as well as additional insights and tools to aid the development and validation of our framework. Evidently, liaisons had to be established with the Cosmological Simulations Science Working Group, the main supplier of galaxy catalogues for Euclid’s core science program.

The organisation of the SGS simulation pipeline within the scope of the SGS infrastructure is depicted in Fig. 2. The True Universe catalogues contain all input sources and their corresponding parameters, spectra and shape, and are described further in Sec. 3. The instrument models and the reference survey are introduced into the pipeline from the Mission Database, which we explain in Sec. 4. Within the so-called simulation processing function (SIM PF), there are four image simulators, capable of accurately reproducing imaging and spectral data for the VIS, NISP-P, NISP-S, and EXT survey channels, including the instrumental effects of the two Euclid instruments and of the ground-based observations; we present each of the four simulators in Sec. 5. The operation of the simulation framework is orchestrated by the SimPlanner via a simulation request and is detailed in Sec. 6.1.

3 The True Universe

The “True Universe” (TU) catalogues⁴, consisting of stars and galaxies, are input data products to the SIM workflow. It is paramount that these catalogues include many realistic features as they serve as the basis for assessing the scientific performance and validating the compliance to the requirements at the different stages of the mission. In the following subsections, we describe, respectively, the four types of galaxy catalogues used in the simulations, the star catalogue built from real and simulated samples, and the common spectra and thumbnail libraries that allow us to transform galaxy and star parameters into fluxes and images.

3.1 Galaxy catalogues

Several types of galaxy samples were used in the simulation (see Table C.1): galaxies in the redshift range (0 < z < 2.3) referred to as standard Flagship galaxies (Std gals), QSOs, referring to a population of quasi-stellar objects at high redshift (6 < z < 14), High-z gals, denoting a high-redshift population (6 < z < 10) of Lyman break galaxies (LBGs), and SL, representing a catalogue of strong lensing systems.

3.1.1 Euclid Flagship mock galaxy catalogues

The Euclid Flagship v1.0 Simulation (Potter et al. 2017) is one of the largest cosmological N-body simulations of the Universe ever produced, with (12 600)³ particles over a simulation box of 3780 h⁻¹ Mpc, leading to a mass resolution of m_p = 2.4 × 10⁹ h⁻¹ M_⊙. Such an unprecedented volume and resolution was required to validate the mission performance at full scale. The simulation was run with cosmological parameters⁵ similar to those of the Planck 2015 cosmology (Planck Collaboration XIII 2016). The 2 trillion dark matter particle simulation was run with PKDGRAV3 (Potter et al. 2017) on the Piz Daint supercomputer at the Swiss National Supercomputer Center (CSCS). An all-sky particle light-cone up to a redshift of z = 2.3 was produced in real time. The dark matter halos were identified from the dark matter particle light-cone using the ROCKSTAR halo finder (Behroozi et al. 2013), an adaptive hierarchical refinement of a friends-of-friends method able to track substructures and relations with the parent dark matter halos. Following the approach described in Fosalba et al. (2015), all-sky lensing convergence maps were built, which enable weak lensing effects to be included in the final galaxy catalogue.

The mock galaxy catalogue was built implementing an improved Halo Occupation Distribution technique (Carretero et al. 2015, Castander et al. in preparation) with SciPIC, the Scientific pipeline at Port d’Informació Científica (Carretero et al. 2017), developed to efficiently generate mock galaxy catalogues using as input a dark matter halo population. SciPIC runs on top of the PIC Hadoop cluster using Apache Spark. We present in Fig. 3 an example illustration of the mock galaxy catalogue, where a small region of the light-cone is shown, and one can distinguish by eye the emergence of the large-scale structure of the Universe.

For SC8, we selected galaxies up to an AB magnitude of I_E = 26 for the main Euclid Wide Survey, and I_E = 28 for the Euclid Deep Survey, which is two magnitudes deeper than the detection requirement. This was essential to maintain the same sources in all instruments (i.e. space and ground) and bands in order to preserve the photometry across the different channels for photometric redshift consistency even if there are sources below the detection threshold in some channels. Lensing measurements are also affected by very low signal-to-noise sources below the detection threshold (Hoekstra et al. 2017; Euclid Collaboration: 2019b).

Particular care was taken to model the Hα emission of the galaxies (following model 3 in Pozzetti et al. 2016), a line that is particularly important for NISP to determine spectroscopic redshifts. For the NISP-S simulations, we included only sources with H_E ≤ 23 or sources with an Hα emission brighter than 10⁻¹⁶ erg s⁻¹ cm⁻², as the continuum of sources H_E > 23 is never detected in the spectroscopic channel. Additionally, the size and shape distributions of the galaxies were updated and improved in SciPIC with respect to previous implementations in the MICE simulation described in Carretero et al. (2015). The morphological parameter distributions from the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey (CANDELS) (Grogin et al. 2011), and Hubble Space Telescope (HST) surveys were used to produce highly accurate and realistic galaxy samples (see Sect. 3.4.1).

The final mock galaxy catalogue contains the following information that allows the pipeline to reproduce its flux at the pixel level: (1) ID: A unique source ID that also contains information about the parent dark matter halo. (2) Positional coordinates: The source location in equatorial coordinates (RA, Dec) and the lensed positions due to gravitational deflection. (3) True Universe kind: An identifier of the object kind from the various TU inputs such as Flagship central or satellite galaxy, QSO, High-z and strong lensing, as different sources are simulated differently. (4) Redshift: The true and observed redshift of the galaxy, different due to the peculiar velocity. (5) Reference magnitude: An absolute, instrinsic and apparent magnitude of the galaxy at a given band. This was used to scale the spectrum at a given brightness. The reference band depends on the object type as some sources are best modelled in the visible bands (stars and low redshift galaxies), while other sources are better characterised in infrared bands (high redshift sources). (6) SED template: A spectral energy distribution (SED) template with dust extinction assigned by sampling the COSMOS galaxy catalogues (Bruzual & Charlot 2003; Polletta et al. 2007), where galaxies are matched to the observed ones using redshift, absolute magnitude, and colour. An additional parameter describes the extinction law to be applied. This was used to reconstruct the spectrum (detailed in Sec. 3.3). (7) Emission line fluxes: The fluxes of Hα, Hβ, [O II], [O III], [N II] used in spectroscopic redshift determination. (8) Morphological parameters: Simulating the light profile of a galaxy requires the bulge and disc half-light radius, the ratio of the flux in the bulge component to the total flux (often written B/T), the Sérsic index of the bulge profile, the axis ratio defining its intrinsic ellipticity, the inclination angle of the disc, and the position angle with respect to the north (see Sec. 3.4, which describes how these parameters are used to reconstruct galaxy shapes). (9) Lensing parameters: The shear and convergence values, γ₁, γ₂, and κ, were used to distort galaxy shapes and determine their flux magnification. These three parameters were derived from the Euclid Flagship dark matter lensing and convergence maps. (10) MW extinction: The optical extinction of the Milky Way in the V band. This was derived from the Planck thermal dust maps at the observed (lensed) coordinates of the galaxy.

With this list of parameters, we can render all the galaxies in the catalogue in any of the simulation channels, both photometric and spectroscopic, with all their complex instrumental effects. Additionally, we computed the true reference fluxes for each galaxy in all the passbands of interest for Euclid and the ground-based EXT surveys, including magnification.

3.1.2 Other extragalactic sources

High-redshift sources. Euclid’s survey area and depth coverage is expected to yield an unprecedented number of new sources at high redshift, which will be catalogued and analysed in its legacy science program: the Primaeval Universe. These are very distant galaxies not contained in the Flagship dark matter run, as its light cone reaches only a redshift of 2.3. The Primaeval Universe Science Working Group provided a sample of 11 million high-redshift galaxies and 2 million quasars at redshifts greater than 6, as is described in Euclid Collaboration: (2019a). They were added to the TU set as an additional set of sources to be simulated on the image, with unlensed magnitude cuts H_E < 26 and H_E < 25 for galaxies and quasars, respectively.

Strong lensing sources. Euclid is expected to also observe tens of thousands of strong lensing sources (Collett 2015). For SC8, the Strong Lensing Science Working Group provided a template set of 801 sources in the form of high-resolution image thumbnails that were injected to the simulation to train algorithms and classify them more efficiently. Strong lensing sources were simulated with the GLAMER ray-tracing code (Metcalf & Petkova 2014), based on the Flagship halo mass properties, and assuming HST-like morphologies for the lensed background sources following the recipe provided in Metcalf et al. (2019).

3.2 Stellar catalogue

Observations of stars are an integral part of image processing and analysis. They serve as reference catalogues for all astrometric and photometric calibrations, and are essential for performing the precise PSF modelling required for all lensing measurements. Conversely, they add confusion to the classification of sources and produce numerous complications, such as blending, bleeding, ghost reflections, and scattered light. This is particularly true for the L- and T-type brown dwarf stars, which contaminate high-redshift galaxy and quasar detections. It is critical, then, to have a realistic and accurate distribution of stars at each location of the sky with representative density and stellar types.

3.2.1 Besançon stars

As Euclid aims to observe thousands of square degrees in the optical and NIR down to a I_E magnitude of 24.5, we require a synthetic model that accurately represents the distribution of stars in the Milky Way. We adopted the Besançon model (Robin & Creze 1986; Robin et al. 2012)⁶ as our main stellar population synthesis model of the Galaxy. The model provides a realistic representation of populations in the disc and halo down to M-type stars, as well as photometry in various bands. The scaling of the SED for the spectra reconstruction is performed with the 2MASS H band. The model provides the surface temperature, gravity, and metallicity for each star, which we used to derive the stellar template from the Basel 2.2 stellar library (Lastennet et al. 2002). Further details of the library and how we reconstructed spectra are explained in Sec. 3.3. We ran an all-sky Besançon model simulation down to H < 26, avoiding the Galactic plane as Euclid will not observe there and it contains too many stars at those magnitudes. Even though the NISP instrument has a defined magnitude limit of H_E ~ 24, we simulated a deeper catalogue to include sources below the detection threshold into the pixel-level images, as these are also relevant in altering the photometry of other sources of interest, such as the main galaxy sample. This resulted in a massive final stellar catalogue containing 3 billion stars with the following set of parameters: (1) ID: A unique TU ID, which is non-identical to the ID used for galaxy sources. (2) Sky coordinates: The equatorial coordinates of the star in RA and Dec. (3) Distance: The radial distance of the source in kpc. This is used to scale brown dwarfs of L- and T-type for which the reference magnitude is given in absolute magnitude. (4) Reference magnitude: The apparent magnitude in the Vega system for the 2MASS H band, which we use to scale the template spectra to its apparent brightness. (5) Extinction: The Milky Way optical extinction in the V band, derived from 3D dust model maps (Schultheis et al. 2014). Like the galaxy catalogue, true fluxes in all Euclid and EXT bands are also computed for the stellar catalogue. The density map of the Besançon simulation run can be seen in Fig. 4.

3.2.2 Augmented Tycho2 catalogue

The design of the Euclid Wide Survey was performed using the reference zodiacal light model and straylight levels from the brightest stars near the galactic plane. For consistency, the brightest end of the stellar catalogue had to be replaced with real stars from an all-sky catalogue. We chose the Tycho2 stellar catalogue (Høg et al. 2000), as we could use the Pickles procedure described in Pickles & Depagne (2010) to match stellar types for each source in the catalogue and then to our reference Basel 2.2 library (Lastennet et al. 2002). Our catalogue contains 2.5 million of the brightest stars in the sky down to V magnitude of 11.

3.2.3 Brown dwarf stars

One of the challenges in identifying very distant quasars is differentiating them from the point-like brown dwarf stars that have a red spectrum of similar NIR colours. To enable the testing of algorithms designed to perform more complex QSO identification, we supplemented our stellar catalogue with MLT type stars. While M-type stars were already included in the Besançon sample, the redder LT stars were not, so we added new templates to the Basel 2.2 library and included the brown dwarfs following recipes described in Euclid Collaboration: (2019a). Figure 5 shows the resulting colour-colour diagram of our sample of LT stars in addition to the various stellar types described previously.

3.2.4 Gaia realisation

The first prototypes of the SGS pipeline used the True Stellar catalogue as the astrometric reference for the calibration algorithms. However, this idealised catalogue will not be applied in the astrometric calibration of real observations; instead, Gaia will be the reference catalogue used. With the astrometric and photometric performance described in the Gaia DR2 (Gaia Collaboration 2018), we could replicate a realisation of a Gaia catalogue in our TU sample, using realistic astrometric and photometric errors based on the magnitude and stellar type. Proper motions are not included in this simulation. For each star in the catalogue, we note the position and magnitudes in G, BP, and RP bands⁷ down to G ~ 21, together with their respective errors. This method yields a more realistic Gaia-like astrometric reference catalogue, which is then provided to the respective calibration pipelines for processing VIS, NIR, and EXT data.

3.3 Spectral energy distribution library

As we shall explain in Sect. 5, the VIS, NISP-P, NISP-S, and EXT pixel simulators are implemented as separate independent codes. However, developing different types of software that implement common routines poses some risks that could result in bugs and incompatible results. To minimise possible sources of error in the simulations, we constructed a common spectra reconstruction library tool called SimSpectra that reconstructs observed spectra from the parameters in the catalogue and the corresponding template library. The VIS, NISP-P, and NISP-S pixel simulators reconstruct the incident spectra every time a given source needs to be rendered. This allows the full chromatic information to be used precisely when simulating instrumental effects such as a wavelength-dependent PSF or quantum efficiency. Although it is computationally inefficient to reprocess the spectra for each source in the different dithers and channels, it provides greater practicalities than storing and transferring large catalogues of spectra. Reconstructing the spectra is a quick process (when optimised) that enables low-memory jobs to perform the processing in parallel under distributed architectures. Reconstructed spectra are only stored as a temporary validation product for NISP-S simulations.

In contrast to the Euclid simulators, simulations for the EXT imaging surveys (described in detail in Sect. 5.3) as well as Gaia (G, BP, RP) are less complex and use pre-computed band fluxes; this is to avoid using spectra in the process of simulating pixels. Indeed, due to their large fields of view, ground-based survey simulators usually have to include more galaxies than the Euclid simulators. Computing spectra for each EXT survey would be computationally expensive and impractical. Instead, we computed the exact values in a matrix, sampling various points in the parameter space, and performed a multi-dimensional interpolation to obtain the integrated fluxes in all bands. This faster interpolation method has an accuracy of 0.005 mag root-mean-square (RMS) error in comparison to the full spectra reconstruction algorithm, which is sufficient for all SGS validation applications (Carretero et al. 2015).

3.3.1 Galaxy spectra reconstruction

Our base galaxy template library is composed of the COSMOS SED templates used in Ilbert et al. (2009) that comprise Bruzual & Charlot (2003) and Polletta et al. (2007). The SEDs and the intrinsic dust extinction laws were selected from Prevot et al. (1984) and Calzetti et al. (2000). We used the luminosity, g − r rest frame colour, and the redshift of each galaxy in the Flagship catalogue to assign the SED and dust extinction of the closest COSMOS galaxy in the Ilbert et al. (2009) catalogue in the luminosity-colour-redshift space. In order to avoid discrete distributions in the observed photometric properties, we applied scatter to the assigned closest COSMOS template, generating a random realisation of the value of the SED. The closest SED was identified as an optimised⁸ function based on colour, luminosity, and redshift. We assigned the two SEDs closest to the resulting realisation value, weighted by their distance. Each galaxy SED was then constructed as a linear combination of these two SED templates (see Appendix B).The galaxy template set used for these simulations is shown in Fig. 6.

We then assigned a star formation rate (SFR) to each galaxy from its unextinguished rest-frame UV luminosity (computed from its SED). These relations have been derived at low redshift. Most of them depend on physical processes that are not expected to depend on redshift, and therefore have been extrapolated to higher redshifts. We used the Kennicutt 1998 relation (Kennicutt 1998) to assign an Hα luminosity from the SFR with some scatter. We finally corrected the Hα luminosity to make the global Hα luminosity distribution resemble the Pozzetti et al. (2016) models. We assigned the luminosities of the other most prominent lines from their SFR and Hα luminosities using observed correlations. An example of a reconstructed spectrum (including the continnuum SED and the emission lines) from the catalogue parameters is shown in Fig. 7.

3.3.2 Stellar spectra reconstruction

As is stated in Sec. 3.2, our stellar library was generated from the Basel 2.2 (Lastennet et al. 2002) template set. We assigned an SED from both the Besançon model simulation and the Tycho2 catalogue to a particular stellar template. Reconstruction of the stellar spectra is much simpler than for galaxies, since there is no redshift, K correction, or lensing magnification in the reconstruction process. There are only two steps to calibrate the template to the incident calibrated spectrum for each star. First, we needed to scale the flux to our reference H 2MASS band. Second, we corrected for the Galactic extinction. The Tycho2 stars were not corrected for extinction because their flux is observed, and therefore already contains the extinction factor. In contrast, as Besançon stars are simulated, we computed an accurate reddening with the radial distance information provided by the model and the 3D dust model from Schultheis et al. (2014). A detailed description of our implementation of stellar spectra reconstruction is given in Appendix B.2. Finally, the stellar library also contains 365 LT star templates that enable the additional brown dwarf stars to be assigned an SED, as the Basel 2.2 library does not contain this class of stars.

3.4 Galaxy profile generators

For our suite of instrument simulators, we built a package called SimThumbnails that transforms galaxy morphological parameters to pixelised galaxy light profiles. In addition, it performs the rendering of the light profiles at the exact pixel scale for each instrument using the ‘World Coordinate System’ (WCS) information of the target image. The core of the library is based on GalSim 2.3.3 (Rowe et al. 2015), which allows plenty of flexibility to simulate different profile configurations. Similar to the SimSpectra library, the SimThumbnails library implies rendering the same galaxies for every pixel simulator, which is inefficient computationally. However, it is a trade-off between pre-computing and storing images for all objects. In addition, every simulator can tune the library parameters to the instrument pixel or sub-pixel scale (see, e.g. Sect. 5.2.2).

3.4.1 Bulge and disc model

For the standard Flagship galaxies, we adopted two Sérsic profiles to trace the bulge and disc components. First, we defined the bulge as a Sérsic profile: $$I_{\text{bulge}}(r)\propto \exp (-b_n{\left(r/r_{\mathrm{e}}\right)}^{1/n}),$$(1)

where r_e is the half-light radius and n is the Sérsic index given in the Flagship catalogue. The b_n parameter was calculated to give the correct half-light radius given n. The Sérsic index was determined from an empirical fit to the HST-CANDELS data (Grogin et al. 2011; Koekemoer et al. 2011; Dimauro et al. 2018), HST-GOODS-South data (Welikala & Kneib 2012), and HST-COSMOS data (Laigle et al. 2016). The bulge model, I_bulge(r), was then sheared to obtain the bulge minor-to-major axis ratio due to its intrinsic ellipticity as given in the catalogue.

Next, we initialised the disc component with an exponential thick disc inclined model (van der Kruit & Searle 1982; Bizyaev 2007). The exponential inclined model is a particular Sérsic profile of index n = 1 characterised by three parameters: the inclination angle (where 0° refers to a face-on galaxy and 90° would indicate it is positioned edge-on), the half-light radius, and the scale height of the disc. We chose a fixed height-to-radius ratio, h_s/r = 0.1, that reproduces observations well. The relation between the inclination angle and the apparent disc axis ratio, b/a, is $${\theta }_{\text{inclination }}=\arcsin \sqrt{\frac{1-{\left(\frac{b}{a}\right)}^2}{1-h_{\mathrm{s}/\mathrm{r}}^2}}.$$(2)

The advantage of this profile is that it accurately replicates the thickness of the disc, which the Sérsic profile does not properly represent when the galaxy is edge-on. Then, we adjusted the flux of the disc relative to the bulge components with the bulge-to-disc ratio parameter given in the Flagship catalogues. The distribution of this parameter reproduces the ratio measured in the data detailed above (Cardamone et al. 2010; Grogin et al. 2011). Finally, the bulge and disc models were summed together and rotated to a position angle on the sky (with a convention of north up, increasing towards the east) given in the catalogues. Figure 8 shows an example of this bulge and disc model for various inclination angles.

3.4.2 Gravitational lensing and optical distortions

We used GalSim to apply the gravitational lensing effect to every galaxy (i.e. the summed bulge and disc components) based on the shear, γ₁, γ₂, and convergence, κ, parameters given in the Flagship catalogues. GalSim requires the following derived parameters: the reduced shear parameters, g₁ and g₂, responsible for the distortion effect, as well as the magnification factor, μ, defined, respectively, as $$g_i=\frac{{\gamma }_i}{1-\kappa }$$(3)

and $$\mu =\frac{1}{(1-\kappa {)}^2-({\gamma }_1^2+{\gamma }_2^2)}.$$(4)

While GalSim can only compute a reduced shear for which $g_1^2+g_2^2<1$, this is not a limitation as we are mostly interested in the weak lensing regime which falls well below this limit. As was stated above, lensing magnification alters the size of a galaxy. As True Universe fluxes already include magnification, we have to apply the inverse of the magnification factor on its incident spectrum (as is explained in Sec. 3.3), so that its total integrated flux matches that given in the catalogues.

The final shape of the galaxy is critical for estimating the weak lensing signal. Instrumental distortions add up to the lensing distortions described above. We accounted for optical distortions of the field as defined by the plate WCS. At every particular position in the focal plane, we associated with each GalSim galaxy object a WCS 2 × 2 local Jacobian matrix that was subsequently used during the simulation of the galaxy on the image.

3.4.3 Flux scaling and truncation

In theory, analytic expressions for the bulge and disc profiles extend to infinity. However, in practice, only limited size stamps can be pasted on the image. This size is automatically computed by GalSim, such that the flux lost outside the stamp is equal to the ‘folding threshold’ parameter, described in Rowe et al. (2015). The pixel values are then increased to match the desired flux. In our pipeline, we set the folding threshold to 0.5%, the default GalSim value. In most cases, where the Sérsic index is high, the stamp contains almost all the flux as it is concentrated in the centre. For low-n Sérsic profiles where the light decays slowly, the flux loss can be closer to 0.5%; however, in such extended profiles it will be very difficult to recover the total light with photometric measurements.

3.4.4 Point spread function convolution

The convolution of the galaxy profile by the PSF was performed in Fourier space with the FFT method. In GalSim, the profile is centred in the middle of the central pixel by default. Yet, the true position of the object may lie in a slightly shifted fraction of it. The offset to the central pixel was computed and provided for an accurate positioning of the source.

Depending on the instrument, the simulator pipeline allows for a polychromatic PSF model that varies in wavelength, such as in the VIS instrument. Due to limited information in the Flagship galaxy mock catalogue, the galaxy profile is rendered without colour gradients. Therefore, the polychromatic PSF was convolved only once with the galaxy stamp, and collapsed along the wavelength direction. This provides very accurate PSF convolution based on the spectra of the source.

3.4.5 Optimisation

The computing time of our simulations was driven by the processing of the galaxy stamp and PSF convolution. Therefore, optimising these two steps allowed us to perform larger and more complex simulations. We found three ways to speed up this process: first, we enabled a mode whereby only the size of the stamp was provided without requiring any complex calculation; this permitted us to quickly discard sources that did not overlap with the detector pixel array (from the radial search of the catalogue to the rectangular shape of the pixel array). As a second step, we rounded the variable Sérsic index in the galaxy bulges to the first decimal digit, as the computation of the 2D Sérsic profile is computationally expensive. With this approximation, many galaxies share the exact same profile and a look-up table in GalSim, allowing us to reuse the internal 2D profiles to speed up the rendering process, together with the specific shear and PSF model. The impact on the simulated profile is below the required precision on the flux. And finally, faint galaxies entering in the undetected background sample $(I_{\mathrm{E}}^{\text{limit }}<I_{\mathrm{E}}<I_{\mathrm{E}}^{\text{limit }}+2)$ were replaced by single component models that require less computation. These changes have almost no effect on the quality or the image analysis, reducing the computation time by several orders of magnitude.

When profiles deliver a very large galaxy stamp (larger than a few thousand of pixels in diameter), the memory necessary to perform the convolution exceeds the limit established by the environment and raises an exception. To handle such cases, we implemented a down-up sampling iterative process. When the memory limit is reached, we down-sample the actual stamp by a factor of 2, providing an equivalent WCS with a scale multiplied by that same factor. Every time the memory limit is reached, we further down-sample the stamp (although most issues have been resolved during the first iteration). Once the stamp is successfully imaged, we up-sample the image with a bi-quadratic interpolation. We minimise the residuals for these types of profiles using this approach. Even though some degradation can occur during the down-up sampling process, this event is rare and happens only for very large Flagship galaxies (above 10” effective radius) with a high Sérsic index. It is not a concern as these sources are not among the list of targets for the main science goals of Euclid. We present an example of the performance of the down-up sampling process in Fig. 9.

High-z galaxies are drawn with a single Sérsic profile component of index n = 1.5. QSO sources are expected to be point-like and we render them using a PSF profile. Finally, strongly lensed galaxies require extra complexity in rendering that is implemented via the generation of stamps at $0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}05$ resolution with the GLAMER code (see Sect. 3.1.2). The stamps are then resampled at the instrument resolution by each simulator using the default ‘Quintic’ interpolant in GalSim.

4 Mission database

In such a complex mission, having a robust data and configuration management system is essential. The instrumental models, reference sky maps, telescope parameters, survey plan, etc., will all evolve through the different phases and maturity of the mission. The Euclid mission Database (MDB) is a single reference repository designed for the Euclid mission system; it provides a temporal record of the official representation of the system consistent with the set of reference models. As such, the MDB provides a centralised version control and distribution system of parameters, which is required by all SGS components.

The MDB has the following set of configurations which we select from and assign to the model that we want to simulate: (1) as required: This specifies the worst-case values still within the related requirements. These derive from the top-level requirements that ensure Euclid can achieve its scientific goals. (2) as designed: This provides the expected performance of a given model with the current design. The model must perform better than the required limit. (3) as built: At this level, the model has been measured in built parts, ideally around the design value. (4) current best estimate (CBE): This replaces the ‘as built’ model with newer or more accurate measurements that contain our best knowledge of the model. Ideally, the simulation will be done with all CBE models, which best represent the status of the system. It is foreseen that after commissioning and in-flight calibrations, CBE will be updated with more accurate instrument models. (5) as simulated: This is a particular configuration used for testing, where the model is defined with a given value, which may be unrelated to a particular measurement or design. However, we generally avoid implementing unproven, hypothetical models.

When preparing a production of simulations for the various instruments, we define a Mission Configuration (MC), which gathers a subset of models from the MDB with a unique status on each parameter. The MC is stored in the EAS along with the simulation data products to provide users knowledge of the true models used as input.

4.1 Models and parameters

The MDB models and parameters allow us to emulate Euclid (and EXT surveys) operations during sky observations. As explained previously, the models in the MDB continually evolve with the construction and testing of the instruments and telescope. At the time of pre-launch, we had a sufficiently accurate model of system performance derived from ground tests. Therefore, we prioritised the models labelled with ‘current best estimate’ (CBE), as these would provide the closest representation of the spacecraft during operations.

The MDB is organised in sections, each covering a specific component of the mission. The Environment section contains physical constants, ensuring all calculations use the exact same reference values for consistency. Other Environment models assumed in the definition of the survey include the zodiacal light scattered from interplanetary dust in the Solar System, all sky out-of-field straylight maps induced by stellar light, intergalactic dust extinction maps, and the cosmic ray energy distribution.

The Space Segment section contains all known models for the PLM, SVM, VIS, and NISP instruments. The SVM subsection lists models associated with the spacecraft such as the dither steps or slew duration. The PLM subsection stores all telescope related information like wavefront errors, collecting surface areas, or plate scales, as examples. The NISP and VIS instrument sections provide all parameters and models describing the instruments such as detector layout, quantum efficiency, read noise, Charge Transfer Inefficiency (CTI) models, wavelength dispersion and distortions, etc.

The Survey section defines the Euclid Wide Survey, the Euclid Deep Survey, and the calibration surveys, as well as the calibration source SEDs for objects such as Vega, a white dwarf, and a planetary nebula. We describe briefly in the following section the reference survey which we use to emulate realistic Euclid observing sequences.

4.2 The reference survey

The Euclid reference survey (Euclid Collaboration: 2022b) was defined to detail the set of fields to be observed within the entire Euclid wide footprint, the dither steps within each field, time of observation of each pointing as well as the various calibration fields and the deep fields during the six-year mission campaign. With this information, OU-SIM can reproduce a realistic sequence of observations covering the area to be simulated. Figure 10 shows an image of the fields in the reference survey that cover the Euclid Wide Survey footprint. We outline in red the region denoting the SC8 footprint.

Fields and calibration frames. Euclid has defined four dithers inside each field, where each pointing results in I_E, Y_E, J_E, and H_E images, and one spectral exposure. The NISP-S dithers alternate different grisms at different dispersion angles that enable spectral decontamination. We show the reference observing sequence (ROS) we implemented for SC8 in Fig. 11. We note, however, that the ROS shown is not the definitive one; for example, the true ROS is scheduled to have a second short VIS exposure. Nevertheless, simulating the ROS provides a realistic representation for spatially varying instrumental effects. Moreover, multiple exposures allow us to cover the detector gaps, reach the desired depth, and correct for transitory effects such as cosmic rays. We implemented the dither slews following the pointing accuracy model of the spacecraft with minimal step jumps plus the associated pointing uncertainty of 11″. The current dithering model (Markovič et al. 2017) follows an S shape and implements the following minimal slews:

• Dither step 1: ΔX_sc = 50″, ΔY_sc = 100″;

• Dither step 2: ΔX_sc = 0″, ΔY_sc = 100″;

• Dither step 3: ΔX_sc = 50″, ΔY_sc = 100″;

plus a random error of 11″ at 3σ after the minimal step. The coordinates X_sc and Y_sc correspond to the X and Y axis in the spacecraft reference frame.

4.3 External surveys

The MDB also contains survey, instrumentation, and bandpass information for each EXT survey (i.e. DES, Rubin, CFIS, JEDIS, Pan-STARRS, and WISHES, which will supplement the Euclid mission; see Sec. 5.3 for more information). In detail, a survey table containing a list of selected pointings along with additional survey parameters (such as magnitude zeropoints, sky brightness, position angle, seeing FWHM for constant PSFs or PSF model files for variable PSFs) for each EXT survey are stored in the MDB. The instrument file describes the detector layout in the focal plane for each camera, as well as characteristics of the CCDs such as the pixel size, read noise, well capacity, and the gain per CCD amplifier. Finally, for each respective EXT survey, the MDB keeps a record of the total throughput curves per filter including the atmospheric transmission.

In the following section, we shall present the various detector, atmosphere, and optical models used for producing realistic pixel images for each of our survey channels: VIS, NISP-P, NISP-S, and EXT. In Appendix A, we mention briefly the methods and resources for acquiring each of the respective transmission curves. More details on the construction of SC8 EXT survey table are shown in Appendix C.4.

5 Instrument simulators

In this section, we describe the implementation of all four simulator codes in the OU-SIM pipeline: ELViS (Euclid VIS), Imagem (Euclid NISP-P), TIPS (Euclid NISP-S), and SIM-EXT (EXT ground-based surveys). These image simulators were originally built for various purposes, in particular, to provide the first evaluations of the mission performance. However, they have significantly evolved to become the main pixel simulators of the SGS. All simulators integrate the TU star and galaxy catalogues, the common SimSpectra⁹, and SimThumbnails libraries, the reference survey, and the various models in the MDB.

5.1 VIS simulator

ELViS is the Euclid VIS simulator capable of accurately reproducing the data that will be obtained by the VIS instrument during the Euclid mission operations, including the required calibration frames; for example, bias, flats, and self-calibration sequences. However, the biggest challenge for this simulator is emulating the optical response of the system and the instrument with the precision required. Euclid plans to measure with unprecedented precision the weak lensing signal of billions of galaxies. To achieve such a task, the VIS instrument needs to deliver extraordinarily good and stable image quality. The VIS simulator is required to deliver images with sufficient precision, such that simulation or model limitations do not drive the quality of the image. Moreover, it is the instrument performance as well as the shear measurement algorithms which define how well we can estimate the shape of the galaxies.

Input sources. ELViS includes all TU stars, comprising the real Tycho2 bright stars, the model Besançon stars, and the MLT type brown dwarfs. Similarly, it renders the TU galaxies from the Flagship simulation, high-z quasars and galaxies as well as strongly lensed galaxies. Recently, we included Solar System Objects (SSO), asteroids that due to their fast and nearby motion appear as traces in the VIS focal plane. Finally, the simulator includes cosmic rays (CR) during integration and readout, which leave substantially different traces. CRs are critical in this simulation due to their high density resulting from the higher levels of radiation in space and to the CTI (explained later in the Detectors subsection). They have a significant effect on image quality, and therefore, impact the ability to measure lensing parameters with high precision. Cosmic rays are simulated in the VIS channel as single pixel thin straight tracks generated from samplings of the posterior probability distributions (PDFs) of the track length and energy deposit. These PDFs are computed using a complex Monte Carlo cosmic rays simulator code STARDUST (Rolland et al. 2007) developed at The National Centre for Space Studies (CNES). The STARDUST code input data are the geometrical and physical description of the detector structure, the environment particle spectra, and an aluminum equivalent thickness table describing the shield around the detector. Full scale images are generated out of which the PDFs are evaluated.

Background models. There are five main contributions to the background in the VIS channel. (1) Zodiacal light, a diffuse background from sunlight scattered off interplanetary dust. This is based on an MDB model that provides the intensity as a function of ecliptic latitude and its average spectrum. (2) Thermal irradiance, a background signal due to radiation from the heat of the various elements of the telescope and instruments. (3) Diffuse scattered light, unwanted reflections at the telescope and optical path from the stellar sources in the field that illuminate the focal plane resulting in an increase of the background. The intensity of this signal is related to the density of bright stars in the field of view and thus increases at lower Galactic latitude. (4) In-field straylight, while similar to diffuse scattered light, contributes to an increase in the localised background due to very bright stars following a specific path in the optical system. (5) Optical reflections, also known as ghosts, an out-of-focus image of an on-sky source inside or in the immediate vicinity of the instrument field of view. It mainly depends on the light path through the dichroic and telescope baffling and the position and brightness of the sources. While the first three components are constant across the focal plane and increase in value depending on the telescope’s thermal state or its pointing in the sky, the last two create abrupt changes in the background which complicate the calibration and photometry.

PSF. As was mentioned, VIS simulations require great care in the reconstruction of the PSF, as it distorts the true shape of the galaxies. This needs to be simulated accurately to estimate the weak lensing signal with the precision required. In the VIS channel, we have four main contributions that affect the final PSF. First, we have the optical PSF, which varies as a function of position in the focal plane and wavelength. The model for the PSF is composed of an array of 240 × 240 positions in the focal plane, each at eight wavelength positions, comprising a total of 460 800 PSFs of 300 × 300 pixels. Figure 12 shows an example of a single model at one wavelength. These were constructed using SHE_PSFToolkit, a software developed by the Organisation Unit for Shear Data (OU-SHE), which performs forward-modelling and generation of a broadband PSF using exit pupil wavefronts. The inputs into SHE_PSFToolkit were derived from the Zemax¹⁰ optical model of the telescope (neglecting, however, polishing errors, the effect of the dichroic and the high-frequency components on the surface). Such large and detailed PSF datasets were built to meet the accuracy requirements necessary to measure weak lensing at the target precision.

The VIS PSF model cannot be interpolated spatially due to its complex shape variation (at small and large scales). Therefore, when we draw a particular star or galaxy, we take the closest PSF to its focal plane position. Each PSF is generated from the Zemax model at eight wavelength values within the VIS passband, and the sampling is incresed by interpolating up to 1 nm. With the reconstructed spectra from the source, we convolve the light profile with the PSF at each interpolated wavelength position, resulting in a chromatic PSF that varies as a function of the colour for each object. Secondly, we have the AOCS described earlier in Sec. 2. This complex system performs tiny corrections to keep the spacecraft attitude stable during an exposure, which slightly increases the width of the final PSF. The jitter of the AOCS can be replicated as a time series that mimics how the correction behaves. However, we model its final effect on the VIS PSF by convolving with a Gaussian of $0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}025$ RMS. The detector degrades the PSF by diffusing the charges in each pixel into its adjacent pixels. This effect can be described in the Pixel Response Function, which gives the distribution of fluxes around each pixel. Due to variations in the CCD thickness, charge diffusion is not constant across the field of view.

Optics and system. The combined optical system of the PLM and the VIS instrument is accurately reproduced in ELViS. First, we have the response of the optical system which comprises the reflection of the mirrors and the transmission of the dichroic element. The PLM not only affects the transmission but its geometry distorts the incoming light beam, modifying the plate scale as a function of focal plane position. This results in objects being displaced from their nominal position and a nonuniform illumination of the VIS array. Geometric distortions also affect the shape of the galaxies which then require an adjustment during the drawing process with SimThumbnails.

Furthermore, we emulate the detector array mount on the focal plane, with accurate gaps, misalignments and rotations as measured in ground tests. This adds further realism and complexity in the astrometric solutions derived from the calibration process. Additionally, we include a calibration unit (VI-CU) with its highly non-flat illumination to replicate flat field calibration images (VIS has up to 6 different LEDs available). Other subtle effects are also included such as the effect of variable illumination and readout trails. The former is caused by the opening and closing of the shutter and the latter effect is caused by the open shutter during the readout process: light coming from astronomical objects create photo-electrons in the opposite direction of the readout.

Detectors. The 36 Teledyne e2v VIS back-illuminated, red-enhanced CCD detectors will provide a great dynamic range with high precision spatial resolution at 12 μm square. While these are state-of-the-art detectors, there are numerous effects that degrade the image and need to be accurately simulated. The response of the pixels is not uniform due to variations in the gain and in the construction process, intrinsic to the material. This is known as Photon Response Non Uniformity or PRNU. We can obtain realistic PRNU maps from the ground calibration campaigns. Additional cosmetic defects from the process of construction and dust are also included. Furthermore, each pixel does not respond uniformly across its square area, that is the centre of the pixel is more responsive than the corners. This is an effect we call intra-pixel quantum efficiency. When the number of electrons exceed the full well capacity (the limit of electrons that a pixel can collect), charges overflow in the readout direction into the neighbouring pixels, causing the effect of bleeding. The effect of Dark Current is also included with a dependence on the VIS focal plane temperature. The final detector effect included is CTI, which creates streaks in the readout direction by electrons that remain in the previous pixel. This effect increases as the radiation from space damages the detectors with time creating charge traps in the silicon chips; it becomes significantly worse towards the end of the mission (and the most relevant degradation that limits the lifetime of the VIS instrument). Our implementation of CTI is based on the model described in Massey et al. (2014).

Electronics. As is typical with any optical CCD detector, a bias voltage is applied before the Analog-to-Digital Converter (ADC) to increase the signal from the 0 value. This requires models for the electronic bias, the ADC gain conversion, and proper simulation of the prescan and overscan sections. An additional complication is that the amplifier does not respond equally at all flux intensities; an effect that is about 1% in the lower regime and 5% when getting closer to the ADC saturation. The ADC has a limited number of bits to sample the charges in each pixel, causing the discretisation of the signal and setting the highest value at which the converter saturates. In the case of the electronics for Euclid with 16 bits per pixel, it is set to 2¹⁶−1 or 65535 ADU. In the process of reading the sensor, the electronics add scatter in the measured values, which vary from detector to detector, typically around 5 e⁻ RMS. As all 36 detectors are being read at the same time, the induction of the current from the wires that transport the signal can produce reflected images on neighbouring amplifiers or detectors (known as electronic crosstalk).

In Fig. 13, we provide an example exposure produced by the VIS simulator. The mosaic of the VIS detectors is seen on the left. On the right, we show a small region of the exposure which exhibits distinctly the various simulated effects described above such as cosmic rays, CTI, ghosts, and saturation.

5.2 NISP simulators

We use two separate simulation softwares, Imagem and TIPS, for the photometric and spectroscopic channels of the NISP instrument, respectively. They share much of the optical path as well as the detector and data-processing units; they also have significant overlap in the models implemented, which we outline in the following before describing the channel specific implementations in Sects. 5.2.2 and 5.2.3.

5.2.1 Common models

Both the NISP-P and NISP-S simulators implement the same type of astronomical sources and share some common background, PSF, optics, detector, and electronics models, which we explain below. Consistency across the two NISP channels is particularly important as the spectroscopic analysis relies on measurements from the imaging channel.

Sources. In addition to the sources already included in the VIS simulator discussed in Sec. 5.1, we simulate additional sources required in the calibration of NISP images. In particular, we simulate two spectrophotometric standard stars which are used to perform accurate photometric calibrations. For the spectroscopic channel only, we produce simulations of a planetary nebula, which enable wavelength calibration due to its strong and well characterised emission lines. Additionally, we simulate standard stars which provide a photometric calibration reference. The planetary nebulae and the standard stars are simulated in separate specific observations, in addition to the TU sources, at five locations in each detector, providing calibration points through the whole NISP focal plane. Both NISP simulators use a CREME96 model (Tylka et al. 1997) using a representative inorbit proton energy distribution to simulate cosmic ray tracks as these particles, arriving from a random direction, traverse a 7 μm thick HgCdTe layer in the Teledyne H2RG detectors.

Background. The main common components of the background are zodiacal light, thermal irradiance, and out-of-field diffuse straylight. Even though these have already been explained in the VIS section (see Sec. 5.1), it is worth noting that due to its redder spectrum, zodiacal light is a significantly more intense source of background for NISP than for VIS.

Point spread function. The NISP PSF contains the same types of effects as explained for the VIS instrument, although it can be simplified thanks to the less stringent requirements on the instrumental PSF. For SC8, a single model at the centre of each detector was used to represent spatial variations, taking into account the telescope and NISP instrument optical models.

Optics or focal plane array. Both simulators implement with their corresponding filter/grism transmission and geometric distortion models the PLM transmission (see Fig. 1) and the focal plane array (FPA) metrology that depends on the exact position and tilt of each detector in the focal plane.

Calibration. Dark and flat simulations are also provided for both NISP channels for the usual data reduction process. Similar to the calibration unit of VIS, the NISP Calibration Unit (NI-CU) can also illuminate its focal plane with a set of light emitting diodes (LED) at five different SEDs in the infrared. They are additionally set at different intensities to produce flat fields, which are also simulated.

Detectors. The 4 × 4 set of Teledyne H2RG infrared detectors are simulated with measured quantum efficiency, dark current, and readout noise maps provided from ground tests (Waczynski et al. 2016; Secroun et al. 2016, 2018; Barbier et al. 2018) and the Poisson distributed shot noise from the incident sources and background is accounted for. While a 4-pixel boundary corresponding to the reference pixels is included in the NISP simulation, it is currently set to zero ignoring baseline shifts. For SC8, we did not simulate the effect of NISP persistence; however, we advise the reader to consult Sec. 7 for a description of our implementation of persistence in our post-SC8 simulations.

Electronics. The readout of the H2RG detectors follows the procedure described in Kubik et al. (2016) called multiple accumulated sampling (MACC), where the incoming signal is sampled multiple times during the exposure in an up-the-ramp readout process. In this configuration, we can estimate the ramp of the signal in each pixel, as well as the quality of the fit to the ramp. This allows us to identify pixels which do not follow a stable rate (like cosmic rays that will produce a sudden jump in the accumulated flux). While NISP photometry and spectroscopy use different MACC readout modes, owing to their different exposure times, each uses an evenly spaced number of groups that are composed of a fixed number of frames read and are separated from the next group by a fixed number of frames that are dropped (see Fig. 14).

For a time t_f separating one readout frame from the next and a MACC readout with n_g groups, n_r reads in each group, and n_d drops between groups, the integration time is $$t_{\text{int }}=(n_{\mathrm{g}}-1)(n_{\mathrm{r}}+n_{\mathrm{d}})t_{\mathrm{f}}.$$(5)

In the following we describe the steps to compute a charge estimate Q that, when divided by the integration time t_int, directly results in a flux estimate, including the signal but also backgrounds and dark currents. In any given exposure, let F_pix(x, y, i, k) be the single frame charge in ADU capturing the accumulated signal and the instrumental effects at frame k in group i for the pixel (x, y), σ_r the single frame readout noise in ADU and g the gain in ADU/e⁻.

First, we combine the single frame charges in each group i into group averages $$G_i=\frac{1}{n_{\mathrm{r}}}{\displaystyle \sum _{k=1}^{n_{\mathrm{r}}}}{F}_{\mathrm{p}\mathrm{i}\mathrm{x}}(x,y,i,k).$$(6)

With these, the overall charge estimate Q in ADU is set as $$\begin{array}{rl}Q=& (n_{\mathrm{g}}-1)\\ & \times [\frac{g(1+\alpha )}{2}(\sqrt{1+\frac{4\sum _{i=2}^{n_{\mathrm{g}}}{(G_i-G_{i-1}+\beta )}^2}{g^2(n_{\mathrm{g}}-1)(1+\alpha {)}^2}}-1)-\beta ]\end{array}$$(7)

where $$\alpha =\frac{1-n_{\mathrm{r}}^2}{3n_{\mathrm{r}}(n_{\mathrm{r}}+n_{\mathrm{d}})}\text{ and }$$(8) $$\beta =\frac{2{\sigma }_{\mathrm{r}}^2}{g{n}_{\mathrm{r}}(\alpha +1)}.$$(9)

The charge estimate’s quality factor QF in ADU for the same readout ramp is computed as $$\mathrm{Q}\mathrm{F}=\frac{2}{(1+\alpha )(n_{\mathrm{g}}-2)}[(n_{\mathrm{g}}-1)K-(G_{n_{\mathrm{g}}}-G_1)]$$(10)

with $$K=\sqrt{\frac{\sum _{i=2}^{n_{\mathrm{g}}}{(G_i-G_{i-1}+\beta )}^2}{n_{\mathrm{g}}-1}}-\beta .$$(11)

This quality factor QF has approximately a χ² distribution with (n_g − 2) degrees of freedom; that is, 2 and 13 degrees of freedom for NISP-P and NISP-S, respectively. The QF is later used to mask pixels that do not follow a linear accumulation of counts, such as those affected by cosmic rays.

5.2.2 NISP-P simulator

The NISP-P channel is simulated with Imagem, which reproduces NIR imaging in the three broadbands of the instrument: Y_E, J_E, and H_E. Being the primary source of information for the photometric redshifts in the infrared bands, it is important to accurately simulate the flux of the sources and any possible instrumental effects that could bias the photometric intensity and colour measurements. Below, we highlight the main features included in the NISP-P simulator.

Sources. In addition to the common sources, the NISP-P simulator adds the specific standard stars with a particularly smooth spectrum, ideal for photometric calibrations where synthetic photometry can be estimated accurately. We used the spectrum of Vega and a white dwarf, scaled to AB magnitude 17.5 in the Y_E band, which delivers a good signal-to-noise below the range of the saturation limit or the non-linear regime of the detectors, respectively.

Background. In addition to the common backgrounds detailed above, we include localised in-field straylight in the form of faint circular symmetric halos with detailed shapes dependent on the source SEDs.

Point spread function. We use a chromatic PSF with three wavelength-dependent PSFs for each of the three photometric filters that are integrated with the incident spectra of each TU source and a wavelength-dependent quantum efficiency. NISP-P PSF pixels are sampled at one-sixth of the size of detector pixels in both dimensions. This is to allow us to emulate differences in the projected PSF depending on whether a source is centred; for example, on a pixel centre, edge or corner. First, we convolve sources with the optical PSF on these sub-pixel scales. After the resulting high resolution stamps are projected onto detector pixels, we apply an additional convolution with a 3 × 3 matrix on the detector pixels to simulate the effect of inter-pixel capacitance (IPC).

Optics. Along with the common optical effects like filter and PLM transmission, FPA metrology, and geometric distortions, we also implement the effect of filter wheel tilt variations; in other words, the inaccuracy of the filter wheel to be positioned always at the same exact position, resulting in small angular deviations with a RMS error of 6′. These additional tilts impact the final distortions which are partly affected by the optical power of the filter element.

Detector. For each pixel in a source, electron fluxes are integrated using measured quantum efficiencies for 40 wavelengths from 0.6 to 2.55 μm for every detector pixel, the source SED and the PSF chromaticity. We simulate each source three times for every exposure to account for the three wavelength-dependent PSFs per filter. Using distinct wavelength dependencies of the quantum efficiency for each pixel can therefore lead to varying pixel sensitivities depending on the source spectrum as well as varying effective PSF shapes according to the source position on the detector. The photometric flat fields, in addition to the calibration unit LEDs spectra, also use the wavelength-dependent quantum efficiency maps.

Electronics. Imagem simulates the readout process as detailed above in Sect. 5.2.1, accounting for each intermediate frame in the specific MACC readout mode used of 4 groups, 16 reads and 4 drops for the three photometric bands. This provides an accurate reproduction of the image and the associated onboard flagging based on the charge estimate’s quality factor QF, where flagged pixels are transmitted as an additional 1-bit image layer. The resulting image is further modified by pixel dependent non-linearity and a single gain value.

Figure 15 shows an example raw exposure in the H_E band produced by Imagem. The mosaic image for the 4 × 4 detector array for the NISP instrument is displayed on the left. The right image shows in finer detail a selected area of one of the detectors.

5.2.3 NISP-S simulator

The NISP spectroscopic simulator is implemented in TIPS (Zoubian et al. 2014), a program built to simulate the slitless spectroscopy images from the NISP instrument. It is based on the slitless simulation package aXeSIM¹¹ (Kümmel et al. 2009) developed by the Space Telescope Science Institute, which performs the core spectra dispersion and convolution with a chromatic PSF of ten wavelength positions, sampled at one-sixth of the native NISP resolution. As stated above, most sources and instrumental effects are shared with the NISP-P channel simulator. The performance of the spectroscopic channel is expected to be dominated by the effects that are currently implemented which we shall describe below.

Spectral distortions. The simulation of the slitless spectra requires a complete characterisation of the trace dispersion, with sensitivity, diffraction coefficients, and PSFs at each dispersion order (expressed as a Taylor series expansion). The dispersion order of interest is the first order, where the main spectrum can be recovered. However, an additional zeroth and second order of dispersion need to be simulated as well, as they provide reference points (for the zeroth order) and contamination in the image. While the average dispersion is 700, 13.4, and 6.7 Å pixel⁻¹ for the 0th, 1st, and 2nd orders of dispersion, respectively, the wavelength steps Δλ are not constant across the field of view. Thus in the simulation of spectral distortions we included a spatially varying Δλ. This significantly complicates the extraction of the spectra, which needs to be calibrated precisely to enable the determination of accurate spectroscopic redshifts.

Point spread function. The plan initially was to use ZEMAX-issued models for each dispersion order of each grism, in a similar fashion to the NISP-P simulations. However, this was later discarded because of the challenges it could have presented at that the time for the SGS pipeline, notably at localising with fine precision the zeroth order spectra, an important requirement for the data-processing of spectroscopic data. This is expected to be calibrated and validated in later processing of the NISP-S pipeline. Instead, we used a double Gaussian PSF to fit the ground-based characterisation data acquired from thermal-vacuum balance tests in order to match the experimental radius encircled energies EE50/EE80 measurements (Maciaszek et al. 2022). This provided smoother non-chromatic pixel PSFs. No IPC was not taken into account in NISP-S simulations.

Grism tilt. Like the filter wheel tilt effect described for the photometric channel, the grism wheel also has a limited positioning accuracy with an error of $0_{\cdot }^{\circ }1$ RMS. In this case, the positioning error translates into slightly tilted dispersed spectra with respect to their dispersion direction at nominal angle. To enable an efficient decontamination of overlapping slitless spectra without the RGS270 grism, the instrument implemented a tilt of +4° and −4° in dithers 2 and 3 (as shown in Fig. 11) in a sequence of 0°, 184°, −4°, and 180° orientation of the dispersion direction.

Vignetting. Turning the grating wheel by ±4° moves the nominal aperture of the grating creating vignetting on the focal plane. This effect is simulated using a basic and conservative model derived from ground calibrations assuming 10% of vignetting at +0°385 field angle, which diminishes to 0% at a field angle of +0° 17. We apply vignetting only to the sources to take into account the flux loss; we neglect the effect to the background.

Electronics. As indicated above in Sect. 5.2.1, TIPS simulates the readout process for each intermediate frame of the MACC mode in spectroscopy with 15 groups of 16 frames separated by 11 drops. The signal slope and the quality factor of each pixel is then derived following Kubik et al. (2016), as it is implemented in the onboard payload data-processing. Non-linearity is applied afterwards on the slope by using the inverse function of the non-linearity correction model, but its impact on the quality factor is ignored. The slope, in ADU, is encoded on 16 bits and the quality factor on 8 bit.

We show in Fig. 16 an example image produced by TIPS for the NISP instrument when emulating readout for the spectroscopic channel. Like the photometric channel, we have the same 4 × 4 detector mosaic on the left. For closer inspection, a small region of a detector is shown on the right containing the characteristic features of slitless spectroscopy.

5.3 EXT ground-based survey simulator

Lastly, the ground-based simulator SIM-EXT is a generic simulation software package to produce accurate pixel data for various ground telescope and instrument systems. Euclid will rely on external optical photometry complementing its own infrared bands in order to obtain accurate photometric redshifts. The necessary photometry in the northern hemisphere will be provided by the Ultraviolet Near Infrared Optical Northern Survey¹² (UNIONS), a joint effort between the Canada-France Imaging Survey (CFIS; Ibata et al. 2017) for the u and r bands, the Javalambre-Euclid Deep Imaging Survey (JEDIS) in the g band, the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS; Chambers et al. 2016) in the i band, and the Wide Imaging with Subaru Hyper Suprime Camera (HSC; Miyazaki et al. 2017) of the Euclid Sky survey (WISHES) for the z band. For the southern sky, it is anticipated that the Dark Energy Survey (DES; The Dark Energy Survey Collaboration 2005) and the Rubin Observatory Legacy Survey of Space and Time¹³ will provide all ugriz photometry (LSST Science Collaboration 2009; Ivezić et al. 2019).

The function of SIM-EXT differs from that of the other Euclid instrument simulators. Rather than producing extremely detailed simulations for each EXT survey (a notably costly effort), its purpose is to provide realistic imaging data that can flow inside the ground segment pipeline with the same TU inputs as the other Euclid channels. The output data products of SIM-EXT vary per EXT survey in accordance with the SGS strategies designated for each one. For example, data-processing of raw DES data will be performed within the Euclid SGS whereas the detrended and/or calibrated data products produced by the other respective experiments and synergies (e.g. UNIONS and LSST) will be received by the Euclid SGS when available. Therefore, for DES we produce outputs consisting of raw images and master calibration frames. For LSST, only raw images are produced with a choice of units: e⁻ or ADU. Moreover, the FITS format for LSST images produced by SIM-EXT version 2.14.2 is similar to the e-image format defined by the Dark Energy Science Collaboration (DESC) image simulator imSim version 0.1.0¹⁴ described in Sánchez et al. (2020). This output format is required for SIM-EXT LSST frames to be processed using the Rubin Observatory LSST Science Pipelines release 16.0¹⁵ (LSST Dark Energy Science Collaboration 2012; Jurić et al. 2017; Bosch et al. 2018; Ivezić et al. 2019). This allows the SGS to emulate ingesting official Rubin Observatory calibrated data products for joint pixel processing with the Euclid data products. For the Northern Surveys, a software package called EXT_NS_SIM_AFTERBURNER developed by the Organisation Unit for External Surveys (OU-EXT) is used in conjunction with SIM-EXT to provide level-2 processed data products consisting of detrended frames, source catalogues, and PSF models.

Sources. The EXT ground-survey simulator performs the generation of all TU sources such as real and model stars, Flagship, primaeval Universe galaxies, high-z, and strongly lensed sources. Using the same SimThumbnails as the Euclid simulators, it renders the exact same light profiles including weak lensing distortions. As the PSF is assumed to be monochromatic, it does not require the input spectra and uses precomputed band fluxes, thus accelerating the computation of these simulations.

Background and transmission. As most of the EXT surveys have already started their observations, we have realistic background magnitudes and zeropoints from their respective reduced image analysis. We use the measurements taken from actual observations contained with the MDB Survey Tables as inputs to the simulation, which provide a very realistic distribution of sky brightness and atmospheric extinction. For the EXT surveys, such as Rubin, that have not started their observations, we can model realistic distributions from the available models of the sky brightness (Yoachim et al. 2016) and passbands (Kahn et al. 2010; Claver et al. 2014; Connolly et al. 2014).

Point spread function. The PSF is significantly simpler in most EXT simulations, and it is modelled as either a simple Gaussian or a Moffat profile. From a realistic distribution of seeing FWHM, we can replicate the average optical quality for each survey. In the case of DES, we obtained a large sample of PSF models extracted from real data analysis using PSFEx (Bertin 2011). We randomly select a model, enabling variable and complex PSF simulations across the focal plane for the ground-based photometry.

Optics and detectors. The design of SIM-EXT takes advantage of the common characteristics of the ground-based instruments to perform simulations of various surveys. These surveys are configurable through the MDB instrument file providing a description of the optics, detector layout and electronics. Figure 17 shows the focal plane layouts for the DES Dark Energy Camera (DECam), Rubin’s LSST Camera (LSSTCam), CFIS Mega Camera (MegaCam), JEDIS (JPCam), Pan-STARRS PS1, and WISHES HSC, respectively. In detail, the camera detector layout for DECam was derived from Flaugher et al. (2015). The detector focal plane layout for the LSST camera (LSST-Cam) was defined with the information provided in Ivezić et al. (2019). For simplicity, our camera model is an approximation to LSSTCam, as all 189 detectors in our focal plane are models of either LSSTCam’s e2V sensor or LSSTCam’s ITL sensor, not a combination of both as will be used during operations. The detector layout for MegaCam, JPCam, Pan-STARRS PS1, and HSC were each inferred, respectively, from a set of real exposures provided courtesy of the CFIS, JEDIS, Pan-STARRS, and WISHES teams. For a majority of the surveys, we deliver a perfectly calibrated WCS by writing the true WCS in the header of the output images. For DES, we reproduced a certain degree of ‘un-calibration’ by writing a slightly different WCS in the output headers. The input WCS are obtained with the astrometry tool SCAMP (Bertin 2006) applied on actual data. If real data are not available, we apply a simpler estimation of the WCS (without distortions). Additionally, we use master bias and master flats from the actual observations to include realistic pixel-to-pixel variations and faithful readout patterns. With the same model, we can create individual flat field and bias calibration images that have their own noise realisation.

A sample image of the SIM-EXT simulator configured to produce a DES exposure can be seen in Fig. 18. On the left, we show the 62 detector mosaic and on the right a close-up image of one of the CCDs. In addition to the features discussed above, we have the additional detector effect of tree rings acquired from sampling from real DES flat data; these are circular patterns due to variations of the silicon dopant concentration in the CCD chip.

With all four instrument simulators presented in this section, we can produce complex and detailed simulations for a large variety of channels and configurations for the same Universe, environment, and common models. In the following section, we present our procedure for launching a large-scale production over segments of the Euclid Wide Survey using our suite of simulation codes.

6 SC8 simulation processing and results

In this section, we present the application of our SIM pipeline framework for SC8 and the corresponding results. We first begin with an in-depth description of our simulation production workflow as developed for SC8.

6.1 Simulation planning and orchestration

The previously defined elements (i.e. TU catalogues, simulators, etc.) enable the generation of mock observation sets of data to be processed and analysed. However, for the production of large datasets in a distributed and controlled manner these simulation elements need to be connected and coordinated. Here we describe the additional components of our software framework which fulfill this objective: the simulation request, the SimPlanner, and the SIM pipeline.

The simulation request is one of the inputs of the SIM pipeline together with a mission configuration and the catalogues. One can specify some aspects of the simulations such as (i) channel: VIS, NISP-P, NISP-S, EXT-LSST, EXT-DES, (ii) image type: Science, Deep, Flat, Bias. (iii) pointings or area to simulate, and (iv) instrumental effects to be enabled/disabled.

The simulation request is handled by the SimPlanner, a program that constitutes the entry point of the SIM pipeline and allows us to orchestrate large sets of simulations. It parses the information in the simulation request and the survey files and configures the corresponding simulator with the appropriate parameters. The SimPlanner contains additional features such as the generation of particular dithering patterns, in-pointing sequence, enabling complex parameter exploration, configuration of flat field sequences, and other high-level tasks that can be done at this point.

The SIM Pipeline framework allows multilevel splitting and parallelisation of the processing, which is critical to producing large sets of simulations in a reasonable time frame. The processing of the SC8 main production, as well as the other productions in the scope of SC8 (e.g. deep field) was carried out on the Euclid SGS infrastructure.

6.2 Pilot phase

SC8 began with a small test production of simulations in preparation for the main run, called the pilot phase. This production had two primary goals: (1) ensure the validity of the simulations and subsequent data products produced during image processing by each OU in the SGS pipeline; (2) measure the amount of resources, such as number of cores, peak RAM, and maximum wall time, needed for each of the OU-SIM tasks, (3) perform the necessary optimisations to guarantee feasibility of the larger-scale SC8 main run with the available resources.

The pilot phase contained four Science Wide observations covering the so-called Euclid tile (Kümmel et al. 2022) (79171) with an area measuring ~0.5 deg². Euclid tiles are non-overlapping ecliptic latitude-longitude rectangles (approximately the size of the Euclid field of view) and oriented along the meridians. With no gaps between adjacent tiles, they are essentially the unit of processing for high level OUs to ensure complete coverage of the survey area when constructing the object catalogues for WL and GC analysis. Even for an area this small, the multiple feedback iterations with the other OUs resulted in a total production of 1866 raw images for all simulation channels. In addition, all the calibration frames required to carry out image processing (such as detrending, astrometric, and photometric calibrations) for the Euclid instruments were also produced during this phase and later reused for the data reduction during the main run. The set of VIS calibration products consisted of a sequence of 60 bias images and 60 short-exposure flat-fields using the VIS calibration unit (VIS-CU). Since the shutter will remain open during flat acquisition, there will be objects entering into the field of view of the telescope, and we have simulated the most luminous objects yielding a flux above the photon noise.

For the NISP instrument, we produced a sequence of 600 dark frames and 750 flat-field images with the respective LED intensities and exposure times for the photometric and spectroscopic channels. We also simulated observations of spectrophotometric standard stars and planetary nebulae to enable the performance of NISP photometric and spectroscopic wavelength calibrations. Finally, we also delivered a self-calibration sequence over a dense stellar field consisting of multiple dithered overlapping images that were to be used to improve the flat-fielding and the astrometric calibration.

In Table C.2, we list the number of files and the total size for each image data product produced during the pilot phase. We show in Fig. 19 raw science images produced by the simulators for the VIS, NISP-P (Y_E band), NIS (RGS0), and three EXT ground surveys in the z band: WISHES (HSC), DES, and LSST over the same area of sky in Euclid tile 79171 for one of the four Euclid observations (Observation ID 18539). The simulated Euclid images show considerably more prominent features (e.g. ghosts, diffraction spikes, and CTI) in comparison to the simplified images produced by the simulator for the EXT ground-based surveys which are known or expected to experience similar effects.

6.3 Main run

The original plan of the SC8 main run was to conduct a full end-to-end test of the SGS pipeline using a large and complex simulation of 1286 observations from the Euclid Wide Survey, covering a contiguous area of ~ 600 deg² with different sky conditions. All known instrumental effects were to be included in the pixel data to enable performance and quality assessments of the level-2 calibration pipelines. Furthermore, the large area would enable the first integration test of the final processing elements of the Euclid SGS pipeline (i.e. the level-3 cosmological measurements) with enough area to measure the baryon acoustic peak in the galaxy two-point correlation function.

However, as explained in the previous section, the results of the pilot phase indicated that this would present a major challenge as producing and processing large catalogues and realistic images carries a high computational cost. The SC8 main run could not have been achieved in a reasonable time frame with the resources available for the challenge. Therefore we have reverted to simulate a subset of 331 observations grouped in 11 noncontiguous patches preserving the variety of conditions: mostly stellar density and zodiacal light as shown in Fig. C.1. The final simulated area measuring ~165 deg² was still sufficient to perform the integration test of the processing elements of both the WL and GC experiments, the topmost priority.

Figure 20 shows the 11 patches and the position of each Euclid observation contained within each of them. The colour of each patch represents the SDC tasked with processing the set of SC8 observations. The assignment of a region to a SDC was according to their expected contribution and the resources available. Moreover, the strategy of having non-contiguous patches processed at the different SDCs was implemented to avoid data transfers between processing centres. Each patch is composed of a set of Euclid tiles. There are 1039 Euclid tiles overlapping with the SC8 observations.

Over the selected Euclid targets in the sky, we simulated all three Euclid channels: VIS, NISP-P, and NISP-S, respectively. The main run also included simulations of the EXT ground-based surveys in the northern and southern hemispheres. The target area was partitioned into three non-overlapping patches. All Northern Surveys – CFIS, JEDIS, Pan-STARRS PS1, and WISHES – covered the same region above 30° in declination. We partitioned the lower rectangular region along the diagonal into two sections: Rubin observations spanned the area above the diagonal and DES covered the opposing field below it. We refer the reader to Appendix C.4, which describes with more detail the selections of the pointings for each EXT survey and provides figures illustrating the EXT survey footprints in each band.

In Table 1, we report the number of data products and the amount of storage required per survey channel participating in the SC8 main run. We produced a total of 342 368 data products amounting to 31.7 TB for the 165 deg² Euclid Wide Survey.

Notably, even with shallower data than the actual Rubin experiment, the unprecedented enormity of the three billion pixel Rubin camera with its $3_{\cdot }^{\circ }5$ (9.6 deg²) field of view, combined with imaging capability in five optical bands,¹⁶ yielded the largest dataset. These results provide insight on the volume of actual EXT ground-based data that has to be processed for the Euclid mission. EXT data may require as much storage and computing time (or more) as the Euclid data themselves, especially if performing pixel-level processing of purely raw data products. Finally, the total wall time to produce the entire main SC8 dataset was 16 730 days. The VIS simulations, which remained full-featured, dominated the processing time with 5200 days followed by NISP-S at 4700 days.

6.4 Deep run

For the SC8 Euclid deep survey, we simulated a small area measuring ~ 1 deg² of the Euclid deep field around the north ecliptic pole. Figure 21 shows the SC8 deep field footprint and the number of layers to reach the desired depth. We produced sets of full-featured simulations for VIS, NISP-P, and NISP-S with ten times more exposures totalling 840 amounting to ~1.4 TB (as reported in Table 2) and reaching a flux limit 2 mag fainter than the nominal wide field. For NISP-S, we also ran simulations of the blue grism (BGS00) in addition to the three other spectroscopic channels. We did not produce simulations of any northern EXT ground-based surveys for which no current strategies have been defined at these fainter magnitude depths. Nevertheless, the SC8 deep field run was designed to be used for testing by the level-2 SGS processing chains, which perform image and spectra calibration, photometry, and coaddition for VIS and NISP, as well as PSF modelling and the generation of optical distortion maps.

6.5 Outcomes of SC8

Here, we provide a brief summary of the SGS end-to-end performance in SC8 as a more in-depth discussion goes beyond the scope of this work. Our total production of simulations for the SC8 main and deep surveys yielded 33 TB of data. The entire SC8 dataset was transferred to the EAS and queried by the level-2 OUs to initiate the data reduction pipelines, signaling the start of the end-to-end integration test.

Data-processing by the full SGS pipeline proceeded gradually with data quality inspection by the respective OUs and updates to various software packages. Nearly all level-3 processing elements¹⁷ were successfully integrated. The full integration test also served as an aid to identify and resolve performance issues concerning infrastructure, bugs, and bottlenecks in software. It also provided an impetus to further improve communications between interfacing OUs in order to maximise SGS capabilities as we approach operational readiness.

We conclude this section by highlighting the work enabled by the SC8 simulations such as the testing of detection algorithms for strong lenses, galaxies high-z quasars and galaxy emissionlines with Euclid. As detailed in Appendix C, along with the standard galaxy and standard stars, the object catalogues also contained a variety of additional sources such as QSOs, high-z galaxies, strong lenses, and LT stars (brown dwarfs). From the sample of 801 strong lenses, 285 sources were inserted into 216 Euclid tiles covering the SC8 main field. The strong lensing sample The catalogues containing these sources were provided as inputs to the various simulator codes for the VIS, NISP-P, and EXT instruments (i.e. ELViS, Imagem, TiPS, and SIM-EXT), which introduced instrumental and environmental effects in each band to produce raw images that were then processed by the SGS pipelines. In detail, VIS and the NISP-P calibrated frames were produced by the respective OU level-2 pipelines for VIS and NIR image reduction.

The VIS and NIR data products along with their corresponding flag maps (e.g. cosmics) then proceeded through the SGS data-processing chain as inputs to the OU-MER level-2 pipeline, where they were stacked. The MER pipeline then performed the background removal and used the flags to produce fully detrended and background subtracted MER mosaics.

Figure 22 presents an example of raw (level-1) and processed (level-2) data products on a zoom-in of Euclid tile 306182 in the SC8 deep field. On the left, we show the raw simulated science exposures for VIS (top-left) and NISP-P (bottom-left; for Y_E band only) with the characteristic features of these two instruments (described in Sects. 5.1 and 5.2.2, respectively). The two rightmost panels show the corresponding calibrated, stacked, and background subtracted mosaic images produced by OU-MER after processing the data products generated by OU-VIS and OU-NIR, respectively.

These MER mosaic images and the source catalogues generated from them were used, respectively, by the Strong Lensing and Primaeval Universe Science Working Groups to perform tests of their corresponding detection algorithms in development.

Figure 23 shows a zoom-in cut-out of the strong lens 200026 in the VIS and NIR deep simulations. The Einstein ring can be seen with fine detail by the VIS instrument, whereas it is indiscernible in the Euclid NIR bands. Another of the Strong Lensing Science Working Group detections was a strong lens system (200094) in the SC8 main run (Euclid tile 74088) in both MER processed Euclid and Rubin mosaic data. Cutout images of the system are shown in Fig. 24; from left-to-right, we show a set of images of the Einstein ring as seen in the VIS I_E band, combined Rubin g+r+i bands, and combined VIS and NISP-P (I_E+Y_E+H_E) bands. These images demonstrate the resolving power of both Euclid and LSST, two forthcoming contemporary and pivotal science programs. They also, however, exemplify the exceptional image quality anticipated with the VIS camera alone with its superb capability to deblend compact systems.

In Fig. 25, we show an example of a successfully calibrated, decontaminated, stacked, and modelled galaxy emission-line spectra produced from the raw pixel data of the NISP-S grism simulations in SC8 processed by the Organisation Unit for Near Infrared Slitless Spectroscopy (OU-SIR) and Organisation for Spectral Measurement (OU-SPE), respectively. The results in the figure show a clearly identifying the redshifted Hα emission line by the OU-SPE spectroscopic emission line detection algorithm. The identified emission-lines were then used by OU-SPE to estimate spectroscopic redshifts.

7 Future perspective

Following SC8, new features and improvements have been added to our simulation framework. For example, to produce more realistic profiles, we implement deep generative models, a machine learning framework presented in Euclid Collaboration: (2022a) that can generate complex galaxy stamps learned from a training set of real Hubble Space Telescope observations. These galaxy thumbnails include spiral arms and other features that cannot be represented with analytical profiles as the ones we currently use. The advantage of this approach is that it can draw complex stamps just from parameterised information such as the one available in the TU galaxy catalogue without complex morphological models in the Flagship galaxy mock. These realistic stamps will enable the development of other complex algorithms to measure shear and to deblend larger objects. The colour gradient and its impact on shear measurments have already been studied (Welikala & Kneib 2012) and an implementation in the SIM pipeline is currently in progress.

For the NISP simulators, we introduced an additional background component caused by persistence: trapped charges from one exposure are released on later image acquisitions. In early ground-based calibrations, a persistence signal was detected more than two hours after the initial exposure for bright sources. Among other dependencies, the charge reached during an exposure, exposure time and history of successive illuminations all affect the release of charges trapped in a pixel. A preliminary model only takes into account the single preceding exposure. For the persistence signal I_persistence that is added to the current signal, we use the following model: $$I_{\text{persistence }}=(p_0+{10}^{-5}{p}_1{I}_{-1})(1-{\mathrm{e}}^{-L_{-1}/p_2}),$$(12)

where I₋₁ is the previous image and p_i are a set of measured persistence parameters. Distinct p_i values are used for each pixel, previous exposure time and the elapsed time between exposures in the survey. Additionally, we have released within the Euclid Consortium newer versions of SIM-EXT (2.17+), which feature spatially varying PSFs for CFIS and WISHES. These are produced using PSFEx models for CFIS u and r bands and WISHES z-band data provided to us courtesy of the CFIS and WISHES teams. Our Rubin simulations now include nominal rotation of the camera per pointing acquired from the OpSim v2.0¹⁸ survey simulations (Jones et al. 2020). Furthermore, these simulations currently implement the FITS format (described in Lim 2019) as well as the camera layout and WCS description of DESC imSim version 0.6.2¹⁹ (Abolfathi et al. 2021). The file formatting and WCS changes fulfill the requisites of the latest generation LSST Science Pipelines software to perform processing of LSST simulations. Finally, SIM-EXT 2.17+ now provides simulated data products in the g band for the Waterloo-Hawaii-IfA G-band Survey (WHIGS) using the Subaru telescope.

The complete set of calibration exposures for the Euclid instrument will encompass a large variety of image types. In addition to the standard calibration exposures: bias, dark, flats, we also produce calibration unit exposures, out-of-focus exposures for phase diversity, different readout modes for NISP, injection line frames, trap pumping for VIS CTI calibration, NISP persistence model validation, bleeding and blooming calibration exposures. For the PLM/instrument calibration exposures, we produce straylight checks and dense stellar fields. Finally, for precise astrometric calibration, we generate white dwarf and planetary nebulae observations.

Upon completion of PVPR, the Euclid Ground Segment will commence its third SPV (SPV3) phase. The focus of this campaign will be on assessing the scientific performance of the Euclid mission. In detail, we plan to produce nominal science exposures for the wide and deep areas spanning a few tens of degrees to yield mock WL and GC measurements (e.g. C_ℓ and power spectrum multipoles) to enable testing of the Euclid Consortium Science Working Groups analysis pipelines.

8 Summary and outlook

OU-SIM is responsible for providing detailed simulations to facilitate the development of the SGS data-processing pipeline as well as end-to-end performance tests. In this work, we have introduced our simulation framework for providing a full pixel simulation pipeline for the entire Euclid mission and presented its application to SC8, the eighth testing campaign of the Euclid Ground Segment and its first end-to-end test. The SC8 dataset was generated from galaxy mocks built from the Flagship v1.0 simulation, the largest cosmological N-body simulation at the time it was developed. Additional sources were added to complement this massive simulation, such as high-redshift QSOs and strongly lensed galaxies. Our stellar catalogue was constructed from a combination of a very large and deep Besançon model and real all-sky stellar catalogues. From these catalogues, we produced and delivered simulated images for a simplified Euclid Wide Survey of 165 deg² and a full-featured 1 deg² Euclid Deep Survey.

The framework we adopted for SC8 follows a highly modular design from the onset, consisting of individual components for the TU catalogue inputs, pixel-level simulators for space- and ground-based instruments, a set of common tools for generating spectra and galaxy shapes, and a software API for orchestrating large sets of simulations at any of the international Euclid SGS data centres. Furthermore, all instrument models and reference surveys used in SC8 as well as all past or future challenges and production cycles are stored in the MDB repository. We also devised a validation program consisting of suites of tests for our simulation pipeline to ensure the accuracy of the released data products as well as adherence to scientific and technical requirements.

We shall continue to upgrade our simulation framework to enable large-scale testing and improvements to the ground segment pipeline. We plan to provide data products to aid the validation of very complex algorithms (e.g. VIS PSF estimation) and to use in conducting mission performance evaluations throughout this active development phase. Our longer-term goals during Euclid operations are to produce simulations to calibrate all possible systematics in the data (e.g. shear measurement biases and selection biases) and to interpret discrepancies or surprises found in the in-flight data.

Acknowledgements

We thank the OU-VIS, -NIR, -SIR, -EXT, and -MER teams for their assistance with validation and providing valuable feedback on the simulations. We are grateful to the System Team for the technical help and continuous support. We thank the Cosmological Simulations Working Group and the Flagship team for the outstanding galaxy mocks and support on their integration into OU-SIM. We thank both VIS and NISP instrument teams for providing instrument models and detailed explanations. We thank OU-SHE for their support with generating the VIS PSF dataset. We would like to thank Benjamin Clément from the Strong Lensing Science Working Group, who produced the images of the strong lenses detected in the SC8 main and deep fields which were presented in Figures 23 and 24. We thank Vincent Le Brun from the Organisation Unit for Spectral Measurement (OU-SPE) for producing Figure 25. We also express our gratitude to Dominique Boutigny, Jim Chiang, and Phillipe Gris of the Dark Energy Science Collaboration as well as Jim Bosch, Tim Jenness, Lynne Jones, Kian-Tat Lim, and the other members from Rubin Observatory Data Management and Survey Strategy teams for providing support, useful insight and sample datasets to aid us with upgrading the SGS Rubin simulations and ensuring compliance with the LSST Science Pipelines. We would also like to thank Jesus Varela and JEDIS, WISHES, Pan-STARRS and WHIGS teams for assisting us with information, data and the models needed to produce the respective image simulations for the Northern Surveys. We acknowledge all Science Data Centres (SDC-CH: University of Geneva, SDC-DE: Max Planck Computing and Data Facility, SDC-ES: Port d’Informació Científica, SDC-FI: CSC -IT Center for Science, SDC-FR: Centre de calcul de l’IN2P3, SDC-IT: INAF Astronomical Observatory of Trieste/ALTEC S.p.A. in Turin, SDC-NL: University of Groningen, SDC-UK: Institute for Astronomy, University of Edinburgh, and SDC-US: Caltech/IPAC) that contributed to the production of simulations for SC8. A portion of this work also used the resources available at CosmoHub. CosmoHub has been developed by the Port d’Informació Científica, maintained through a collaboration of the Institut de Física d’Altes Energies and the Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas and the Institute of Space Sciences, and was partially funded by the “Plan Estatal de Investigación Científica y Técnica y de Innovación” program of the Spanish government. The author acknowledges support from the Spanish Research Project PID2021-123012NA-C44 [MICINNFEDER]. This work has been financed by MCINN/AEI and the EU NextGeneration/PRTR project EQC2021-007479-P. This study was supported by MICIIN with funding from European Union NextGenerationEU(PRTR-C17.I1) and by Generalitat de Catalunya. The authors acknowledge the Euclid Collaboration, the European Space Agency, and a number of agencies and institutes that have supported the development of Euclid, in particular the Academy of Finland, the Agenzia Spaziale Italiana, the Belgian Science Policy, the Canadian Euclid Consortium, the French Centre National d’Etudes Spatiales, the Deutsches Zentrum für Luft- und Raumfahrt, the Danish Space Research Institute, the Fundação para a Ciência e a Tecnologia, the Ministerio de Ciencia e Innovación, the National Aeronautics and Space Administration, the National Astronomical Observatory of Japan, the Netherlandse Onderzoekschool Voor Astronomie, the Norwegian Space Agency, the Romanian Space Agency, the State Secretariat for Education, Research and Innovation (SERI) at the Swiss Space Office (SSO), and the United Kingdom Space Agency. A complete and detailed list is available on the Euclid web site (http://www.euclid-ec.org).

Appendix A Throughputs

Spectra and band fluxes were computed using the total transmission curves estimates for VIS (I_E), NISP-P (Y_E, J_E, H_E), NISP-S (Red and Blue Grisms), and each EXT survey (e.g. Gaia, DES, LSST, and UNIONs) provided by each of the respective instrument teams. We describe below the generation and/or acquisition of the passbands for each instrument and their status for SC8 where applicable:

• VIS The VIS total transmission was computed using the product of the MDB parameters: the VIS quantum efficiency and the PLM VIS transmission curves provided by the VIS Instrument team and Airbus Space and Defense, respectively.

• NISP We used the current best estimates of the NISP photometric passbands (Y_E, J_E, H_E) and the transmission curves for the red and blue grisms produced by the NISP Instrument Development Team and validated and crossed-checked by the SGS. The datasets used for SC8 are stored in the MDB, and the latest current best estimates are described in Euclid Collaboration: (2022c).

• Gaia As stated in Sect. 3.2.4, Gaia band fluxes were derived from the DR2 release (Gaia Collaboration 2018)²⁰.

• DES We used the DR1 griz passbands (Abbott et al. 2018)²¹.

• LSST The baseline total throughput curves (CCDs+Filter+Optics+Atmosphere) at airmass=1.2 (Kahn et al. 2010; Claver et al. 2014; Connolly et al. 2014) were computed by the LSST system engineering team. For SC8, we used the earlier release-1.4 total throughputs for ugriz²².

• CFIS The bandpasses for u and r filters, respectively, were generated with the information for the MegaCam filter set²³.

• JEDIS The JEDIS team provided full efficiency curves (CCD+Filter+Optics+Atmosphere) in the g band for 14 CCDs measured in a grid of 12 × 13 points, which we used to compute an average throughput.

• Pan-STARRS PS1 The bandpass model for the i band²⁴.

• WISHES The bandpass data for the z band²⁵.

Appendix B Spectra and fluxes

Appendix B.1 Galaxy spectra reconstruction

Here we describe the process for reconstructing the observed spectra of the galaxies from the parametrised information in the catalogue.

We begin by homogenising all wavelength-dependent elements into a common uniform sampling to enable direct operation between models that were sampled at different wavelength ranges and resolutions. To ensure the spectra cover the wavelength range of interest and with enough resolution, we set a range between 600 Å and 24,000 Å, in steps of 1 Å. With a spectral resolution of R = 480 in NISP-S, we are able to sample properly the dispersed spectra.

Our reconstruction algorithm commences with the initialisation of a spectrum. The SimSpectra library allows for the linear combination of two galaxy templates, enabling a wider variety of colours and shapes, which gives more realism to the simulation. We compute the combined template as follows: $$f_{\text{combined}}(\lambda )=(1-i_{\text{frac }})f_{\text{template}}(\lambda )[i]+i_{\text{frac }}{f}_{\text{template }}(\lambda )[i-1]\text{, }$$(B.1)

where i_frac is the fractional contribution of template with index i in the combined template. Templates are ordered from early to late type galaxy in the SED library. High-z galaxies and QSOs use only one template. Next, we apply one out of 5 dust extinction laws taken from Prevot et al. (1984); Calzetti et al. (2000) and presented in Ilbert et al. (2009). The dust extinction curve k(λ) is scaled by the galaxy colour excess E(B − V) and applied to the combined template, such that $$f_{\text{ext}}(\lambda )=f_{\text{combined}}(\lambda ){10}^{-0.4E(B-V)k(\lambda )},$$(B.2)

where the extinction curve k(λ) is $$k(\lambda )=-2.5{\text{\hspace{0.17em}log}}_{10}\left(\frac{E_j(\lambda )}{E_0(\lambda )}\right)E(B-V{)}_{0.2}^{-1}.$$(B.3)

where E_j(λ) corresponds to the extinction law used (Prevot et al. 1984; Calzetti et al. 2000; Ilbert et al. 2009), with j = 0 corresponding to no extinction, j = 1 to the Prevot extinction law, j = 2 to the Calzetti extinction law with no 2175Å bump and j = 3 and j = 4 to the Calzetti extinction law with different strengths of the 2175Å bump (see Ilbert et al. 2009 for details). For computational ease, we compute all the extinctions in the filter passbands for a value of E(B − V) = 0.2 and then re-scale the value to the actual E(B − V) in Eq. B.2.

The next step is to shift the spectrum to the observed redshift (including peculiar velocities), z_obs. This operation is performed by scaling the wavelength array elements by (1 + z_obs) and dividing the fluxes by the same factor (1 + z_obs). With the redshifted spectrum, we can measure the flux in the reference filter, and scale it to the reference apparent AB magnitude. For standard and strongly lensed galaxies, the reference filter is the SDSS r passband redshifted to z = 0.1. For high-redshift galaxies and QSOs, the template is scaled according to the apparent magnitude at 1500 (1 + z_obs) Å and 1450 (1 + z_obs) Å respectively.

For standard and strongly lensed galaxies, the reference magnitude to scale the spectrum is determined from the absolute magnitude in the SDSS r band, M_r. The M_r magnitude is a product of the semi-analytic recipe in the simulation, and is given in h-normalised units. The conversion from absolute to apparent magnitude requires computing the luminosity distance D_L and the K_Corr correction.

We compute the luminosity distance with a Flat LCDM cosmology, setting the Hubble constant to H₀ = 100 km s⁻¹ Mpc⁻¹, and the non-relativistic matter density parameter to the value used in the Flagship simulation Ω_m = 0.319.

As our reference flux is not bolometric but computed within a particular passband, we need the K_Corr correction term to transform from observed to restframe flux to account for the galaxy redshift. This correction term, expressed in magnitudes, is defined as $$K_{\text{Corr }}=2.5{\text{\hspace{0.17em}log}}_{10}\frac{f_{\text{restframe}}}{f_{\text{observed }}};$$(B.4)

where we obtain both the rest-frame, f_restframe, and the redshifted f_observed fluxes integrated in the reference passband using the following formula $$f_{\nu }=\frac{\int f_{\lambda }R(\lambda )\frac{\lambda }{c}\mathrm{d}\lambda }{\int R(\lambda )\mathrm{d}\lambda /\lambda },$$(B.5)

where R(λ) is the system response of the reference magnitude passband in photon counts, resampled at 1Å resolution. With the K_Corr correction and the luminosity distance d_L in parsec, we compute the apparent magnitude, $$m_r=M_r+5[{\log }_{10}\left(d_{\mathrm{L}}\right)-1]+K_{\text{Corr }}.$$(B.6)

We note that the absolute magnitude and the luminosity distance are computed for the same value of the Hubble constant and therefore its dependence can be disregarded in the equation making the apparent magnitude independent of its value.

The scaled observed flux in erg cm⁻² s⁻¹ Hz⁻¹ at the reference passband from the AB definition is then $$f_{\nu ,r}={10}^{-0.4m_r+48.6},$$(B.7)

and the scaled continuum spectrum is therefore $$f_{\text{continuum}}(\lambda )=f_{\text{ext}}(\lambda )\frac{f_{\nu ,r}}{f_{\nu ,\text{observed }}}.$$(B.8)

At this stage, we add the emission lines to the scaled template. As emission lines are already scaled and modelled with extinction, we compute them after applying the dust to the scaled continuum. The central frequency of each line is taken from vacuum or air, depending on whether the channel we simulate is ground or space-based. In Table B.1, we list the set of emission lines and their corresponding central wavelength used for both air and vacuum in the simulation.²⁶

We estimate the width of the emission line, σ_el, in km s⁻¹ following a luminosity-velocity dispersion relation (e.g. Faber-Jackson Faber & Jackson 1976 and Tully-Fisher relation Tully et al. 1975) as shown below: $${\log }_{10}\left({\sigma }_{\mathrm{e}\mathrm{l}}\right)=(-0.10+0.01z_{\mathrm{o}\mathrm{b}\mathrm{s}})(M_r-3.0)-0.05z_{\mathrm{o}\mathrm{b}\mathrm{s}}.$$(B.9)

High-redshift galaxies and QSO templates already include emission lines. We limit the approximation to a minimum velocity of 50 km s⁻¹ to avoid too narrow lines. We generate all 10 emission lines as Gaussian profiles with the integrated flux as defined in the catalogue at the corresponding central wavelength with a width equal to σ_el. As was done for continuum spectra, the set of reconstructed emission lines are shifted in wavelength by the factor (1 + z_obs).

The velocity dispersion is converted to wavelength and scaled to the observed frame as $${\sigma }_{\mathrm{e}\mathrm{l},\lambda }={\sigma }_{\mathrm{e}\mathrm{l}}\frac{{\mu }_i}{c}(1+z_{\mathrm{o}\mathrm{b}\mathrm{s}}).$$(B.10)

Then, we compute the spectrum containing only the emission line contribution, f_el(λ), as $$f_{\mathrm{e}\mathrm{l}}(\lambda )={\displaystyle \sum _i^{10}}\frac{{\left[f_{\mathrm{e}\mathrm{l}}\right]}_i}{{\sigma }_{\mathrm{e}\mathrm{l},\lambda }\sqrt{2\pi }}\exp (-\frac{{(\lambda -{\mu }_i)}^2}{2{\sigma }_{\mathrm{e}\mathrm{l},\lambda }^2}\frac{1}{1+z_{\mathrm{o}\mathrm{b}\mathrm{s}}}),$$(B.11)

where μ_i and [f_el]_i are the central wavelengths and TU line fluxes, respectively, for the ten emission lines in the corresponding medium.

The resulting line profiles can now be added to the calibrated continuum component of the spectra with the following expression $$f_{\text{galaxy }}(\lambda )=f_{\text{continuum }}(\lambda )+f_{\mathrm{e}\mathrm{l}}(\lambda ).$$(B.12)

With all the elements from the source galaxy, we introduce the effect of lensing magnification with $$f_{\text{magnified }}(\lambda )=f_{\text{galaxy }}(\lambda )\mu ,$$(B.13)

where μ is the magnification factor as defined in Eq. 4.

The final remaining step is to apply the Milky Way extinction to the magnified galaxy spectra at the specific location in the sky. From the Planck dust maps (Planck Collaboration XI 2014), we obtain a precise colour excess E(B − V), and following an O’Donnell (1994) extinction law with the ratio of absolute to differential extinction R_V = 3.1, we apply the MW extinction to obtain the final incident spectra: $$f_{\text{incident }}(\lambda )=f_{\text{magnified }}(\lambda ){10}^{-0.4A(\lambda )},$$(B.14)

where the extinction law A(λ) is the product of the extinction terms A(λ) = E(B − V)R_V.

Appendix B.2 Stellar spectra reconstruction

With the already available apparent magnitude in the H band we can simply compute the template flux f_v,template in the observed band as defined in Eq. B.5 and the 2MASS H-band flux f_v,H as Eq. B.7 and scale the star spectrum: $$f_{\text{star }}(\lambda )=f_{\text{template }}(\lambda )\frac{f_{\nu ,H}}{f_{\nu ,\text{template}}}.$$(B.15)

The only remaining step is to correct the Galactic extinction. The Tycho2 stars are not extinction corrected and therefore we do not apply further correction. For the additional stars with BaSeL and LT spectra, we apply a procedure similarly to the galaxy spectra as $$f_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}}(\lambda {)}_{\mathrm{e}\mathrm{x}\mathrm{t}}=f_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}}(\lambda ){10}^{-0.4A(\lambda )},$$(B.16)

with the O’Donnell (1994) law, R_V = 3.1, and the absorption coefficient A_V determined assuming the Drimmel et al. (2003) 3D Galactic extinction model.

Even though the wavelength range of the stellar templates cover the whole spectrum of interest, the reddest part from 12 000 Å onwards has a reduced sampling in the Basel 2.2 library with 50 Å intervals. The wavelength calibration in NISP-S has to be accurate to better than half a pixel, which translates to approximately 6 Å. Therefore, we would need stellar spectra sampled at 2 Å to 3 Å intervals. For this reason we replaced the feature-less low sampled redder part of the spectrum with real high resolution measurements following the continuum as defined in the template.

With support from OU-SIR, we did a fit for the spectral continuum for each BaSEL spectrum, using functions provided by our redshift and spectra measurements package EZ,²⁷ resampled the smooth continuum to a higher resolution, and added the continuum-subtracted spectrum of a star taken from the NASA Infrared Telescope Facility IRTF Spectral Library²⁸ to the high-resolution continuum.

From the full IRTF library we have taken ten stars at random for each spectral class (F, G, K, M – earlier classes O, B, and A are not covered by the library), primarily of luminosity class III (giants stars) and V (main sequence stars), but with at least one star of class either I (supergiants) or IV (subgiants).

The choice of which IRTF continumm-subtracted spectrum to add to the BaSEL continuum-only spectrum was based on broad-band colour: the IRTF template with the Y − H colour closer to the one stored in the original catalog was chosen as the match. There was no attempt to include also the luminosity or other properties in the match. So a very red star was assigned an M type spectrum, without considering if the star luminosity was compatible with an M-type object or not.

In this way we do not alter the colours of the spectra but we allow accurate wavelength calibrations in the simulated NISP-S channel.

Appendix B.3 Synthetic photometry

We need to generate synthetic photometry from the high resolution spectra to reproduce the passband fluxes that each instrument will measure. This information is provided in the catalogue for the EXT surveys simulations, and for validation purpose only for the Euclid instruments simulators, because these later make use of the true spectra to take into account all chromatic instrumental models.

Starting from the high resolution spectra, we compute the integrated flux density of the source f_v in erg cm⁻² s⁻¹ Hz⁻¹ for a specific passband R(λ) using Eq. (B.5). Then, the synthetic magnitudes in the AB system are obtained with the following transformation: $$m_{\text{syn }}=-2.5{\text{\hspace{0.17em}log}}_{10}\left(f_{\nu }\right)-\mathrm{48.6.}$$(B.17)

Appendix C SC8 preparation

Appendix C.1 Input catalogues

Galaxy and stellar catalogues containing the variety of sources described in Sec. 3 were produced for both the SC8 main (wide) and deep fields and ingested into the EAS. In particular, for the SC8 main run, the spatial extent of the galaxy and stellar catalogues exceeds ~ 600 deg², the original wide area survey planned for SC8. We added an additional margin to ensure complete coverage for the EXT ground-based survey simulations. Figure C.2 shows the footprint of the standard galaxy catalogues and we refer the reader to Figure C.1 in Sec. 3.2 which shows the footprint of the star catalogues used for the SC8 main run.

Standard galaxies, high-redshift galaxies, and QSOs are directly stored in CosmoHub²⁹ (Tallada et al. 2020) just after being produced by SciPIC. Taking advantage of the interactive data exploration and fast distribution capabilities of SciPIC, a dedicated team has validated the catalogues. These three catalogues, plus the one containing the strongly lensed galaxies, are then stored as distinct FITS files in a format defined in the Euclid data model. The catalogues are further divided into different files covering non-overlapping sky areas for each type of object. Each catalogue file covers a HEALPix pixel of N_SIDE = 2⁵ (corresponding to an area of about 3.35 deg²), except for strong lensing galaxies which are all included in a single file. All the FITS files along with their associated metadata are subsequently ingested in the EAS.

Stellar catalogues were stored on a set of non-overlapping FITS files for each kind of object, covering the original SC8 footprint with an area ~600 deg². The catalogues are arranged on a grid with different cell sizes depending on the stellar density of the area. As shown in Fig. C.1, there is an inverse relation between the area covered by a catalogue file and the density of objects.

In Table C.1, we report the number of objects per galaxy/star type for both wide and deep areas. For the SC8 main run, we produced a total of > 1 billion objects of which 922M were standard Flagship galaxies (described in Sec. 3.1). The catalogues are divided among 3078 files for a total 254 GB of disc space.

Appendix C.2 Feasibility analysis and validation

During the pilot phase, we ran profiling tests of each simulator code and instrument channel for the six EXT surveys. Scaling up the total processing time of the pilot phase to the SC8 main run, we estimated a total simulation time in excess of four million CPU hours to generate full-featured simulations over the SC8 main run target area. We assessed the cost of each feature identifying those effects for which the computational expense was high, and yet, had minimal impact on image quality and mission performance. This would enable us to introduce simplifications to carry out the main run production within the appropriated computing budget. Finding the balance between feature complexity and resource usage required several iterations, which thus prolonged the time and effort to validate each resulting product. The validation of the pilot phase data products and specific features contained therein for each simulation channel (i.e. VIS, NISP-P, NISP-S, and EXT) was performed by different teams using independent approaches, which we describe further below.

For VIS, short-loop validation of the simulated pilot phase data products were performed by the level-2 OUs and reported any issues identified. We, in turn, worked to resolve the errors reported, delivering new VIS images to be analysed until no further problems were raised.

For the NISP-P channel, validation of the simulation products was performed by the NISP-P validators in SDC Finland. We performed a separate evaluation of the technical requirements during the pilot phase, which revealed the high computing cost for the NISP-P simulator Imagem would exceed the CPU time constraints for the SC8 main run. Therefore, we decided to reduce the intra-pixel sampling to speed up the processing by a factor of 4. We verified the astrometric accuracy of all sources to be generated below one milli-arcsecond RMS for both the relative (between images) and absolute astrometry (with respect to the TU catalogue). This ensured that a lower resolution intra-pixel response simulation would not affect the positioning accuracy. The photometry error was less than 0.02 mag with respect to the TU reference magnitude in both aperture and PSF photometry, which also suggested a minimal dependence to lower intra-pixel sampling.

In our validation of the spectroscopic channel for NISP, we verified the χ² distribution quality map was consistent with the expected performance of the H2RG detector. Our validation tests confirmed the extracted flux for stellar sources reproduced sufficiently accurately the expected TU flux, with occasional outliers overestimating the flux due to contamination from other sources. This was expected as the validation analysis was not optimised to perform accurate decontamination of nearby spectra. The polychromatic PSF was simplified by implementing a stacked monochromatic version; improving the computational time by a factor of 7.

We validated the SIM-EXT science data products for astrometry, photometry, background and PSFs for each EXT survey. We used the SourceExtractor software to identify sources and measure fluxes and compared the derived positions and fluxes with the reference TU catalogue values. The results of our validation tests showed that the generated EXT data products were accurate and consistent with the reference input values.³⁰ Finally, to help reduce computational costs, we decided to simplify the galaxy simulation across all simulation channels; rendering very small galaxies below $$0_{\cdot }^{\mathrm{\prime }\mathrm{\prime }}01$$ with a simpler exponential profile, instead of the more complex inclined Sérsic profile.

Appendix C.3 Pilot phase processing

The main objective of the pilot phase was to profile and validate the OU-SIM pipeline in preparation for the SC8 main run. Extrapolations of the pilot phase computing costs revealed the resources available for SC8 would not be sufficient to generate full-featured simulations for a Euclid Wide Survey of 600 deg² within the appropriated time frame. While strategies were being implemented to lower computing costs such as the reduction to a smaller area of 165 deg² and the simplification of certain features for the NISP instruments (e.g. intrapixel sampling and monochromatic stacked PSFs), as many as 19 new features and data products (e.g. a cosmic ray map, a ghost map, stars-only image, etc.) were requested to be simulated for VIS to test and validate the OU-VIS processing and analysis software. Thus, while the pilot phase area only included four Euclid observations overlapping one Euclid tile and a small sample of EXT pointings per band, multiple productions were launched with each new release due to new and/or modified features. In Table C.2, we list all products produced for a given instrument channel during the pilot phase, noting the number of images per production, number of production sequences and the final number of products and storage requirements. For the Euclid instruments, as each observation has 4 dithers a total of 16 images were to be simulated for VIS, 48 images for NISP-P, and 16 images for NISP-S for the pilot phase production. However, we performed 12, 14, and 6 productions, respectively for VIS, NISP-P, and NISP-S channels yielding 192, 672, and 96 data products. As seen in Table C.2, multiple productions were also carried out for each EXT survey. A total of 3,614 exposures were produced during the pilot phase. Furthermore, for Rubin and the Northern Surveys, each simulated detector frame comprising an image was saved as an individual file. Thus, the total number of data products ingested into the EAS was 64,679 amounting to 7.1 TB.

For each production sequence, we characterised the resources needed for each processing element (i.e. memory, CPU, processing time, and storage). This was to assess whether the resources originally requested to the HTC systems at each SDC for each payload job would suffice for the SC8 Main run. In Table C.3, we report the final profiling results of the main processing elements of the SIM pipeline described in Sec. 5 for each instrument.

We used the upper-bounds reported to issue a request for resources to the HTC system, which had to be sufficiently higher than the observed maximum but somewhat conservative to optimise SDC performance and capacity. Table C.4 lists the resource characterisations for each software release and the configuration of each processing element used to produce image simulations for the SC8 main run.

Appendix C.4 SC8 main survey files

The SC8 main area was reduced from a continuous region measuring 600 deg² to 11 patches with a total area of approximately 165 deg². Figure C.3 shows the distribution of Euclid observations among 11 discontiguous regions. For the EXT surveys, we subdivided the original SC8 main area into three non-overlapping patches. The area above 30° in Declination was covered by the Northern Surveys in certain broadbands: CFIS (u and r bands), JEDIS (g band), Pan-STARRS (i band), and WISHES (z band). The rectangular region below 30^° in Declination was further sub-divided into two sections along a diagonal to allow a range of sky conditions in both segments. The area above the diagonal contained the set of Rubin pointings for each broadband filter (e.g. u, g, r, i, and z bands), whereas the region below the diagonal contained only DES pointings for the bands (g, r, i, and z bands). OU-EXT provided a set of files containing the respective observing cadence strategies for each survey from which we selected exposures that intersect the SC8 main area for processing. The preparation of each survey file is described below and their corresponding file contents are summarised in Table C.5:

• CFIS The pointings from the available CFIS data for SC8 preparation did not cover the whole SC8 area of the sky. Therefore, the pointing pattern from another area of the sky with similar declination was translated to the SC8 area as shown in Figure C.4.

• JEDIS By the start of SC8, JEDIS observations had yet to be performed. The pointing pattern was generated following procedures outlined by the JEDIS team. For any given sky location, the pattern consists of 14 overlapping pointings with a dithering step of the size of the JEDIS CCD, so that each point in the sky is imaged with all the 14 CCDs of the camera. The JEDIS SC8 footprint and depth in layer count can be seen in Figure C.5.

• Pan-STARRS A pre-selection of all pointings overlapping the SC8 area was performed by OU-EXT. They reduced the original number of pointings to 6.5 times of the total in order to minimise data storage. To reach the required depths, the exposure times were increased from 240.0 s to 292.5 s. The saturation limit of the CCDs was increased accordingly in order to retain the saturation limit of the stars. Figure C.6 shows the Pan-STARRS PS1 SC8 footprint and the depth indicated by the contours denoting the layer count.

• WISHES The survey file in the z band was built from real WISHES pointings overlaying the SC8 area (as seen in Figure C.7 which includes the corresponding number of layers).

• Rubin The LSST scheduler (Naghib et al. 2019) and associated simulated pointing histories (OpSim outputs) provide many examples of survey strategy options (Delgado & Reuter 2016; Connolly et al. 2014). For this work, a survey table was derived courtesy of the Dark Energy Science Collaboration (DESC) using the wide-fast-deep output from the v1.5 simulation, footprint_sky_dustv1.5_10yrs (Jones et al. 2020), which we downsampled in order to reach the desired limiting magnitudes in ugriz for the Euclid Wide Survey. We show in Figures C.8 and C.9, respectively, the number of layers in the u and g bands for the Rubin SC8 main survey. Magnitude zero points estimates for each bandpass were provided by DESC. Furthermore, the same position angle was used for all visits; therefore, we did not not account for rotation of the camera denoted by the OpSim parameter ‘rottelpos’.

• DES Similar to CFIS, the SC8 region is also outside of the DES footprint, observations were extracted from the same range of Declination in the southern celestial hemisphere at a shifted Right Ascension and transformed to cover the SC8 area. An additional margin of 1.05 deg, half of DECam FOV, (Flaugher et al. 2015), was applied to the selection to ensure that all exposures overlapping the target area were included. Fig. C.10 shows the resulting DES SC8 footprint and the blue contours indicate the number of layers to reach the desired depth for bands griz.

References

---

All Tables

All Figures