Polarization aberrations in next-generation giant segmented mirror telescopes (GSMTs) I. Effect on the coronagraphic performance

Next-generation large segmented mirror telescopes are expected to perform direct imaging and characterization of Earth-like rocky planets, which requires contrast limits of $10^{-7}$ to $10^{-8}$ at wavelengths from I to J band. One critical aspect affecting the raw on-sky contrast are polarization aberrations arising from the reflection from the telescope's mirror surfaces and instrument optics. We simulate the polarization aberrations and estimate their effect on the achievable contrast for three next-generation ground-based large segmented mirror telescopes. We performed ray-tracing in Zemax and computed the polarization aberrations and Jones pupil maps using the polarization ray-tracing algorithm. The impact of these aberrations on the contrast is estimated by propagating the Jones pupil maps through a set of idealized coronagraphs using hcipy, a physical optics-based simulation framework. The optical modeling of the giant segmented mirror telescopes (GSMTs) shows that polarization aberrations create significant leakage through a coronagraphic system. The dominant aberration is retardance defocus, which originates from the steep angles on the primary and secondary mirrors. The retardance defocus limits the contrast to $10^{-5}$ to $10^{-4}$ at 1 $\lambda/D$ at visible wavelengths, and $10^{-5}$ to $10^{-6}$ at infrared wavelengths. The simulations also show that the coating plays a major role in determining the strength of the aberrations. Polarization aberrations will need to be considered during the design of high-contrast imaging instruments for the next generation of extremely large telescopes. This can be achieved either through compensation optics, robust coronagraphs, specialized coatings, calibration, and data analysis approaches or by incorporating polarimetry with high-contrast imaging to measure these effects.


Introduction
The polarization-dependent phase and amplitude pupil patterns originating from differences in the Fresnel coefficients (s and p polarization states) are called polarization aberrations (McGuire Jr & Chipman 1987;Chipman 1987).The polarization aberration theory to estimate these polarization aberrations and their effect on the point spread function (PSF) of optical systems has been very well developed and presented over the last two decades (Chipman 1989;McGuire Jr & Chipman 1988;McGuire & Chipman 1990, 1994;Sánchez Almeida & Martinez Pillet 1992;Sánchez Almeida 1994;Breckinridge et al. 2015).Chipman et al. (2018) describes the dominant polarization aberrations in optical systems to be diattenuation piston, ⋆ NASA Hubble Fellow.tilt, and defocus, and retardance piston, tilt, and defocus.Diattenuation is a polarization-dependent amplitude apodization calculated using the difference in maximum and minimum reflection or transmission through an interface.Retardance is a polarization-dependent phase aberration calculated from the difference in the maximally and minimally phase-delayed polarization state.The change in diattenuation and retardance can be expressed in elementary vector 1 shapes that show the orientation and magnitude of these aberrations.The diattenuationand retardance-piston axes are of uniform magnitude across the pupil and oriented in the same direction.The diattenuationand retardance-tilt axes rotate by π/2 around the pupil, and the magnitude increases linearly.The axes of diattenuationand retardance-defocus rotate by π around the pupil, and the magnitude increases quadratically.We refer the readers to chapter 10 of Chipman et al. (2018) to the maps of these polarization aberrations.
In the context of telescopes, for an unpolarized light incident on a Cassegrain telescope with a fold mirror, quadratic variation in phase and amplitude of the reflection coefficients from primary and secondary mirrors causes the polarization in X and Y to be astigmatic, giving rise to diattenuation-and retardancedefocus.In addition, the fold mirror's linear phase and amplitude variation cause diattenuation piston and tilt, and retardance piston and tilt, leading to a beam shift in the final image (van Holstein 2021).The polarization-induced on-axis astigmatism and chromatic aberration were estimated for an F/1.5 Cassegrain telescope by Reiley & Chipman (1992).The polarization aberrations were evaluated for the Solar Activity Measurements Experiments (SAMEX) Solar Vector Magnetograph by McGuire & Chipman (1989), and significant improvement was obtained in the polarization accuracy by minimizing the angle of incidence and the difference between the reflection coefficients.Clark & Breckinridge (2011) shows the compensation of Fresnel aberrations arising in low F-number telescopes due to mirror curvature and coating.
Recently, Breckinridge et al. (2015) and Chipman et al. (2015) estimated the polarization aberrations for a 2.4 m Cassegrain telescope with a fold mirror to understand the magnitude of these aberrations and their effect on the exoplanet detection.The diattenuation aberrations cause amplitude apodization, but have a smaller effect on the image quality, while the retardance aberrations (piston, defocus, tilt) give rise to a ghost PSF, leading to the ellipticity of the Airy disk.The intensity of the ghost PSF is estimated to be about one part in 10 −4 of the two primary PSF images for the X-and Y-polarized light.Though the magnitude is small, the ghost PSF has a complex structure and larger spatial extent.For an unpolarized star, the orthogonally polarized components (X and Y) have different polarization aberrations, and a regular adaptive optics system cannot optimally correct these aberrations simultaneously.Thus, in a coronagraphic system designed for exoplanet studies, the ghost PSF may overfill the focal plane mask and result in the burying of the terrestrial exoplanet signal.
The observations from the high-contrast imaging instruments at the ground-based large telescopes have already shown the beam shift introduced as a consequence of the retardance tilt from the telescope mirrors.The polarization aberrations caused by the fold mirror and the derotator system in the Very Large Telescope (VLT) give rise to a differential polarimetric beam shift of about 1 mas (57.3 nm) between the orthogonal polarization components, which manifests as positive and negative features on the opposite sides of the stellar PSF (Schmid et al. 2018).For the Gemini telescope, Millar-Blanchaer et al. (2022) have observed a retardance defocus of ∼3 nm between incident X and Y polarization states in GPI polarization observations.It has become evident that, although the magnitude of the polarization aberrations is small, they cause significant errors in the coronagraphic performance and bias high-contrast polarimetric measurements (Safonov et al. 2022;Millar-Blanchaer et al. 2022).Thus, the polarization aberrations are now modeled for some of the telescopes with a high-contrast imaging instrument to understand their effect on the achievable contrast and for designing the mitigation or calibration strategy (Breckinridge et al. 2018;Krist et al. 2018;Mendillo et al. 2019;Will & Fienup 2019;van Holstein et al. 2020).
The next generation of giant segmented mirror telescopes (GSMTs), the Extremely Large Telescope (ELT)2 , the Thirty Meter Telescope (TMT)3 , and the Giant Magellan Telescope (GMT) 4 , have the potential to expand the discovery space of the exoplanets from gas giant planets to rocky planets (orbiting M dwarfs) owing to their larger collecting area and greater resolution.Estimating the polarization-induced aberrations of these large telescopes is crucial for understanding their magnitude and effect on the high-contrast imaging observations of planets and disks.The spatially integrated polarization analysis for the TMT and ELT has already been presented by Anche et al. (2018c,b), andde Juan Ovelar et al. (2014), respectively, who show that the fold mirrors in the optical configuration are the primary source of polarization effects.In the case of ELT, de Juan Ovelar et al. ( 2014) estimated an instrumental polarization (IP) of 6% and crosstalk (CT) of 30% at a wavelength of 0.55 µm with the primary mirror of ELT considered as a monolith.For the TMT with primary mirror as a monolith, Anche et al. (2018c) estimates an IP of 4.5% to 0.6% and CT of 73% to 11% in the wavelength range of 0.4 µm to 2.583 µm for field angle and zenith angle equal to zero.Although these results indicate the source and magnitude of polarization introduced by the telescope optics, they focus more on the polarimetric instruments for these telescopes rather than the high-contrast imaging instruments.Yabar et al. (2022) have also analyzed polarimetric behavior for the segmented mirrors due to aging, dust deposition, and oxidation of the coating.However, a detailed analysis of the polarization aberrations and their effect on the final achievable contrast is required for all three large telescopes using their actual optical configurations, including primary mirror segments.As GSMTs will have a primary mirror of F/1 compared to the F/2 for the VLTs, we can expect the effect of the polarization aberrations to be more significant compared to the VLTs as the aberrations increase quadratically.
In this series of papers we investigate the effect of the polarization aberrations on the coronagraphic performance and stellar PSF structure incorporating the telescope dynamics (e.g., pointing variations), segment-to-segment variations, coating nonuniformity, missing segments, and wavefront control of the highorder adaptive optics system.In this paper we present the static polarization aberration model and the estimation of polarization aberrations and their effect on coronagraphic performance.
Section 2 provides a brief description of the optical configurations of all three telescopes.The summary of the polarization ray-tracing algorithm and estimation of polarization aberrations is given in Sect.3. The Jones pupil maps evaluated at the exit pupil of the telescope are given in Sect. 4. Section 5 shows the diattenuation and retardance variation on all the mirror surfaces for the coatings used in these telescopes.The description of the coronagraphs and simulations to estimate the achievable contrast is given in Sect.6. Mitigation and calibration strategies are discussed in Sect.7. Finally, a summary and conclusions are provided in Sect.8.

Description of the optical layout and proposed high-contrast imaging instruments
We provide a brief description of the optical layout and the proposed coronagraphic instruments for each telescope.Table 1 provides a summary of the optical design for all three telescopes.will have a multi-conjugate adaptive optics (MCAO) module (as a part of the second-generation capability) feeding the coronagraphic imager in the wavelength range of 0.8-2.4µm (Davies et al. 2021).One of the first-generation instruments of ELT, the Mid-Infrared ELT Imager and Spectrograph (METIS) 6 will also have a coronagraphic imaging capability with a single conjugate adaptive optics system in the wavelength range of 3-13 µm with a field of view (FOV) of 10 arcsec (Brandl et al. 2021).The instrument with eXtreme adaptive optics (XAO), the Planetary Camera, and Spectrograph (PCS), will perform coronagraphy with spectroscopic and polarimetric capabilities.

Thirty Meter Telescope
The TMT, with a 30 m diameter segmented primary mirror, has been proposed to be built on Mauna Kea, Hawaii, USA, and is expected to have its first light in the early 2030s.The telescope is a Ritchey Chrétien with a fold mirror (shown in Fig. 1) to direct light to different instruments on the Nasmyth platform.The Planetary Systems Imager (PSI) 7 is a proposed secondgeneration instrument with coronagraphic capability in both blue (0.6-1.8 µm) and red channels (2-5 µm and 8-13 µm) and is expected to achieve a post-processed contrast of 10 −8 at 2λ/D.This requires a raw contrast of 10 −5 at 1 to 2λ/D (Fitzgerald et al. 2019).The current design of PSI supports polarimetric imaging and fiber-fed high-resolution spectroscopy (R ∼ 100 000) (Fitzgerald et al. 2022).

Giant Magellan Telescope
The 25m diameter GMT is being built on Las Campanas Peak at the southern edge of Chile's Atacama Desert and is expected to have its first light in 2029.It is designed to support imaging at both the Gregorian and Nasmyth focus.The optical layout of GMT is shown in Fig. 1 with the fold mirror.The Giant Magellan Adaptive Optics eXtreme (GMagAO-X) instrument is one of the proposed first-generation instruments with coronagraphic capabilities.It will provide wavelength coverage from g band to K band with broadband imaging and a fiber-fed integral field unit (IFU) spectrograph with R ∼ 100 000.It is expected to achieve a post-processed contrast 10 −8 for an eighth magnitude star at a wavelength of 800 nm, which requires a raw contrast of 10 −5 at 1-2λ/D (Males et al. 2022).

Optical design of telescopes in Zemax
We used the exact segment prescription for the primary mirror of TMT and GMT.In the case of ELT, we simulated a monolithic primary mirror and incorporated the segment aperture mask after performing a ray trace (explained in the next section) in Ansys Zemax ® OpticStudio.For TMT and GMT, the telescope is designed in the mixed mode (combination of sequential and non-sequential mode), where the segments are added as nonsequential components.First, each segment is specified using a user-defined aperture (UDA) file containing the positions in the global coordinate system (e.g., vertices of the hexagonal segments for TMT).Then, the secondary and Nasmyth mirrors of the telescope are added in sequential mode.

Polarization ray tracing algorithm
Modeling the effects of polarization in optical systems is typically conducted in the Jones or Mueller formalism.Jones calculus treats polarizing optical elements as 2 × 2 complex matrices and traces the evolution of the complex electric field.The Mueller calculus is a 4 × 4 matrix formalism that propagates the incoherent power in three polarization states and the degree of polarization.Because modern detectors can only measure the irradiance from optical fields rather than the fields themselves, the Mueller calculus represents the irradiance distribution that can be calculated from a system.However, we require the Jones representation to trace the influence on the k q-1 η q k q Pq Fig. 2. Diagram illustrating a typical polarization ray tracing interaction in reflection with the q-th surface in the optical system.The wave vector before the surface k q−1 is incident on the surface with a linear polarization shown in purple.The angle of reflection is determined by the angle of incidence with respect to the surface normal η q , which points into the surface to maintain a right-handed coordinate system.Next, the polarization state is transformed by the surfaces PRT matrix P q and propagated along the existent wave vector k q .Here P q is a partial retarder, so the polarization state becomes elliptical.
complex amplitude to simulate the diffraction effects in highcontrast imaging instruments.Therefore, we employ polarization ray tracing (PRT) to calculate the total Jones matrix experienced by every ray propagating through the optical system.These Jones matrices can be converted to Mueller matrices (Chipman et al. 2018) to retrieve the final intensity in response to an unpolarized star.
Polarization ray tracing is a method of computing how the polarization state transforms through an optical system (illustrated in Fig. 2).The polarization state is propagated along geometrical ray paths in global coordinates through the optical system.When a ray encounters the surface, the polarization state is rotated into the local coordinate system of the ray-surface interaction.The orthogonal transformation matrices (O in/out ) are constructed from the eigenpolarizations of the local surface and the surface normal (η).In the local coordinate system the Fresnel reflection coefficients are computed for each eigenpolarization and organized into a diagonal matrix J q .The matrix that encodes the orthogonal transformations and Fresnel reflection coefficients is called the PRT Matrix, P q , and is computed using A121, page 4 of 17 Anche,R. M.,et al.: A&A proofs, Eq. ( 1): (1) The influence of the entire optical system is accomplished through a matrix multiplication of the P q matrices for Q optical elements: (2) The resultant total PRT matrix P tot represents the total threedimensional transformation of the polarization of light from the optical system in global coordinates.To transform this matrix into something useful for diffraction models of coronagraphs, we must compute the Jones pupil.The Jones pupil is the result of an orthogonal transformation of the P tot matrix into the local coordinates of the exit pupil.To accomplish this transformation, we derive the basis vectors of the entrance pupil and exit pupil and organize them into orthogonal transformation matrices (O EP and O XP , respectively).The Jones pupil is computed by solving for J tot in Eq. ( 1) using O EP as O in and O XP as O out : The J tot matrix should be zero valued in the last row and column, except for the element on the diagonal, which should be unity.This indicates that the matrix only operates on the components of the Jones vector orthogonal to propagation, which is where the electric field is located.There are several bases to choose from to derive the orthogonal transformation matrices for the entrance and exit pupils (Chipman et al. 2018): the s,p,k basis, dipole basis, and double-pole basis.We use the double-pole basis because of its insensitivity to polarization singularities.For an instructive description of the polarization ray tracing algorithm and double-pole coordinate system, we refer to Chapters 10 and 11 of Chipman et al. (2018).
In this work we perform ray tracing for an array of 256× 256 rays in Zemax ® using the Python ZOS-API, and generate Jones matrices using a polarization ray tracing module (based on Chipman et al. 2018) developed in Python8 (Ashcraft 2022).The ray trace in Zemax ® provides the incident direction cosine and surface normal at each ray intercept and the corresponding angle of incidence.We calculate the direction cosines of the reflected ray and local s,p eigenpolarizations at each mirror surface which form the O out,q and O −1 in,q matrices.

Angle of incidence (AOI) on the mirror surfaces
The AOI is obtained by performing a ray trace in Zemax ® for all the mirror surfaces.The maximum AOI for the primary mirror is 16.26°, 14.01°, and 19.46°for ELT, TMT, and GMT, respectively.The secondary mirror has a similar range of AOI for all three telescopes.In the case of ELT, the maximum AOI on M3 and M4 is estimated to be 2.56°and 9.49°, respectively.For the final fold mirror, the AOI varies between 35.5°and 39.017°for ELT, 42.98°and 46.89°for TMT, and 41.5°and 48.5°for GMT, respectively.The AOI obtained from the ray trace in Zemax ® is shown for the mirror surfaces for TMT and GMT in Fig. 3, ignoring the spider structure.In the case of ELT, we show the mirror M5, which feeds the light to different instruments on the Nasmyth platform.The incident angles on M3 and M4 of ELT vary in the range 0-2.56°(center to edge) and 6.01-9.49°(along the y-axis), respectively.
The polarization introduced from the mirror surface increases with the AOI on the mirror surface as the difference between the reflection coefficients r p and r s increases (Giro et al. 2003).The primary and secondary mirrors have a smaller angle of incidences ∼15-20°and introduce nonzero (∼0.1%) instrumental polarization and crosstalk due to their segmented apertures.On the other hand, the fold mirrors with the highest angle of incidence contribute to a significant part (1-2% in V band) of the instrumental polarization and polarization crosstalk.In the context of polarization aberrations, M1 and M2 of TMT and GMT contribute to the diattenuation-and retardance-defocus, and M3 gives rise to diattenuation piston and tilt, and retardance piston and tilt.For ELT, M1, M2, and M3 contribute to the diattenuation-and retardance-defocus, and M4 and M5 give rise to diattenuation piston and tilt, and retardance piston and tilt.

Computation of Fresnel reflection coefficients from thin films
The ELT and TMT observatory mirrors are overcoated with a dielectric material to protect the reflective silver mirror coating.This dielectric alters the effective Fresnel reflection coefficient of the mirrors, which was considered in our model.To simulate the effective Fresnel reflection coefficient, we employ the method outlined in Macleod (2010), which is reproduced below.The effective Fresnel reflection coefficient is derived from the characteristic matrix of the thin film.This matrix is given by where δ q is the phase thickness of the qth film, given by Here, η q is the characteristic admittance of the qth film, which is defined as η q,s = n q cos(θ q ), (6) η q,p = n q / cos(θ q ), ( 7) and θ q is the angle of the wave vector in the qth film.Solving the characteristic matrix for the B and C coefficient yields the effective reflection coefficient (r eff ) of the thin film stack where η 0 is the characteristic admittance of the medium (free space).This formalism assumes that the substrate is a solid substrate of the reflective material (silver, aluminum) with a single dielectric coating as the skin depth of metals at optical frequencies is typically a fraction of the wavelength (Born & Wolf 1999) 9 .The resulting characteristic matrix is where SiN corresponds to Si 3 N 4 as the dielectric layer in our model.The ELT and TMT mirrors will have Gemini-like fourlayer coating.The ELT coating has 60 Å thick NiCrN x on the Zerodor substrate followed by 1100 Å of silver, 3 Å of NiCrN x , and finally 55 Å thick aluminum-doped Si 3 N 4 (Schotsaert et al. 2020).The TMT will have 65 Å thick NiCrN x on the Zerodor substrate followed by 1100 Å of silver, 6 Å of NiCrN x , and finally 85 Å thick Si 3 N 4 as the top layer (Anche et al. 2018c).The GMT mirrors will be coated with bare aluminum.In our analysis we do not consider NiCrN x for the ELT and TMT mirrors due to the unavailability of the refractive index information and the influence of the aluminum oxide layer for the GMT mirrors (Van Harten et al. 2009).However, we expect the impact of NiCrN x to be small due to the small phase thickness of this layer.Figure 4 shows the amplitude and phase of the reflection coefficients for five astronomical filter bands (b-N) for all three coatings.The amplitude of the reflection coefficients is >0.975 for TMT and ELT, as the coatings have been optimized for higher reflectivity.In contrast, the reflection coefficient phase varies by almost 1 radian over these wavelengths.For GMT, the amplitude of reflection coefficients varies from 0.90 to 0.98, and the phase varies on the order of 0.

Jones pupil maps
As explained in Sect.3, the polarization aberrations can be expressed in terms of the Jones pupil map (Jones matrices as a function of object and pupil coordinates) obtained using the PRT through the telescope.The Jones pupil map shows how incident X and Y electric fields are manipulated through the system.Ideally, in the absence of polarization aberrations, the Jones matrix obtained at the exit pupil will be an identity matrix, and the integrated on-axis PSF will be unpolarized.However, due Fig. 4. Variation of amplitude reflection coefficients in p and s with the angle of incidence for different astronomical filter bands.ELT and TMT mirrors will have a Gemini-like coating with silver as the main reflective layer and Si3N4 as the protective layer, whereas GMT will have a bare aluminum coating.
to the complex refractive indices of the coating and the curvature of the mirrors, it deviates from the ideal scenario.Figure 5 shows the Jones pupil map generated at the exit pupil of each of the telescopes in the V band.Axx and Ayy show the transmission for X and Y polarized light, respectively.Axy and Ayx show the amplitude of Y-polarized light converted to X-polarized light and vice versa.A shifted Maltese cross pattern is seen in Axy and Ayx for all three telescopes, indicating crosstalk between the X and Y polarization.The crosstalk terms Axy and Ayx are highly apodized for all three telescopes, and their amplitude is highest (7.45%) for the ELT and lowest for the GMT (3.5%).The difference in the Axx and Ayy terms is ∼0.6% for ELT and TMT and ∼3% for GMT, which can be attributed to the difference between the amplitude of reflection coefficients.
The panels ϕxx and ϕyy show the phase shift for X and Y polarized light, respectively, and ϕxy and ϕyx show the phase shift of Y-polarized light converted to X-polarized light and vice versa.ϕxx and ϕyy vary over the pupil for all three telescopes showing differential astigmatism between XX and YY, which manifests as retardance defocus and tilt.ϕxx and ϕyy vary ∼12 nm, 10 nm, and 7.5 nm over the pupil for all three telescopes.Comparing the amplitude apodization and phase variation in the Jones pupil, it can be seen that polarization aberrations in A121, page 6 of 17 ELT will have a larger impact on the coronagraphic performance than the other two GSMTs.These Jones pupils of the three telescopes are fit using the six analytical polarization aberration terms as shown in Breckinridge et al. (2015), and the coefficients of the aberrations are provided in Appendix B for all the filter bands.

Comparison of coatings: Diattenuation and retardance
To study polarization aberrations, it is convenient to decompose the Jones pupil into diattenuation and retardance.are computed from the eigenvalues of the Jones matrix (ξ x , ξ y ) where ∠ is the angle operator and ξ x , and ξ y are the eigenvalues of maximum and minimum polarization aberration, respectively.This operation is performed on the Jones pupil to examine the diattenuation and retardance expressed in the local basis vectors of the exit pupil, which serves as the entrance pupil of a coronagraph.We examine the spatial variation and performance versus astronomical band to assess how each polarization aberration influences the coronagraphic performance.Figure 6 plots the diattenuation and retardance for X polarization across the Jones pupil in the V band to reveal a shifted astigmatic pattern that is characteristic of Cassegrain-type telescopes with a fold mirror (Breckinridge et al. 2015).The protected silver coating used by the TMT and ELT is less absorbing in the V band when compared to the aluminum coating used by the GMT.However, due to the protective dielectric coating, the ELT and TMT experience greater retardance than GMT.Between the TMT and ELT, the ELT has lower peak-to-valley polarization aberration, which can be credited to the inclined fold mirror at 37.5°, which causes diattenuation and retardance tilt.Figure 3 shows that the primary and secondary mirrors have comparable angles of incidence.The TMT tertiary mirror imposes significantly greater polarization aberration due to the 45°angle of incidence.The ELT employs two separate flats with lower angles of incidence than the TMT tertiary (see Table 1).The sum of the contributions to the polarization aberration of the two mirrors is less than that of a single mirror with a higher angle of incidence, resulting in lower peakto-valley diattenuation and retardance than the TMT.To examine the performance versus the astronomical band, we compute the minimum, maximum, and mean values of the absolute diattenuation and retardance for each band.These data are plotted in Fig. 7 and substantiate the trends discussed earlier.The GMT diattenuation is the highest, particularly near the high absorption band in the visible and near-infrared (V − y band), followed by TMT and then ELT.The TMT has the greatest retardance, followed by ELT and then GMT.The overall polarization aberration (implicitly represented by the shaded regions in Fig. 7) tends to decrease with an increase in wavelength.
As an indicator of coronagraphic performance, retardance represents a polarization-dependent phase aberration that will directly shape the point-spread function supplied to the coronagraph.On the other hand, diattenuation represents a polarizationdependent amplitude smoothing rather than explicitly changing the distribution of the PSF.Therefore, we expect the telescopes with high retardance to have worse coronagraphic performance, and diattenuation will be of lesser consequence.To accurately understand the exact relationship of polarization aberrations to coronagraphic performance, we must inject a Jones pupil into a diffraction model of the coronagraph.

Effect on the achievable contrast
Any differential aberration between the X-and Y-polarization states will leak through a coronagraph because only the common aberration can be compensated.The Jones pupils from the ray trace of the GSMTs are used as an input to our high-contrast imaging simulations with the High-Contrast Imaging in Python (HCIPy) module (Por et al. 2018).The Jones pupils have to be post-processed before propagating through the coronagraphic simulations because they contain empty areas due to the finite ray sampling.The empty pixels are interpolated with a local second-order polynomial in x and y (1, x, y, xy, x 2 , y 2 ).The local interpolator uses a 5 × 5 pixel area around each empty pixel to determine the local polynomial coefficients.The 5 × 5 area is the minimum area size with the empty pixel in the center and enough pixels to fit a second-order polynomial.A 3 × 3 area creates very strong artifacts at the edge pixels where there are not enough pixels to constrain the solution.Higher-order interpolation or larger pixel areas did not significantly change the results.We kept the interpolation scheme at 5x5 to make it as local as possible.The interpolation is done in real imaginary space, which are continuous because amplitude and phase space are not due to phase jumps (see Fig. 5).
The now interpolated pupils still contain aberrations common between the X and Y states.An AO system cannot distinguish between phase aberrations from the atmosphere or the telescope from polarization aberrations when the system is observing a star.Therefore, the AO system will compensate for all common polarization aberrations between X and Y.This is incorporated in our simulations by taking the average between the phase of the xx and yy elements of the Jones pupil.The input A121, page 8 of 17 into our simulations is assumed to be an unpolarized star.HCIPy uses the Stokes vector to generate several fully polarized electric fields that add up to match the input Stokes vector.Each fully polarized electric field (e.g., an X field or Y field) can be propagated through the entire optical system.The impact of the aberrations on the contrast is estimated by propagating the electric fields through a set of perfect coronagraphs (PCs; Cavarroc et al. 2006;Guyon et al. 2006).These coronagraphs remove the first N electric-field modes.The lowest-order PC is a secondorder coronagraph that removes the piston electric field mode, while a fourth-order coronagraph also removes the tip and tilt electric fields.The PC is defined as an orthogonal projection operation: Here, E is the input electric field and E cor is the output electric field.The PC removes modes W i from the input.The operator < a, b > is the inner product between two functions a and b.This operation is applied to each polarization state separately.While the PC does not exist in reality, there are several coronagraphs that closely follow the performance, such as the Phase Induced Amplitude Apodization Complex Mask Coronagraph for segmented apertures (Guyon et al. 2010;Belikov et al. 2018) and the Vortex Coronagraph (Foo et al. 2005;Mawet et al. 2005) for clear apertures.For other coronagraphs, the residuals depend on the particular design for each instrument.However, the dedicated high-contrast imaging instruments for the GMT (Males et al. 2022), TMT (Fitzgerald et al. 2022), and ELT (Kasper et al. 2021) are currently under development, and there are no coronagraph designs yet.The PC provides a fundamental limit on the performance of more realistic coronagraphs, which is why these have been applied in each case in this paper.
The coronagraphic residuals for the TMT are shown in Fig. 8.They include a second-, fourth-, and sixth-order coronagraph in I band, H band, and L band.The results show that, after compensation of the common aberrations, a combination of diattenuation-and retardance-defocus is the dominant aberration.These are clearly visible in Fig. 5.The defocus caused by the diattenuation is at a similar level to that of the phase aberration, which was found by simulations that considered only phase or amplitude aberrations in the Jones pupils.There is also some differential polarization beam shift (i.e., tip and/or tilt), which is visible in the second-order coronagraph residuals.The differential beam shift causes blurring of the stellar residuals.The change to a fourth-order coronagraph that also removes tip or tilt modes makes this very apparent; the nulls in the diffraction pattern are much sharper and deeper.The results show that polarization defocus is more important for these large and fast telescopes (F/1) than polarization beam shifts, which are the dominant source of error in current groundbased telescopes (Schmid et al. 2018;van Holstein, in prep.).Higher-order modes barely play a role in the contrast budget.The sixth-order coronagraph residuals due to polarization aberrations are at a contrast level of < 10 −8 , which is well below the requirements of any ground-based direct imaging instrument.The downside of the sixth-order coronagraph is that its inner working angle increases compared to the second-and fourth-order coronagraphs (Belikov et al. 2021).Theoretically, it is possible to achieve an inner working angle of 1.5 λ/D with an optimal sixth-order coronagraph.This may be a sufficiently small enough inner working angle, depending on the exact requirements of the instrument.However, this is most likely not enough for GMagAO-X, which has an inner working angle goal of 1 λ/D.The coronagraphic residual images for the other two telescopes are similar in appearance and behavior and can be found in Appendix B.
The polarization aberrations are wavelength dependent.The chromatic behavior is summarized by using the peak raw contrast in the residual images for each wavelength.The peak raw contrast as a function of wavelength is shown in shows the upper limit of the stellar leakage due to polarization aberrations.The results show that switching from a second-order coronagraph to a fourth-order barely affects the contrast.A sixthorder coronagraph would gain almost four orders of magnitude across all wavelengths.The peak contrast goes down as the wavelength increases, which was expected based on the chromatic behavior of the diattentuation and retardance.The expected raw contrast of the next generation of high-contrast imagers is about 10 −5 at 1 λ/D and is set by the performance of the AO systems (Kasper et al. 2021;Males et al. 2022).Therefore, the polarization aberrations should be at most at the same contrast level, or they will be the dominant factor.The 10 −5 level is reached for a wavelength around H band for the GMT and TMT.The ELT has worse performance since it has five mirrors instead of three and the fastest primary mirror.The ELT reaches the 10 −5 level at wavelengths longer than K band.
The predicted raw contrast of the GMT is significantly better than that of the TMT, although both only have three mirrors.This could either originate from the geometric shape of the mirrors (incidence angles) or the coating specification (retardance and diattenuation).The simulations of the GMT were redone with two other coatings to disentangle the effects of geometry and coating.The first coating is identical to the TMT coating, and the second is a bare silver coating.The bare silver coating was investigated because the TMT coating is a multilayer structure with silver as its base.The results can be seen in Fig. 10.The nominal aluminum coating of the GMT has the best performance.It outperforms the other coatings by an order of magnitude for both coronagraphs in the visible wavelength range.The aluminum coating is still better at longer wavelengths, but the gap between the coatings closes.These simulations demonstrate that the significant gain for the GMT over the other two telescopes is its coating.The performance gain is lost when the GMT uses the TMT coating.The contrast curves from the bare silver coating also show that the main cause of the polarization aberrations is the silver layer, which accounts for nearly 80% of the residual contrast.

Mitigation and calibration strategies
There are several ways that the impact of polarization aberrations can be reduced or even completely removed.

Coating of the mirrors
The performance of the GMT is significantly better than that of the TMT, although both only have three mirrors.The main driver for the magnitude of the aberrations is the coating.The A121, page 10 of 17 ELT 2-order ELT 4-order ELT 6-order TMT 2-order TMT 4-order TMT 6-order GMT 2-order GMT 4-order GMT 6-order Fig. 9. Peak contrast as a function of wavelength and different coronagraphs for the ELT (blue), GMT (purple), and TMT (red).Each color represents a different telescope, and the line style changes for each coronagraph.The solid lines show the performance of a second-order coronagraph (theoretically "perfect" for an unaberrated system), the dashed lines that of a fourth-order coronagraph, and the dotted lines that of a sixth-order coronagraph.The center wavelength of each of the simulated photometric bands is shown at the bottom of the figure.The peak contrast an exponential decline with increasing wavelength.A peak contrast of 10 −5 is reached between H and K bands.The solid lines show the contrast for a fourth-order coronagraph, and the dashed lines show the for a sixth-order.The nominal aluminum coating for the GMT has the best performance.The difference in performance is even larger for the sixth-order coronagraph.polarization aberrations could be reduced by optimizing the coating recipe not only for reflectivity, but also for retardance and diattenuation.One method for minimizing diattenuation and maximizing reflectivity was created for the Multiangle Spec-troPolarimetric Imager (MSPI; Mahler et al. 2008).The coatings of the MSPI mirrors were nominally protected silver coatings with two dielectric layers on top.The thickness of the two dielectric layers for each mirror was optimized against a merit function that weighted diattenuation and reflectivity equally.By doing so, the total diattenuation of the system was reduced to <1% while maintaining a high reflectivity.A similar optimization could be conducted for the GSMT mirrors with additional consideration for retardance to mitigate the influence of polarization aberrations.As new coatings will be available in the future, this is a feasible approach for all the GSMTs as long as mirror coating facilities (the coating facility of the ELT and TMT has a modular design that is easy to upgrade) are designed to accommodate upgrades and delivery of different coatings (Schotsaert et al. 2020).

Compensation optics
The straightforward approach for reducing polarization aberrations is the design of mirrors with an optimized curvature radius to minimize the incidence angle, avoiding fold mirrors in the optical configuration.Nevertheless, in all these GSMTs, fast primary mirrors have to be used to get to buildable telescope sizes, and the use of fold mirrors is inevitable because of instruments placed on the Nasmyth platform.Therefore, one of the mitigation techniques used to cancel the retardance tilt or beam shift from the fold mirror is using another crossed-fold mirror with its s-p planes rotated orthogonally to the M3 mirrors.However, this approach requires a trade-off study between the polarization aberration cancellation with the additional wavefront error and throughput loss.Lam & Chipman (2015) simulated the compensation of polarization aberrations using crossed-fold mirrors for a single field point.They obtain residual aberrations of linear variation of retardance and diattenuation, which is easier to compensate with optimized orientations of the mirrors.A preliminary analysis of using crossed-fold mirrors to mitigate the A121, page 11 of 17 A&A 672, A121 (2023) instrumental polarization and crosstalk for TMT is explored by Anche et al. (2018a), where IP reduces to 0.1% from 4%.Since our simulations show that the prominent polarization aberrations in these telescopes are retardance defocus and tilt, we could compensate for these using a spatially varying retarder optimized over wavelengths of interest.The design and analysis of the compensation optics and the calibration strategies will be explored in detail in the following papers in this series.

Focal plane wavefront control
The coronagraphic simulations were performed after a classical AO system.The AO systems optimize the wavefront for maximum Strehl by compensating the wavefront aberrations.For high-contrast observations, this is not necessarily optimal because of stellar speckles that leak through the coronagraph.Focal plane wavefront sensing and control can create dark holes in the PSF where the contrast is enhanced with respect to the planet.Typically, the electric field in the focal plane is estimated with some sensor (pair-wise probing or phase diversity), and then the DM is actuated to create an electric field that destructively interferes with all the light in a particular region.This approach to wavefront control is called electric field conjugation (EFC).EFC has been used on several testbeds to create very deep contrasts (Seo et al. 2019;Ruane et al. 2022).It is now being implemented and tested for ground-based telescopes (Potier et al. 2022;Haffert et al. 2022;Ahn et al. 2022).
Electric field conjugation has also been extended to multiwavelength solutions and systems with phase and amplitude control by using multiple DMs.A similar approach with multiple DMs might make it possible to remove polarization aberrations.However, most EFC approaches have only been developed for scalar electric fields despite the fact that high-contrast testbeds are limited by polarization aberrations.The current testbeds try to reduce the influence of polarization aberrations by placing the coronagraphic arm of the instrument between polarizers to select only one polarization state.A single polarization state can be completely controlled.The downside to this approach is that half of the light is thrown away.This might not be an option for ground-based telescopes that still rely on advanced post-processing techniques that require high throughput.EFC can be extended to include control and sensing of both polarizations (Mendillo et al. 2021).The main problem for EFC is disentangling the electric field of each polarization state during the estimation process.This is highly degenerate and makes the estimation process more difficult.The model-free approach of implicit EFC (iEFC) that only uses intensity-based measurement could be a solution to this (Haffert et al. 2022(Haffert et al. , 2023)).Future work will show us whether the telescope-induced aberrations can be canceled to a deep enough level.

Post-processing
Any polarization aberration errors uncorrected by focal plane wavefront control will add a stellar photon noise floor, static or quasi-static speckles, to observations that cannot be completely removed.Speckle subtraction techniques such as reference differential imaging (RDI) and angular differential imaging (ADI) allow imaging that approaches the photon noise floor.The residual contrast due to the polarization aberrations discussed here should be largely static during observation, and thus well suited to removal by ADI (Marois et al. 2006).In addition, RDI using PSF libraries images from other stars (Soummer et al. 2011;Ruane et al. 2019) removes many of the observational constraints of ADI; however, through the life of an observatory, change to coatings may limit the effectiveness of RDI as will degenerate solutions to image plane wavefront control.

Summary and conclusion
The analysis of polarization aberrations is crucial for the nextgeneration GSMTs as they aim to reach an on-sky contrast of 10 −5 to 10 −4 .Below, we summarize the significant results from our simulations of polarization aberrations: 1. We estimated the polarization aberrations arising due to the telescope optics of next-generation GSMTs, including the segments of the primary mirror for all the astronomical bands; 2. Our analysis indicates the presence of diattenuation defocus and tilt, and retardance defocus and tilt as the prominent polarization aberrations, which cannot be corrected by the high-order adaptive optics instrument in these three telescopes; 3. The peak raw contrast that can be achieved in these telescopes for the different order of the coronagraphs decreases with wavelength pointing to the correlation with the behavior of the mirror coating.The peak contrast in the blue region is >10 −4 , which is far below the requirements of highcontrast imaging instruments for these telescopes.Contrast better than 10 −6 can be achieved only in the L, M, and N bands for the second-and fourth-order coronagraphs; 4. The raw contrast is estimated for the inner and outer working angle for the proposed high-contrast imaging instruments for each of the telescopes.For R and I bands the raw contrast at 1λ/D for the second-and fourth-order coronagraphs is > 10 −4 for ELT and TMT and > 10 −5 for GMT, which is an order less than the required contrast for the high-contrast imaging instruments; 5. We also find that the performance of the aluminum coating is better than the Gemini-like coating, although the Gemini-like coating is optimized for higher reflectivity.This indicates the necessity to develop a coating optimization technique to incorporate the coating retardance and diattenuation in addition to the reflectivity; 6.To overcome the beam shift caused by the retardance tilt of the fold mirror, it is crucial to design the compensation optics as a part of the high-contrast imaging instrument, especially considering the fact that only the telescope mirrors have been simulated here.Other elements inside the instruments themselves may generate polarization aberrations as well.The stellar residuals will become stronger if these are added to the telescope aberrations; 7.As a part of this paper, we developed ZOS-API and Pythonbased polarization ray tracing routines (Ashcraft 2022) that could be easily used to estimate the polarization aberrations of any optical system; 8.The primary goal of direct imaging is to search for biomarkers with the GSMTs.One of the strongest bio-markers is the oxygen A band at 730 nm (between R and I bands), which requires a raw contrast of 10 −5 to 10 −4 (Snellen et al. 2015;Lovis et al. 2022).Unfortunately, the polarization aberrations arising from the telescope optics already introduce residuals at the required contrast levels.Additional aberrations from the instrument optics will only add to the current estimates.The polarization aberrations will need to be included in the coronagraph design of future HCI instruments.
A121, page 12 of 17 We plan to expand our simulations to include coating nonuniformity, coating aging, segment errors (including missing segments), realistic coronagraphs, and post-AO wavefront errors, including tip, tilt, and low-order aberrations and to evaluate the polarization aberration structure and its statistical nature in the final PSF and post-processing residuals in the following paper.

Appendix C: Refractive indices of coatings used in our simulations
Table C.1 provides the list of refractive indices for different coatings used in our simulations for all the astronomical bands.

Fig. 3 .
Fig. 3. Variation of incident angles (°) for different mirrors in the three telescopes.Mirror M5 is shown for the ELT, which directs the light to the different instruments on the Nasmyth platform (analogous to M3 in TMT and GMT).px and py corresponds to the normalized entrance pupil coordinates.
3 radians over the wavelengths.The difference between the amplitude (|R p | − |R s |) and phase (|ϕ p | − |ϕ s |) reduces from blue to red wavelengths, and predicts similar behavior in the polarization aberrations.

Fig. 5 .
Fig. 5. Jones pupil maps for the GSMTs shown for V-band filter estimated at the telescope's exit pupil.Axx and Ayy show the amplitudes for X and Y polarized light, respectively, and Axy and Ayx are the cross-coupled components.ϕxx and ϕyy show the phase in radians for X and Y polarized light and ϕxy and ϕyx are the cross-coupled components.

Fig. 6 .
Fig. 6.Comparison of the total diattenuation (left) and retardance (right)of the E-ELT (top), TMT (middle), and GMT (bottom) in the V band for X-polarization.These data reveal the tilted astigmatic pattern characteristic of the polarization aberrations expected for Cassegrain telescopes with fold mirrors.Y-polarized light experiences a similar pattern rotated by 90 degrees.In addition, these data reveal that the protected silver mirrors experience greater retardance and lower diattenuation than the bare-aluminum GMT.

Fig. 7 .
Fig. 7. Total diattenuation and retardance for each astronomical band.Each solid line represents the mean value of the total diattenuation and retardance.The shaded areas around these lines represent the maximum and minimum absolute values of total diattenuation and retardance.The total diattenuation and retardance decrease with wavelength, as expected, except for the GMT diattenuation due to aluminum's high visible and near-infrared absorption.

FigFig. 8 .
Fig. 8. Stellar residuals for different wavelengths and coronagraphs for the TMT.The residuals are shown for I band (top), H band (middle), and L band (bottom).The coronagraphs are a second-(left), fourth-(middle), and sixth-order (right) coronagraph.These images show that the stellar residuals is mainly defocus.

Fig. 10 .
Fig.10.Peak contrast as a function of wavelength for the GMT with different coatings.The different colors correspond to different mirror coatings.The solid lines show the contrast for a fourth-order coronagraph, and the dashed lines show the for a sixth-order.The nominal aluminum coating for the GMT has the best performance.The difference in performance is even larger for the sixth-order coronagraph.
The coronagraphic residuals due to polarization aberrations for theELT and GMT are shown in Figures B.1 and B.2.

Fig. B. 1 .Fig
Fig. B.1.Stellar residuals for different wavelengths and coronagraphs for the ELT.The residuals are shown for I band (top), H band (middle), and L band (bottom).The coronagraphs are second-(left), fourth-(middle), and sixth-order (right) coronagraphs.These images show that the stellar residuals are mainly defocus.

Table C
.1.Refractive indices used in our calculations A121, page 17 of 17