Planet search with the Keck/NIRC2 vortex coronagraph in the Ms band for Vega

Context. Gaps in circumstellar disks can signal the existence of planetary perturbers, making such systems preferred targets for direct imaging observations of exoplanets. Aims. Being one of the brightest and closest stars to the Sun, the photometric standard star Vega hosts a two-belt debris disk structure. Together with the fact that its planetary system is being viewed nearly face-on, Vega has been one of the prime targets for planet imaging efforts. Methods. Using the vector vortex coronagraph on Keck/NIRC2 in the M s band at 4 . 67 µ m, we report the planet detection limits from 1 au to 22 au for Vega with an on-target time of 1 . 8 h. Results. We reach a 3 M Jupiter limit outward of 12 au, which is nearly an order of magnitude deeper than for other existing studies. Combining our observations with existing radial velocity studies, we can confidently rule out the existence of companions more than ∼ 8 M Jupiter from 22 au down to 0 . 1 au for Vega. Interior and exterior to ∼ 4 au, this combined approach reaches planet detection limits down to ∼ 2–3 M Jupiter using radial velocity and direct imaging, respectively. Conclusions. By reaching multi-Jupiter mass detection limits, our results are expected to be complemented by the planet imaging of Vega in the upcoming observations using the James Webb Space Telescope to obtain a more holistic understanding of the planetary system configuration around Vega.


Introduction
Vega (α Lyrae, Keoe, wāqi', Zhinü 织 女 一), one of the historical photometric standard stars (Johnson & Morgan 1953), is an A0 V star (e.g., Johnson & Morgan 1953) that is located at 7.68 ± 0.02 pc from the Solar System (van Leeuwen 2007).Despite its proximity, Vega is not included in the Gaia Catalog of Nearby Stars that are within 100 pc from the Sun, since its brightness exceeds the Gaia limits (Gaia Collaboration 2021).With an age of 445 ± 13 Myr (Yoon et al. 2010), Vega shows a prototypical mid-infrared excess in the Infrared Astronomical ⋆ Contrast curve is only available at the CDS via anonymous ftp to cdsarc.cds.unistra.fr(130.79.128.5) or via https:// cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/670/A162 Satellite observations, situating it in an evolutionary stage that is between star formation and our Solar System (Aumann et al. 1984).
The two-belt debris disk system around Vega may result from planet-disk interactions under various planetary configurations (e.g., Matrà et al. 2020).In fact, most existing directly imaged planets are found in bright debris disk systems (e.g., HR 8799: Marois et al. 2008, β Pic: Lagrange et al. 2009, 51 Eri: Macintosh et al. 2015), making it more likely to find giant planets in debris disk systems than around stars without disks (Meshkat et al. 2017).Combined with the observational fact that giant planets are more likely to exist at 1-10 au from their host stars in both direct imaging and radial velocity surveys (Nielsen et al. 2019;Fulton et al. 2021) of Vega makes it one of the best systems for giant planet searches.
Before the resolved imaging of debris belts around Vega, dust structures around Vega have suggested the existence of potential planetary perturbers (e.g., Holland et al. 1998;Wilner et al. 2002) and called for the deep imaging of them.Combining the Gemini Altar adaptive optics (AO) system and the Gemini Near-infrared Imager (NIRI), Marois et al. (2006) obtained 5σ detection limits of better than 18 Mag at 3 ′′ -10 ′′ in the offmethane 1.58 µm 6.5% filter, or ∼3 M Jupiter at 8 ′′ .Using the MMT Observatory AO and the Clio camera, Hinz et al. (2006) and Heinze et al. (2008) obtained 10σ limits of better than ∼12 Mag at 2 ′′ -11 ′′ at the M band, or ∼10 M Jupiter .
Recent observational studies on the Vega planetary system are in direct imaging, transiting, radial velocity, and (sub)millimeter interferometric imaging.In direct imaging, Meshkat et al. (2018) presented observations from the coronagraphic integral field spectrograph P1640 at the Palomar Observatory in J and H bands.Despite a nondetection of planets, they obtained 5σ planet detection limits from 0. ′′ 25 to 2 ′′ , reaching a best sensitivity of ≈20 M Jupiter at 1. ′′ 5 or 12 au.From a complimentary approach, using the Atacama Large Millimeter/submillimeter Array (ALMA), Matrà et al. (2020) observed and modeled the outer belt from ∼60 au to ∼200 au.To explain the observed disk architecture, Matrà et al. (2020) discussed three mechanisms which include a single giant planet, multiple low-mass planets, and no outer planets.Combining a decade of Tillinghast Refector Echelle Spectrograph (TRES) spectra for radial velocity and two sectors of Transiting Exoplanet Survey Satellite (TESS) photometry, and under the scenario that the planetary orbits have inclinations between 1. • 5 and 11. • 5, Hurt et al. (2021) obtained a nondetection of 1-10 M Jupiter planets within 1-10 au, while reporting a candidate Jovian signal with a period of 2.43 days in radial velocity measurements.Nevertheless, depending on the orbit orientations of the planets, the mass limit can vary by up to a factor of ∼10 in Fig. 6 of Hurt et al. (2021).
To improve existing high-contrast imaging limits, test different mechanisms for the formation of the Vega debris disk system observed in ALMA, and explore beyond the nearly edge-on limitations from radial velocity and transit studies, we observed Vega in the M s band using the vortex coronagraph on Keck/NIRC2.In the M s band, planets have relatively larger planet-to-star brightness ratios than in the J or H band (e.g., Spiegel & Burrows 2012;Skemer et al. 2014), and the advantage of the M s band over shorter-wavelength bands improves the intermediate system ages for a planet of a given mass (e.g., Currie et al. 2022), both enabling us to explore around Vega for possible cooler and less massive planets than existing studies.
Faint planetary signals are often overwhelmed by the stellar point spread function (PSF) of the central star.In order to remove the stellar PSF and reveal faint surrounding signals, we first preprocessed the data using the VIP package (Gomez Gonzalez et al. 2017), which was further customized for NIRC2 vortex observations by performing flat-fielding, bad pixel and background removal, and image centering (Xuan et al. 2018).We then reduced the preprocessed data using the Karhunen-Loève image projection (KLIP) algorithm (Soummer et al. 2012;Amara & Quanz 2012), which performs principal component analysis to capture the stellar PSF and speckles in the observation.For an image, we removed its stellar PSF and speckles by first projecting it to the KLIP components, and then subtracting the projection from the original image to obtain the residuals.Astrophysical signals including planets and disks thus reside in these residuals.To obtain the final image for each observation, we obtained the residual maps for each individual readout, then rotated and median combined them.We present the combined two-epoch result using 10% of the KLIP components with angular differential imaging (ADI: Marois et al. 2006) in Fig. 1 for demonstration purposes, in which we did not identify point sources that are beyond 5σ of the noise that is within the same angular separation from the stars.
Using VIP and taking into account self-subtraction and overfitting with ADI while using KLIP, we generated the 5σ contrast limits for each observation following Xuan et al. (2018) by varying the reduction parameters to obtain the corresponding detection limits while performing corrections for small sample statistics in Mawet et al. (2014).Specifically, VIP performs injection recovery for companions at different locations to measure the throughput from ADI and KLIP (Gomez Gonzalez et al. 2017).While measuring our throughput, we injected companions along three radial branches spread throughout the image (originating from the masked center) where the averaged throughput at each radial location in the image was determined from these multiple estimates of the throughput at different branches (Xuan et al. 2018).We computed the contrast for the entire image for each combination of inner and outer mask size and number of principal components, where we computed up to 30 principal components.The ADI reduction was performed with no rotation gap; for one combination of inner and outer radii from Table 1 of Xuan et al. (2018), the full-frame reduction was performed on the annulus zone with the region inward of the outer radius or outward of the inner radius included.For example, for an outer radius of 0. ′′ 5 and an inner radius of 0. ′′ 08, all pixels with radial separations between 8 pixels and 50 pixels from the center of the image, or between 0. ′′ 08 and 0. ′′ 5, were included in the reduction (see the reduction details in Xuan et al. 2018).While we did not use annular ADI, we did utilize the best contrast achieved from our full-frame ADI at each one-pixel annulus.
To obtain the final detection limit for each angular separation from the star with a step size of 1 pixel, we compared the detection limits from different combinations of reduction parameters.The reduction parameters include the frame size (i.e., algorithmic inner and outer radii) and the number of principal components (see Sect. 2 and Table 1 of Xuan et al. (2018) for details on computing most optimal contrasts using five different frame size combinations).Therefore, while our optimal contrast is the combination of frames processed using different parameters, we were optimizing at each radial location.For our observations, the number of principal components adopted for the final contrast curve ranges from 8 to 29, with a median of 15 and a standard deviation of 7.4.
The faintest companion that can be detected from such combinations at 5σ level was adopted as our final detection limit.For the 2019 data, the exposures that were used to image the unblocked central source were saturated, we thus fit the unsaturated first Airy ring to that of a theoretical model of the vortex stellar PSF (while taking into account the PSF broadening effects due to weather by convolving a 2D Gaussian distribution), and used the best-fit model to generate the corresponding contrast curve.We then combined our detection limits from the two observations, and present them in Fig. 2.Although the 2018 observation has a shorter total on-target exposure time than the 2019 one, their total parallactic angle change difference is only 6. • 7.With the Differential Image Motion Monitor (DIMM) seeing of 0. ′′ 41 ± 0. ′′ 11 in 2018 being more stable than that of 1. ′′ 18 ± 0. ′′ 55 in 2019, the 2018 data dominate the detection limits in the combined dataset.With an age of 445 ± 13 Myr (Yoon et al. 2010) and adopting the AMES-Cond evolutionary models (Baraffe et al. 2003), we converted the contrast to 5σ mass detection limits in Fig. 2. We reached a detection limit of less than 5 M Jupiter beyond 9 au, and 3 M Jupiter beyond 14 au.

NIRC2 imaging and TRES radial velocity
Combining the 2018 NIRC2 results with the radial velocity data from TRES in Hurt et al. (2021), we followed Hurt et al. (2021) to obtain the mass limits assuming the planetary orbits are well aligned with the spin axis of Vega.To explore the detectability of companions from both direct imaging and radial velocity measurements, we randomly generated 10 6 radial velocity samples of companion orbits following Hurt et al. (2021).The semi-major axis follows a log-uniform distribution ranging from 0.1 au to 22 au.The companion mass follows a log-uniform distribution ranging from 0.1 to 100 M Jupiter .The sine value of orbital inclination follows a uniform distribution from 1. • 5 to 11. • 5.The orbital eccentricity follows a beta distribution described in Kipping (2013).The argument of periastron follows a uniform distribution ranging from 0 to 2π radian.The time of periastron passage follows a uniform distribution which was determined by the orbital period.The stellar mass follows a Gaussian distribution using the measurements from Monnier et al. (2012).In each radial velocity sample, we scaled Gaussian noise according to the uncertainties of the TRES measurements in Hurt et al. (2021).We fit a flat line to each synthetic radial velocity curve using RadVal (Fulton et al. 2018).
For all simulated radial velocity samples, we considered a synthetic signal to be detectable in radial velocity when its pvalue was < 0.001 (i.e., 3.3σ), while ignoring correlated noise (Hurt et al. 2021).At a specific radial separation from the star, we further required that a companion be detectable when its mass is above the detection limit in Fig. 2.
We present the detection probability of companions from the above injection-recovery procedure in Fig. 3.The detectability of companions from combined radial velocity and direct imaging follows two trends as a function of stellocentric separation.Interior to ∼4 au, the detectable planets increases with radial separation, approaching planets with ∼8 M Jupiter down to ∼1 M Jupiter at ∼0.1 au.Exterior to ∼4 au, the detectability is dominated by NIRC2 imaging, reaching down to ∼2 M Jupiter at ∼22 au.Under the Hurt et al. ( 2021) framework, there is a possible nonabsolute detection of companions near ∼22 au, which is limited by the NIRC2 field of view in our study, since the sampled orbital eccentricity adopted from Kipping (2013) can position planets with a semi-major axis less than ∼22 au beyond the 22 au angular radius.2021) (see Sect. 3.1.2).We can reach an ∼8 M Jupiter detection limit at ∼4 au, while probing down to ∼2-∼3 M Jupiter at 0.1 au and 22 au, respectively.

Disk formation from mass limits
The ALMA observation in Matrà et al. (2020) resolves the outer dust belt of Vega extending from ∼60 au beyond ∼150 au.To explain the observed planetary system architecture, the authors have analyzed the two scenarios that involve planets: either a chain of small planets within 70 au with mass ≳6 M ⊕ , or a sole ∼5 M Jupiter at 50-60 au.
The combined Keck/NIRC2 M s -band and TRES result can exclude the existence of ∼8 M Jupiter planets from 0.1 au to 22 au.Despite the fact that with an age of ∼400 Myr there is no clear brightness difference between the hot-start and cold-start models (e.g., Spiegel & Burrows 2012), the observations presented here cannot rule out the sole giant planet which resides at 50-60 au in Matrà et al. (2020) for Vega.
For future exploration of the proposed sole giant planet at 50 au to 60 au using Keck/NIRC2 in the M s band, a field of view that is between 6. ′′ 5 and 7. ′′ 8 is needed.With a pixel scale of 9.942 mas, this corresponds to a half width between 654 pixels and 785 pixels, which exceeds the current 512 pixel half width of the narrow camera for NIRC2 and such an observation is thus not achievable.Alternatively, NIRC2 offers sampling setups that have pixel sizes of 20 mas or 40 mas, yet these setups are not feasible for M s -band imaging due to the corresponding increased sky background.Nevertheless, the planned upgrades of the NIRC2 detector electronics may permit faster readouts to enable M s -band imaging for such purposes.
To explore far separation imaging for Vega using the existing confirguration of NIRC2, either offsetting the vortex center away from the center (e.g., the observation of HR 8799 b using the Gemini Planet Imager in Wang et al. 2018) of the NIRC2 narrow camera, or executing observations without the vortex coronagraph, could enable a half width of 785 pixels to test the sole giant planet scenario.With this 1024 × 1024 pixel field of view, the minimum permitted readout time is 0.18 s 1 , which could enable a nearly identical exposure sequence in the 2018 dataset presented here.Given that extra parallactic angle change is needed to cover the entire field of view for these setups, they are beyond the scope of this study.

Implications for JWST observations
At an age of 445 ± 13 Myr for Vega (Yoon et al. 2010), giant planets with several Jupiter masses do not have clear brightness distinction between different formation models in the M band (e.g., Fig. 7 of Spiegel & Burrows 2012).For planets with less than ∼5 M Jupiter , their brightness is expected to peak at 4 µm to 6 µm (e.g., Fig. 6 of Spiegel & Burrows 2012).
To image planets that are brightest at these wavelengths, we applied the Keck/NIRC2 M s band which operates at a central wavelength of 4.67 µm with a bandpass of 0.24 µm2 .In comparison, the NIRCam instrument onboard the James Webb Space Telescope (JWST) can cover nearly half the expected brightest wavelengths with its F444W filter (4 µm to 5 µm)3 .By reaching a multi-Jupiter mass limit for 10 au to 20 au within this study, and one Jupiter mass or better beyond 20 au with NIRCam (Meshkat et al. 2018, Fig. 5 therein), we can combine Keck/NIRC2 and JWST/NIRCam to reach the deepest planet detection limit to investigate the planetary architecture for the Vega system.
Using the on-sky JWST/NIRCam F444W contrast curve from Carter et al. (2022) in the JWST ERS-1386 program (Hinkley et al. 2022), we calculated the expected on-sky contrast for Vega with NIRCam F440W in Fig. 2 for JWST GTO-1193 observations as follows.Assuming an identical instrument performance, which is an optimistic estimation given that JWST GTO-1193 uses MASK430R since it has a larger inner working angle -and thus lower throughput -than MASK335R in the JWST ERS-1386 observations in Carter et al. (2022), we scaled the exposure times with the SUB320 subarray (i.e., Observations 35 and 36) from the JWST GTO-1193 observations and we recalculated the contrast based on the exposure time difference.We then converted the JWST/NIRCam F444W contrast to point source mass following Carter et al. (2021) for Vega while adopting the same age and apparent magnitude as for NIRC2.
In terms of reaching nominal contrast, the Keck/NIRC2 M Sband observations perform better than NIRCam F440W within 1 ′′ and reach a similar quality in exterior regions.In terms of mass detection limits, the Keck/NIRC2 M S -band observations perform systematically better than the NIRCam F444W under the abovementioned optimistic assumptions.In addition, the degradation of contrast close to the transmission near the edges of the coronagraphic masks4 from MASK335R in JWST ERS-1386 to MASK430R in JWST GTO-1193 has been ignored.All three aspects demonstrate that the NIRC2 M S -band observations presented here establish the deepest high-contrast imaging exploration of planetary companions for Vega in the probed regions.Nevertheless, we emphasize that NIRCam should be better than NIRC2 for most other systems, since the study presented here should be the best-case scenario for ground-based M S -band imaging due to the brightness of Vega; for other targets that are fainter than Vega, they should have less favorable contrasts due to the relative background levels on the ground from NIRC2.

Planet detection toward 0.1 arcsec
To obtain a more general understanding of planetary existence, the detection of intermediate separation planets near ∼0.′′ 1 are expected to yield the most thorough discovery of planets (e.g., Nielsen et al. 2019;Fulton et al. 2021).In fact, for the detection of far-separation planets, existing direct imaging surveys equipped with extreme AO systems have experienced a degradation of contrast close to the central sources (e.g., Nielsen et al. 2019;Vigan et al. 2021;Xie et al. 2022).In comparison, for the detection of close-in planets, existing radial velocity surveys have less completeness for long orbital period planets (e.g., Wittenmyer et al. 2020;Fulton et al. 2021).As a result, in the detection probability map of companions for Vega in Fig. 3, there is a clear gap for middling separations.
To fill the gap, the concept of the vortex fiber nuller (VFN; Ruane et al. 2019;Echeverri et al. 2020) is designed to detect and characterize planets near and within 0. ′′ 1.Using nulling interferometry in the near-infrared, VFN suppresses on-axis starlight while retaining off-axis companion light with acceptable loss, which increases the signal-to-noise ratio for companions and thus better enables their detection and characterization.The VFN has already been installed for the phase 2 development of the Keck Planet Imager and Characterizer (KPIC; Jovanovic et al. 2020, Echeverri et al., in prep.) on the Keck Observatory, providing access to planets between 30 and 90 mas in the K band.Furthermore, the limitation of VFN in localizing the companions in their orbit will be further resolved with the concept of photonic lantern nuller (PLN; e.g., Xin et al. 2022).We expect that the application of VFN and PLN will fill the planet detection and characterization gap between direct imaging and radial velocity.For Vega, they will better explore possible hidden planets toward ∼0.′′ 1 in Fig. 3. Nevertheless, the limitation of VFN in the K band is that it is less sensitive to evolved and cooler giant planets due to the drop in planet brightness in shorter wavelengths.

Summary
We report 1.8 h of M s -band imaging observations of Vega using the Keck/NIRC2 vortex coronagraph.Despite a nondetection of companions, we have pushed the mass detection limits from existing high-contrast imaging observations in Meshkat et al. (2018) by nearly an order of magnitude smaller (see Fig. 2).Combining the NIRC2 results with an existing radial velocity study using TRES in Hurt et al. (2021), we can confidently rule out companions more massive than ∼8 M Jupiter from 0.1 au to 22 au for Vega.Within this range, NIRC2 can reach planets that are less massive than 5 M Jupiter beyond 9 au.
While the NIRC2 observations presented here are sensitive to planets of 5 M Jupiter at 9 au down to 3 M Jupiter at 22 au, it is limited by the field of view and thus cannot be used to test the scenario in Matrà et al. (2020) that one multi-Jupiter mass planet at 50 au to 60 au is responsible for the planetary system architecture.Alternatively, observing without using the vortex coronagraph while performing multipoint dithering5 , combined with extra parallactic angle change, may provide enough sensitivity to image such a perturber for the Vega system by pushing deeper than Heinze et al. (2008) for these separations.
Combining Keck/NIRC2 in the M s band and NIRCam in its F444W filter assuming identical performance with JWST from ERS-1386 observations (Hinkley et al. 2022;Carter et al. 2022), we can reach comparable detection limits of companions interior to 20 au.Interior to ∼10 au, although companion imaging limits increase from ∼5 M Jupiter at 10 au to ∼100 M Jupiter at 2 au in Fig. 2, this region is where radial velocity measurements can A&A 670, A162 (2023) provide multi-Jupiter detection limits or better as shown in Hurt et al. (2021).Future works following Mawet et al. (2019) and Llop-Sayson et al. (2021) in combining measurements including direct imaging here and radial velocity in Hurt et al. (2021), as well as upcoming imaging observations with JWST GTO-1193, in addition to the VFN and PLN concepts to detect planets toward ∼0.′′ 1, could allow us to obtain the most holistic understanding for the planetary system of this historical photometric standard star -Vega.
, the proximity A162, page 1 of 6 Open Access article, published by EDP Sciences, under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0),which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.This article is published in open access under the Subscribe to Open model.Subscribe to A&A to support open access publication.
Fig.1.Combined two-epoch NIRC2 image of Vega the M s band using ADI with 10% of the KLIP components for demonstration purposes.We did not identify point sources that were more than 5σ levels beyond the noise of a similar angular separation from the star.The pixel values correspond to lower limits of contrast values due to over-and self-subtraction with KLIP and ADI, respectively.
Being the photometric standard, Vega's apparent magnitude is defined as 0 in the M s band.Adopting a distance of 7.68 ± 0.02 pc in van Leeuwen (2007), the absolute magnitude for Vega in the M s band was 0.573 ± 0.006.

Fig. 2 .
Fig. 2. Detection limits of point sources around Vega in the M s band.Left: 5σ contrast, in comparison with the JWST/NIRCam F444W on-sky contrast calculated from Carter et al. (2022).Right: 5σ mass detection limit, and expected mass detection limit with NIRCam F444W.We note that the NIRCam F444W values have an assumed identical instrument performance in the JWST/ERS-1386 program (see Sect. 3.3).

Fig. 3 .
Fig. 3. Detection probability of point sources as a function of the semimajor axis of point sources for Vega, using a combination of the 2018 NIRC2 observation and the TRES radial velocity measurements from Hurt et al. (2021) (seeSect.3.1.2).We can reach an ∼8 M Jupiter detection limit at ∼4 au, while probing down to ∼2-∼3 M Jupiter at 0.1 au and 22 au, respectively.