© ESO 2022

1. Introduction

In the hierarchical theory of structure formation, initial small density fluctuations give rise to the formation of the first stars and galaxies. These structures subsequently grow larger and more massive via mergers and accretion (e.g., White & Rees 1978). This produces denser and denser regions with a strong gravitational potential that can influence the distribution and the evolution of galaxies.

The densest regions at high redshift can be pinpointed by searching for massive dusty galaxies, such as submillimeter galaxies (SMGs). Submillimeter galaxies (Blain et al. 2002; Casey et al. 2014) are characterized by strong star-formation events, typically associated with large amounts of gas and dust which often obscures rest-frame ultraviolet to optical wavelengths (Dudzevičiūtė et al. 2020). They typically present rapid gas consumption through high star-formation efficiencies that are associated with major mergers. Overdense regions around SMGs are interesting because SMGs are thought to be the progenitors of the most massive galaxies in the local Universe (Chapman et al. 2005; Swinbank et al. 2008; Stach et al. 2021).

By studying the distribution of the neutral gas around SMGs and within their overdense environment, we can investigate accretion events and obtain information on their evolution. An efficient method to reveal the distribution and study the kinematics of the neutral hydrogen (HI) gas at high redshift is the detection of Lyα emission (e.g., Cantalupo et al. 2014; Matthee et al. 2020a; Daddi et al. 2021)

It is important to note that Lyα is the strongest recombination line of neutral Hydrogen. It is mainly produced in star-forming regions where the ionizing radiation emitted by young, hot O and B stars ionize their surrounding gas which recombines in relatively short timescales, depending on its density (> 1 atoms cm⁻³) and temperature (10⁴ K) conditions (Dijkstra 2016; Cantalupo 2017). Due to their short wavelength, Lyα photons are easily absorbed by dust and, due to the resonant nature of the Lyα transition, they are scattered by HI atoms. Also, the escape of Lyα photons out of the dense interstellar medium (ISM) gas can be favored by HI kinematics, which can produce an asymmetric profile where the red peak is the dominant one in case of an outflowing gas (Verhamme et al. 2006; Schaerer & Verhamme 2008; Laursen et al. 2009; Orsi et al. 2008; Gurung-López et al. 2019a). These phenomena produce an emission line with a red and a blue peak and some absorption at the resonant wavelength of Lyα. The relative intensity and the separation between the two peaks is related to the amount and distribution of dust and HI atoms, and the Lyα emission is typically spatially more extended than the UV continuum (e.g., Steidel et al. 2011; Wisotzki et al. 2016; Leclercq et al. 2017). Another process that can produce Lyα photons and that has been proposed to explain the very extended Lyα nebulae observed at high redshift is collisional excitation (Haiman et al. 2000; Fardal et al. 2001) with electrons. This process converts the thermal energy of the gas into radiation, and therefore cools the gas. It is typically less energetic than recombination, except in the case of a strong gravitational potential.

Together with Lyα, also [C II]158 μm has been detected in galaxies at z > 4 and used to investigate the distribution and the properties of their gas (e.g., Capak et al. 2015; Maiolino et al. 2015; Pentericci et al. 2016; Carniani et al. 2018). [C II]158 μm is the dominant cooling line of the ISM in star-forming galaxies, where it can carry up to 1% of the far-infrared luminosity (e.g., Israel et al. 1996). It can originate from different phases of the ISM (Hollenbach & Tielens 1999; Stacey et al. 1991; Goldsmith et al. 2012; Pineda et al. 2013); mainly in the photo-dissociation regions where the UV radiation from hot stars can dissociate molecules and ionize atoms (Stacey et al. 1991), also in the cold neutral atomic medium, but sometimes either in the warm or in the ionized medium (e.g., Pineda et al. 2013; Maiolino et al. 2015). The association with the molecular CO lines can help in understanding its origin from molecular clouds. Therefore, [C II]158 μm is an ideal tracer of star formation, but also of the distribution, dynamics, and enrichment of the ISM in star-forming galaxies and also of their circum-galactic medium (CGM).

We expect that the HI gas in dense environments affect the stage of evolution of the galaxies inside the dense regions and the shape of the emission of Lyα and possibly [C II]158 μm. For instance, the observations of protoclusters around radio galaxies had shown an inside-out picture in which dusty starburst galaxies are located in the cores and young Lyα emitters are distributed in the outskirts (Kuiper et al. 2011). However, this picture is not so straightforward because there are also dense regions at high redshift composed of more than one dense clump (e.g., Cortese et al. 2006; Zheng et al. 2016; Guaita et al. 2020) and also dense clumps could be traced by different galaxy populations. Shi et al. (2019), for example, showed an overdensity of Lyα emitters separated from a peak traced by more massive Lyman Break galaxies, maybe indicating the presence of a stream of gas falling into one side of the structure. The SSA22 protocluster (Steidel et al. 2000) at z ∼ 3 is an example in which the overdensity is traced by both Lyα emitters and Lyman Break galaxies. The filamentary structure traced by the Lyα emitting sources shows the effect of the gravitational potential that seems to result in the formation of giant Lyα nebulae in the intersection of the filaments.

Furthermore, in dense regions, the major, gas-rich merger rate can be higher than in the field, given the possible galaxy encounters (e.g., Hine et al. 2016; Tacconi et al. 2013). In a merging process, the gas can be compressed either in the central and in the external regions of the progenitor galaxies, favoring strong episodes of star formation (e.g., Hernquist 1989). The properties of the merging gas and of the star-formation phenomenon could shape the Lyα emission of the merging system. Yajima et al. (2013) studied the formation of extended Lyα emission from interacting galaxies at high redshift using a combination of hydrodynamic simulations with three-dimensional radiative transfer calculations. They showed that at the first passage the triggered star-formation event produces two Lyα peaks coinciding with the two progenitor nuclei. The Lyα peaks decrease in intensity after the first passage and increase again in the second passage. The Lyα emission continues to decline and its distribution becomes more compact in the final stage of the merger. Recently, Romano et al. (2021) studied 75 main-sequence star-forming galaxies at 4.4 < z < 5.9 through their [C II]158 μm emission and reported about the presence of significant merger activity at z ∼ 5.

In this paper, we study the environment surrounding the submillimeter galaxy, AzTEC-3, at z = 5.3 (Capak et al. 2011). AzTEC-3 was discovered (Younger et al. 2007) in the AzTEC survey (Austermann et al. 2009) and it was the first SMG found at z > 5 with a large overdensity associated with it. Capak et al. (2011) discovered at least ten star-forming galaxies in a circular area of 2 comoving Mpc (cMpc) around AzTEC-3 and estimated that the SMG makes at least one-fourth of the mass of the entire system. Riechers et al. (2010) detected CO molecular gas emission from a compact (< 2.3 kpc scale) region on top of the SMG, estimating a CO mass on the order of 5 × 10¹⁰ M_⊙ that could be depleted in 50 Myr at a star-formation rate (SFR) of 1100 M_⊙ yr⁻¹. The detection of luminous CO emission implies enrichment with heavy elements in the material that fuels the star formation in AzTEC-3. The compactness of the CO-emitting region and the consequent high star-formation rate density make AzTEC-3 a special object to study, probably affected by its special environment (see also Riechers et al. 2020). In fact, a Lyman Break galaxy (LBG-3, nomenclature from Riechers et al. 2014) is observed at 2″ (about 75 comoving kpc = ckpc) and another one (LBG-1, nomenclature from Riechers et al. 2014) is observed at 15″ (about 580 comoving kpc) from AzTEC-3. LBG-1 is composed of three main clumps as seen in the rest-frame UV images from the Hubble Space Telescope (HST) and it makes a significant fraction of the mass of the protocluster (about half of the mass of AzTEC-3). From the spectral energy distribution point of view, LBG-1 can be considered as a typical star-forming galaxy at z ∼ 5, with little dust obscuration and a young starburst in addition to an old underlying stellar population (Capak et al. 2011). No CO emission was detected in LBG-1 (Riechers et al. 2014; Pavesi et al. 2019) and this is consistent with its low metallicity. Riechers et al. (2014) used the Atacama Large Millimeter/submillimeter Array (ALMA) and detected [C II]158 μm emission from the same compact region of AzTEC-3 as CO. Also, they observed a hint of a tidal feature toward LBG-3. [C II]158 μm emission was also detected at the position of LBG-1 with a hint of velocity gradient, which could be related to a merging event within the three clumps.

Keeping the previous observations in mind, we study here deep VLT/MUSE (Multi Unit Spectroscopic Explorer) observations of the region around AzTEC-3 and LBG-1. The MUSE data cover the 800−1500 Å rest-frame wavelengths at z ≃ 5.3, including Lyα. We take advantage of the synergy between the previous [C II] information and the MUSE Lyα detections to study the properties of the galaxies in the protocluster and to investigate whether the environment could play a role in the stage of evolution of AzTEC-3. It is worth noting that many z > 5 SMGs are not detected at UV wavelengths in deep HST data (e.g., HDF850.1 and GN10 as explained in Riechers et al. 2020), so AzTEC-3 is a relatively rare case where MUSE UV studies are possible.

The paper is organized as follows. In Sects. 2 and 2.1, we explain the MUSE observation and the reduction of the data. In Sect. 2.2, we describe the method adopted to analyze the MUSE datacube and to detect Lyα emitting sources. In Sect. 3, we present the sample of Lyα emitting galaxies and star-forming galaxies in the field. In Sects. 4.1 and 4.2, we discuss in detail the spectroscopic properties of the Lyα emission around AzTEC-3 and LBG-1, the two galaxies with known systemic redshift from previous [C II] observations. In Sect. 5, we present the radiative transfer model used to fit the Lyα emission profiles. In Sect. 6, we provide a discussion about the AzTEC-3 environment coming from our MUSE Lyα, the CO and [C II] observations. In Sect. 7, we summarize our work. Throughout the paper, we adopt a standard cosmology (H₀ = 70 km s⁻¹ Mpc⁻¹, Ω₀ = 0.3).

2. Observations

The field around the AzTEC-3 (RA = 150.0863, Dec = 2.589) submillimeter galaxy was selected to match the central region of the CO Luminosity Density at High-z (COLDz) survey (Pavesi et al. 2018a; Riechers et al. 2019), which corresponds to the deepest field with CO(2−1) coverage at z ≃ 5.3. It also matches the COSMOS-XS survey field (van der Vlugt et al. 2021; Algera et al. 2020) which has the deepest radio continuum data in the COSMOS field (see also Algera et al. 2021).

Data from the HST are available in the F606W, F814W, F105W, F125W, and F160W filters. The F814W image was obtained in Cycle 12 and 13 (Scoville et al. 2007), the F606W, F125W, and F160W images were obtained in 2014 (Riechers 2013), and the F105W image was obtained in Cycle 22 (PI: P.L. Capak) (see also Barisic et al. 2017). The environment around AzTEC-3 is quite rich. As we can see in Fig. 1, within a radius of 2″ there are at least three sources at similar redshift. At 15″ from AzTEC-3, there is also LBG-1 (Riechers et al. 2014).

Fig. 1. Left panel: mosaic of the entire area observed with MUSE, obtained collapsing the cube information in the 7600−7700 Å wavelength range (“mosaic”), which corresponds to the redshifted Lyα at z ≃ 5.3. We identified six regions characterized by similar background level, the center of the four pointings (P1, P2, P3, P4), the central vertical and horizontal (black rectangles) stripes of overlapping regions. The white square corresponds to the area zoomed in the right panel. Right panel: zoom of the region containing AzTEC-3 (upper red circle) and LBG-1 (lower red circle) in the HST F160W image. We show the position of LBG-3 as black square, the position of LBG-2 (1447523 in Capak et al. 2011 and COSMOS2015_849887 in Laigle et al. 2016) as blue circle, and the position of COSMOS2015_848724 as black circle. These sources are all at z ∼ 5.3. The yellow circle indicates the position of a knot of star formation which can also be part of AzTEC-3. It is worth noting that LBG-1 is located in the center of the mosaic, in the lower corner of the 20 h overlapping region.

During December 2014 and February 2015, we observed the AzTEC-3 field with the Multi Unit Spectroscopic Explorer (MUSE, Bacon et al. 2010, 2014, 2015). MUSE is a second generation instrument installed on the Nasmyth focus of UT4 at the Very Large Telescope (Chile). The spectral resolution of MUSE at 7000 Å is about 2700 and each resolution element is sampled by 2.5 pixels along the spectral direction. The spectral sampling is 1.25 Å per pixel.

The pixel scale of MUSE is 0.2″ per pixel (about 8 ckpc per pixel) and the point spread function (PSF) of our MUSE data is 0.7″, corresponding to about 4 kpc or 27 ckpc at z = 5.3. The field was observed in four MUSE pointings, two of which contain AzTEC-3 at the edges, and the total area is 1.4 × 1.4 arcmin².

Each pointing was observed with 20 frames of 900 s, making a total exposure time of 20 h in the very center of the overlapping regions. For every observation night, we were able to use bias, dark, dome flat, sky flat frames from our run to perform the basic reduction steps, wavelength-calibration files, standard stars, and illumination-correction files close to each observing night were also used for proper calibration.

2.1. Reduction of the MUSE datacubes

We used the standard ESO MUSE pipeline (Weilbacher et al. 2014, v2.2) to perform the initial reduction of the MUSE datacubes. The pipeline carries out bias, dark and flat field corrections, calibrates the data in wavelength and astrometry, and applies a basic illumination correction. Afterwards, we improved the quality of the illumination correction and sky subtraction by applying the CubExtractor (CubEx) software following Cantalupo et al. (2019) (see also Borisova et al. 2016; Fumagalli et al. 2016, 2017; Mackenzie et al. 2019, for details).

We reduced the MUSE dataset in two ways. First, we reduced the datacubes corresponding to each pointing separately. In the final combinations, we generated a mean and median stack of the four pointings. Then, we produced a mosaic by combining the information contained in all the 80 MUSE OBJECT_PIXTABLEs and by generating individual cubes, all of the size of the final mosaic. Basically, NaN values are inserted in the areas of the individual cubes outside the observation field of view. We calculated the frame offset before combination, by cross-matching the coordinates of the sources identified with the SourceExtractor software (Bertin & Arnouts 1996). We made a mean combination of all the frames (“mosaic”), two mean combinations of the even (“even mosaic”) and odd (“odd mosaic”) numbered subexposures to have independent sets of data to be used as sanity checks of the detections, and a median combination of all the frames.

We performed the two reductions because the mosaic is more difficult to handle in the stage of the analysis due to its size, but it is fundamental to improve the signal-to-noise ration (S/N) of the overlapping areas among the pointings, in particular the region just to the south of AzTEC-3. The difficulty comes from the fact that the background level of the mosaic is different at the center of the pointings and in the overlapping areas and this could affect the measured S/N in the source detection. To overcome this difficulty we divided the mosaic into six regions, the center of each of the four pointings, the vertical overlapping area (vertical stripe), and the horizontal overlapping area (horizontal stripe), as shown in Fig. 1. However, these regions have sharp separations and the reduction of the four pointings separately are used to identify sources otherwise missed due to border effects. As expected, the detection limit increases in the overlapping regions (Table 1). For comparison, the 5σ detection limit of the white light image of the entire area is 26.2 when the central wavelength of the MUSE coverage is taken as the reference wavelength. We matched the astrometry of MUSE to that of the images of the Hubble Space Telescope (HST) and we reached an accuracy better than 0.1″ on average. In Fig. 1, we show the narrow bands obtained collapsing the MUSE datacube in the 7600−7700 Å wavelength range, that contains the observed wavelength of the Lyα emission line at z = 5.3 (λ_air ∼ 7654 Å).

2.2. Detection and identification of the line emitters

The inspection of the combined MUSE datacubes was performed with the CubEx software (Cantalupo et al. 2019). To maximize the S/N of the detections, we first ran a continuum-subtraction procedure, which is performed by the CubeBKGSub function of the CubEx package. The function subtracts the continuum sources in the field, likely to be low-z objects. The continuum subtraction is performed through a median filtering along the spectral dimension, spaxel by spaxel. Following Marino et al. (2018), we chose the size of one cell of continuum in the wavelength direction equal to 50 and the continuum filter radius in the wavelength direction equal to 3.

To avoid the subtraction of emission lines in the wavelength range of our interest, some wavelengths were masked in the CubeBKGSub procedure. For the specific case of the detections of the members of the AzTEC-3 protocluster, we masked the wavelength range corresponding to the Lyα emission line at the redshift of the AzTEC-3 submillimeter galaxy (2324−2336 pixels in the spectral direction, corresponding to [−40, +550] km s⁻¹ from the AzTEC-3 systemic redshift at the Lyα wavelength).

Then, CubEx was used to perform extraction, detection, and photometry of sources with arbitrary spatial and spectral shapes directly in the datacubes. We rescaled the data variance (see Mackenzie et al. 2019) and we applied smoothing before extraction. The smoothing function is a Gaussian with a two-pixel radius in the spatial direction. The datacube elements, called ‘voxels’, were selected as detections if they matched the following criteria (see also Marino et al. 2018; Mackenzie et al. 2019; Cantalupo et al. 2019): signal-to-noise ratio of the individual voxels larger than 3; integrated S/N ratio in the 3D space equal to 7; integrated spectral S/N (using 1 ds optimally-extracted spectra) equal to 3; S/N of an individual voxel equal to 3 if the voxel is connected to a previously detected voxel; minimum number of connected voxels for detection equal to 30; minimum number of spectral pixels for detection equal to 3. The main sources of fake detections were sky line residuals.

We ran CubEx in the 1000−1100 pixel wavelength range (arbitrary much lower wavelength than 7660 Å) to immediately remove bright stars from the catalog. With the CubEx package we also generated optimally extracted images (OEimages) of the detections showing the extension of the detection emission line in the spatial direction (details in Borisova et al. 2016), weighted by signal to noise, and the signal-to-noise maps. We chose OEimages that are smoothed with a box car of a 2-pixel radius in the spatial direction for inspections.

To confirm a line detection at z ∼ 5.3, we therefore, made sure that it did not show any counterpart in the HST F606W image; it had reasonable segmentation and S/N datacubes; it was not at the wavelength of a strong sky line residual; it had a redshift not more than 2000 km s⁻¹ from the AzTEC-3 systemic redshift; it was not associated to a detection at a different wavelength implying the emission line was not Lyα; it had a reasonable S/N also in the even and odd mosaics (integrated S/N ratio larger than 5). This was also useful to exclude detections from cosmic rays (e.g., Lofthouse et al. 2020). Finally, we checked that its 1 d spectrum showed a reasonable Lyα shape at the detection wavelength, for example not too narrow like a spike of a few bad pixels. The criteria were applied in the form of a visual inspection of the OEimages and of the spectra.

To estimate the fidelity of our detections (i.e., down to which signal to noise the line detections start to be spurious), we applied the same set of criteria for the detection datacubes and for the datacubes multiplied by −1, that is the negative cubes. In Fig. 2, we show the ratio between the number of negative and positive detections as a function of signal to noise for the entire mosaic and for the four individual pointings. The signal to noise is calculated as the ratio between the isophotal flux integrated over the line detection by CubEx and its error. This signal-to-noise definition is different from the one based on pixel-to-pixel noise and that can produce lower values, but directly uses the output of CubEx. A S/N equal to 11 allowed that more than 60% of the detections were likely to be real detections in the mosaic. Therefore, we fixed to 11 the S/N of the line detections that made our “Main” sample. We propagated the 40% chance to have a fake detection in the estimation of the density of the protocluster (see the following sections), even if the visual inspection described above already removed 20% of the sources in the fidelity-cut catalog.

Fig. 2. Detection fidelity as a function of signal-to-noise ratio. The fidelity is expressed as 1 − Nneg/Npos, where Nnegis the numbers of the sources detected in the datacubes multiplied by −1 and Npos in the number of the detections in the original datacubes, when we apply the list of detection criteria described in the text. The red stars and line correspond to the detections in the “mosaic”, while the other lines correspond to the individual pointings, magenta for P1, blue for P2, cyan for P3, and green for P4. The signal-to-noise ratio in the x-axis correspond to the ratio between the isophotal flux and its error as provided by CubEx. The shapes of the lines reflect the uncertainty in the fidelity measurement.

The choice of the detection parameters was guided by the extensive tests of the code performance made in the mentioned papers (Marino et al. 2018; Cantalupo et al. 2019; Mackenzie et al. 2019; Lofthouse et al. 2020), with the scope of exploiting CubEx for the detection of the faintest Lyα nebulae. However, we also tested the code with a variety of combinations of parameters before choosing the best set that allows a fidelity of 60% at S/N larger than 11.

We tested the detection of sources in the mosaic as well the detection of fake sources in the negative of the mosaic cube, by changing the spatial and spectral S/N of the individual voxels (values of 2, 3, 7, and 10), the integrated S/N (values of 5, 7, and 10), and the minimum number of connected voxels (values of 5, 10, 30, and 50). We found that with a spatial S/N larger than 6, we only detect a few sources with very high isophotal S/N. For a S/N of connected voxels lower than 3, we detect more fake sources than real sources. In the case the integrated S/N is larger than 8, we miss 80% of the real sources for an isophotal S/N of 11. We found that the minimum number of voxels does not change the fidelity fraction more than 5%, however, for a value larger than 30 and for a spatial and spectral S/N of 3, we miss 15% of sources. Therefore, we fixed this parameter to 30.

As described in the following section, we kept a few lower S/N sources, with a counterpart in the HST F160W image and which Lyα emission line seemed convincing in the 1D spectrum. These sources make the “Supplemental” sample (Table 2). For one of the supplemental sources, the counterpart in the F160W image is also present in the COSMOS2015 catalog (Laigle et al. 2016) with a redshift consistent with that of the Lyα detection (see Table 2). However, given the high density of sources in the HST image, we calculated the probability of chance alignment for all the supplemental sources. We first ran Source Extractor (Bertin & Arnouts 1996) with the F160W image as detection and a detection threshold of 2σ. In this run, we identified the counterparts of all the three sources in the Supplemental sample. Then, we calculated the surface density, n(< mag), of the sources detected in the F160W catalog, brighter than those counterparts, and obtained n(m < m_446) = 0.09 arcsec⁻², n(m < m_414) = 0.07 arcsec⁻², and n(m < m_199) = 0.04 arcsec⁻². By following the discussion in Downes et al. (1986) (and references therein), we estimated the probability of chance alignment as P = 1 − exp(−nπR²), where n is the surface density (sources per arcsec²), πR² is the searching area of the counterparts, and R is the searching radius in arcsec given by the MUSE PSF. The probabilities resulted P_446 = 0.09, P_414 = 0.07, and P_199 = 0.04. A chance probability of less than 10% supports the idea that the low signal-to-noise sources are unlikely to be chance detections. However, we treat them as candidates in the estimation of the density of the protocluster in Sect. 3.4.

3. Results

We present here the list of Lyα emitting sources found in our MUSE datacube with the method described in Sect. 2.2. In addition, we also discuss sources from the literature with photometric redshift consistent with z ≃ 5.3 that could show a Lyα emission in the MUSE datacube, but that do not match our detection criteria.

3.1. Line detections compatible with Lyα at z ≃ 5.3

By using the method described in Sect. 2.2, we identified ten sources with Lyα emission line compatible to be at z ≃ 5.3. Eight of them are detected in the mosaic, one is detected in P2, and one is detected in P3. In Table 2, we report the properties of the ten sources, naming them based on the detection cube. The first seven sources make our Main list; mosaic_1513 corresponds to the system composed of AzTEC-3 and the Lyman break galaxy (LBG-3) located at 2″ north of it; mosaic_1496 corresponds to LBG-1; mosaic_1513 and mosaic_1496 are the most extended we have in our list of Lyα emitting sources and are characterized by previous ALMA [C II] detections. They deserve a special discussion (see the following sections). In addition, mosaic_1548, mosaic_1520, and mosaic_199 overlap with sources in the COSMOS2015 catalog, where their photometric redshifts agree with the Lyα redshift.

In Fig. 3, we show the spatial distribution of the ten Lyα detections on the RGB image corresponding to the MUSE mosaic. The inserts show their Lyα emission line in velocity space with respect to the systemic redshift of AzTEC-3. As we can see in the inserts, the ranges in spectral pixels of the CubEx detections are large for a few sources. In particular, for the mosaic_1548 source, CubEx identified 12 spectral pixels which correspond to a line with a tail.

Fig. 3. RGB image composed of the HST images in F814W, F125W, and F160W filters covering the area of our MUSE observations. The overlaid contours represent the 3σ levels of the Lyα emission. Red contours correspond to the sources with counterparts in the COSMOS2015 catalog. Green contours correspond to the Lyα detections without counterparts. The inserts contain the normalized Lyα profiles (normalized flux density versus velocity) of the Lyα detections indicated in the title of the inserts. The wavelengths of the maximum of the line detected by CubEx is indicated with a vertical dashed blue line. The blue shaded areas correspond to the spectral pixel ranges of the lines detected by CubEx. The zero velocity corresponds to the systemic redshift of the [C II] detection of AzTEC-3 and it is indicated as a vertical red line in each insert.

In Fig. 4, we show the 1D MUSE spectra extracted with CubEx from an area corresponding to the Lyα detection toward AzTEC-3 and LBG-1. These are the only sources in the field with a known systemic redshift from [C II] (Riechers et al. 2014). In the next section and in the appendix, we show the optimally extracted narrow-band images of all the ten Lyα detections.

Fig. 4. Left panel: 1D spectrum of the AzTEC-3+LBG-3 system in the Lyα wavelength range. The original-sampling spectrum is shown in blue with error bars, the one smoothed by 3 × 3 wavelength channels in black. The error spectrum comes from the rescaled variance cube. The magenta spectrum is the theoretical sky spectrum from Rousselot et al. (2000) in arbitrary units. Right panels: same as the left panel for LBG-1.

3.2. Sources at z ≃ 5.3 with HST counterparts

To find additional Lyα emitting sources that might have escaped our detection threshold, we also selected sources with photometric redshifts consistent with that of AzTEC-3 (4.9 < z_phot < 5.6 which is z = 5.3 ± 3σ given the typical z_phot uncertainty at 3 < z < 6) in the COSMOS2015 catalog and extracted their MUSE spectra within a fixed aperture of 0.7″ radius (about twice the mosaic PSF size). Five sources in addition to the ones mentioned in the previous section did not show any Lyα emission in the MUSE spectra.

We also investigated a method to identify a z ∼ 5 star-forming galaxy from the HST F606W, F814W, F105W, F125W, and F160W images. As we can see in Fig. 5, such a source is characterized by a sharp decrement between the F606W and F814W filters. We considered representative spectra of star-forming galaxies with Lyα in absorption and in emission (Shapley et al. 2003) and we convolved them with the HST filter transmission curves. We found that a star-forming galaxy at z ∼ 5.3 is characterized by a V₆₀₆ − I₈₁₄ color of about 3, given the non detection in F606W, and a flat spectrum redder than the F814W filter. We ran Source Extractor in dual mode, with the F105W image as a detection and the other HST-filter images as measurement images. We chose a detection minarea equal to the PSF, a detection threshold of 2, and an optimal extraction aperture given the image PSF. Also, we asked for a signal-to-noise ratio of 3 in the detection flux. We found one source with color consistent with a z ∼ 5.3, that is also contained in the COSMOS2015 catalog with z_phot = 5.2 (COSMOS2015_842471). However, the MUSE spectrum does not show any significant emission line at the wavelength of Lyα.

Fig. 5. HST filter transmission curves (ACS F606W in yellow, ACS F814W in cyan, WFC3 F105W in magenta) and representative spectra of star-forming galaxies at z ∼ 5.3 (Lyα absorber in red, Lyα emitter with a small, intermediate, and large EW(Lyα) in green, blue, black, respectively from Shapley et al. 2003). We corrected all the representative spectra for the intergalactic-medium absorption at z = 5.3 by assuming the formalism in Madau (1995), before using them.

The best photometric filter to identify the members of the AzTEC-3 protocluster would be centered at ∼7600 Å (the wavelength of Lyα at z ∼ 5.3). Sobral et al. (2018) studied the galaxies detected in the I767 filter and compiled a sample of Lyα emitters at z ≃ 5.3. However, all the sources in their sample are located outside our MUSE coverage.

Along the same line, we extracted the MUSE spectra of the galaxies discovered by Capak et al. (2011) in the AzTEC-3 overdensity. The ones showing Lyα emission at z = 5.3 in our MUSE datacube are LBG1447524 and LBG1447526 (LBG-1 and LBG-3).

3.3. Other catalogs from the literature

We also inspected other catalogs from the literature covering the AzTEC-3 area, the catalog of CO detections from Pavesi et al. (2018a) and the radio-continuum detection catalog from Smolčić et al. (2017) and van der Vlugt et al. (2021). We required non detection in the F606W image and we found only one source with Lyα at z ∼ 5.3 in both catalogs, corresponding to the AzTEC-3 galaxy. In the Pavesi’s catalog there is a source at z = 5.3 (in addition to AzTEC-3) without Lyα emission from our MUSE observation that is shown in Fig. 6 (see next section).

Fig. 6. Location of the Lyα emitting sources detected in the AzTEC-3 environment. The ones with Lyα central wavelength larger (red) and smaller (blue) than that of the AzTEC-3 system are shown as small circles. The AzTEC-3 submillimeter and the LBG-3 galaxies are shown as yellow star and yellow diamond. Sources from the literature with photometric redshifts consistent with z ∼ 5.3 are also shown, five COSMOS2015 sources with 4.9 < zphot < 5.6 (cyan big circles), Lyman break galaxies listed in Capak et al. (2011), but without Lyα emission in our MUSE datacube (green squares), one CO detection from Pavesi et al. (2018a) at z = 5.3 (gray circle). The black cross is the center of coordinates of all the sources shown in the figure and the gray cross the center obtained without considering AzTEC-3. The blue is the center of coordinates only of the sources with stellar mass larger than 109 M⊙ and the red cross is the weighted mass barycenter of the Lyα emitting sources detected in the field. The green cross is the weighted mass barycenter of the sources with a stellar mass estimation but without considering AzTEC-3.

3.4. Space distribution of all the candidates at z ≃ 5.3

In Fig. 6, we show the location in the RA–Dec plane of the Lyα emitting sources detected in the MUSE datacube and of the sources from the literature with photometric redshift consistent with being at z ≃ 5.3.

To verify if the group of sources around AzTEC-3 constitutes an overdensity of galaxies and confirms the presence of a protocluster, we compare them with the catalog of Lyα emitting sources detected with MUSE in the Hubble Ultra Deep Field (Inami et al. 2017, mosaic datacube). For a minimum Lyα luminosity of 4 × 10⁴¹ erg s⁻¹, there are 77 galaxies at 4.9 < z < 5.6 (redshift bin of 0.7) in the mosaic area of 3′×3′ (30 800 sources per deg²), which means 2200 sources per deg² in the redshift range corresponding to the peak of our Lyα line detections (5.28 < z < 5.33, redshift bin of 0.05). At the same limiting Lyα luminosity, we detected eight emitters in the Main sample, taking AzTEC-3 and LBG-3 as two separated galaxies, and 11 if we also consider the Supplemental sample. Therefore, we found 9 ± 2 emitters in our field (=4 × 10⁻⁴ deg²), that can be translated into 9 ± 3 emitters if we take also into account the 40% chance of fake detections due to the fidelity-cut criterion in the uncertainty budget or 22 500 ± 7500 sources per deg². This implies an overdensity of 10 ± 3. We, also, considered all the sources at 4.9 < z_phot < 5.6 in the COSMOS2015 catalog located outside the area covered by our MUSE observations and found a density of 1862 ± 300 galaxies per deg². The overdensity of Lyα emitters in the MUSE area is 12 ± 5 times the density of all the sources in the COSMOS2015 catalog, with and without Lyα emission. These estimations confirm the presence of an overdense region around AzTEC-3.

For comparison, the SSA22 protocluster (Steidel et al. 2000), which also contains a few SMGs, is an overdensity of about six at z ∼ 3.1. The overdensity of Lyα emitters discovered by Shi et al. (2019) contains about four times more galaxies than the field. In the Hubble Deep Field, Walter et al. (2012), Calvi et al. (2021) found an overdensity around the submillimeter galaxy HDF850.1 at z = 5.183. They presented 23 spectroscopically confirmed star-forming galaxies, including Lyα emitters, at z = 5.2 with a surface density of at least a factor of two higher than the field. None of the sources, in addition to HDF850.1, are detected at submillimeter wavelengths. Pavesi et al. (2018b) discovered an overdensity at z ∼ 5.7 similar to that around AzTEC-3. At the center of the overdensity there is a dusty starburst, probably in the phase of an on-going merger. A star-forming galaxy as massive as LBG-1 is located at 13″ away from the starburst. An overdensity of Lyα emitters was discovered in the same area with an overdensity parameter of about ten. This and our area of observations are therefore ones of the regions at z > 5, characterized by the highest overdensities in the COSMOS field.

Other overdensities of Lyα emitters have been discovered in the literature at 4 < z < 6 around radio galaxies or around quasars expected to trace dense regions of the Universe. For instance, Venemans et al. (2002) discovered that the region around the luminous radio galaxy TN J1338–1942 at z = 4.1 is 15 times more dense than the field; Venemans et al. (2004) found a density up to six times higher than in the field around the radio galaxy TN J0924–2201 at z = 5.2; Zheng et al. (2006) found an overdensity on the order of six around the radio-loud quasar SDSS J0836+0054 at z = 5.8.

Assuming that all the galaxies shown in Fig. 6 belong to the protocluster, we can see that they follow the location of our Lyα emitting sources. The center of coordinates of the most massive sources with stellar mass larger than 10⁹ M_⊙ and the weighted mass barycenter of the sources with a stellar mass estimation, even without considering AzTEC-3, are close to the submillimeter galaxy on the side of its Lyα emission peak. This implies that the region of the protocluster where the gravitational potential is the strongest is close to the SMG, which in turn is very close to the center of the protocluster.

4. Spectroscopic properties from the MUSE spectra

4.1. AzTEC-3

In Fig. 7, we show the continuum subtracted Lyα surface brightness toward AzTEC-3 corresponding to 7658−7670 Å. The original narrow band image and the continuum subtracted one are produced within CubEx optimizing the signal-to-noise ratio, as described in Borisova et al. (2016). The two main peaks of the emission roughly correspond to the positions of AzTEC-3 and the LBG-3. The emissions toward AzTEC-3 and LBG-3 are blended at the 5σ level. The entire emission occupies an area of 4.4″ × 6.3″ (about 27 × 38 kpc² and 170 × 240 ckpc²).

Fig. 7. Upper left: signal-to-noise optimally extracted narrow-band image (Borisova et al. 2016) of the continuum-subtracted Lyα emission toward the AzTEC-3+LBG-3 system, rescaled to surface brightness units, and smoothed using a 2-pixel Gaussian kernel. The narrow band is built considering the 13 wavelength channels of the detection. The 5 white contours reveal the 1.3 to 3.8 erg s−1 cm−2 arcsec−2 levels, which correspond to 3−10σ in this image. The black letters indicate the directions we refer to in the text (N = north, NW = northwest, S = south, SE = southeast, W = west of the SMG). The small black segment indicates the 1″ scale. We also show the continuum-subtracted Lyα surface brightness contours overplotted to the HST ACS F606W (upper right), F814W (lower left), and F160W (bottom right) images. The 0.5″-side square indicates the center of the F160W position of LBG-3. The PSF of our MUSE data is 0.7″, corresponding to about 4 kpc or 27 ckpc at z = 5.3. The eight red contours correspond to the [C II] emission (from 3σ to 10σ where 1σ = 2.38 × 10−4 Jy beam−1) at 301.671−302.142 GHz, corresponding to −230 to 230 km s−1 from the peak emission. The synthesized beam of the [C II] ALMA observations is 0.63″ × 0.56″ as reported in Riechers et al. (2014). The 3σ contour extends toward LBG-3.

As expected for a source at z ∼ 5, the sources are not visible in the F606W filter and the Lyα wavelength is contained in the F814W filter. The peak of the Lyα emission toward the SMG is offset toward the southeast with respect to the star-formation knots seen in the HST images and traced by the [C II] detection. This may be related to the presence of dust on the main knot of star formation of the SMG (see next section). The LBG peak is offset by 0.5″ (20 ckpc) with respect to the source seen in the HST images and elongated to the north. Since the Lyα emission is very sensitive to the even small presence of dust, this is in agreement with the idea that on the main star-formation knot of LBG-3 some of the Lyα photons could be absorbed by dust (see next section).

Even if the Lyα emissions toward the SMG and the LBG seem connected, the spatial shape of the highest S/N contours indicates that the peak Lyα emission of the SMG is quite symmetrical. However, that of the LBG seems to be elongated and tilted toward the north (N position in the figure) and extends up to 2″ (almost 80 ckpc). Also, the Lyα emission of the SMG shows extended components toward various directions. It is worth noting that a low S/N emission from the SMG toward the direction of LBG-3 was also observed in the [C II] observations, despite the main [C II] flux was found to be concentrated in a compact position on top of the main knot of star formation, as described in Riechers et al. (2014).

In Fig. 8, we show the Lyα channel maps of the AzTEC-3+LBG-3 system. Each channel is separated by approximately 1.5 Å. The figure shows that the Lyα emissions toward the two galaxies peak at similar wavelengths, 7662 Å for LBG-3 and 7664 Å for AzTEC-3, respectively. The Lyα emission associated to the SMG is visible in all the channels at 4σ, while that probably associated to LBG-3 is significant at 7660−7665 Å. At 7665 Å, the emission is mainly concentrated on top of the SMG, but shows a kind of a bridge between SMG and LBG, which could represent a region of interaction between the two close-by galaxies (see Sect. 6). The Lyα emission of the bridge is observed from 7660 to 7666 Å, with a maximum at 7664 Å. It occupies a wide range in Lyα wavelengths, that could correspond to a region of a large range of gas velocities, together with large HI column densities. These gas characteristics could be the result of the interactions in the AzTEC-3+LBG-3 system.

Fig. 8. Lyα channel maps toward the AzTEC-3+LBG-3 system. Each panel corresponds to the indicated wavelength and velocities with respect to the systemic redshift inferred from the [C II] emission line. The white contours represent the isophotes at S/N = 4, 5, and 6. The red square corresponds to the F160W position of LBG-3 and the red circle to the location of the peak of the [C II] emission of AzTEC-3. The green contours in the fifth panel represent the contours of the continuum-subtracted Lyα surface brightness. The horizontal gray-color bar shows a signal-to-noise ratio from 3 to 5. We refer to Fig. 7 for the 1″ scale.

At the position of the main star-forming knot of the SMG and of the peak of the Lyα emission associated to the SMG, the Lyα emission extends from 200 to 600 km s⁻¹. The velocity could be associated to a star-formation driven outflow of neutral gas, departing from the SMG and mixing with the interacting gas in the bridge between SMG and LBG. This interaction could produce some of the extended tail seen also in the Lyα 1D spectrum. The blue shift of low-ionization absorption lines, directly tracing the kinematics of the HI gas, together with the radiative-transfer modeling of the Lyα emission (Sect. 6), sensitive to kinematics and amount of HI gas, could provide support of this hypothesis.

At 7662 Å, the Lyα emission appears elongated to the north of LBG-3 (N position in Fig. 7) and to the southeast of AzTEC-3 (SE position in Fig. 7). Above LBG-3, the Lyα emission is compact in wavelength. The Lyα emission is compact in wavelength (around 300 km s⁻¹) also in the region below the peak of the Lyα emission associated to the SMG. In this position, the gas kinematics could be associated with neutral gas flows between the SMG and the mosaic_199 source (Table 3).

Since Lyα photons can be scattered by HI gas and so reveal its presence, the elongated shape of the Lyα emission to the north of LBG-3 (N position in Fig. 7), in the bridge between SMG and LBG, and to the southeast of the SMG (SE position in Fig. 7) could be the result of the interaction between AzTEC-3, LBG-3, and possibly mosaic_199. As shown in the simulations described in Yajima et al. (2013), at the first passage of a merging event between star-forming galaxies, the Lyα emission coming from the progenitors can be intense due to the triggered star-formation event and its shape could also follow the gas distorted during the interaction in the outer regions.

In Fig. 4, we can see the integrated 1D spectrum of the AzTEC-3+LBG-3 system in the Lyα wavelength ranges, free of strong sky line residuals. Neither metal nor AGN-diagnostic lines are detected in the spectrum. As expected from Fig. 8, the main peak of the Lyα line is redshifted with respect to the systemic redshift by about 400 km s⁻¹. The entire line including its red tail occupies a velocity range up to 900 km s⁻¹. As shown in Verhamme et al. (2006), the shift of the Lyα red peak can be 2 or 3 times the velocity of the outflowing gas depending on the HI column density conditions. Riechers et al. (2014) measured that the central velocity of the OH163 μm doublet from the ALMA observations was blueshifted of about 100 km s⁻¹ with respect to the [C II] line and this could be associated to an outflow, maybe produced by the AzTEC-3 starburst.

The integrated flux of the Lyα emission of the entire system is (16.92 ± 0.37)×10⁻¹⁸ s⁻¹ cm⁻². By estimating a continuum on the order of 3 × 10⁻¹⁹ erg s⁻¹ cm⁻² Å⁻¹ on the red side of Lyα by linear fit, we calculate a rest-frame EW(Lyα) = 13 ± 2 Å. The FWHM(Lyα) of the main peak excluding the long tail is 220 km s⁻¹.

In Fig. 9, we show the spectroscopic comparison between [C II] and Lyα emission in velocity space. It is worth remembering that the Lyα emission contains contributions from AzTEC-3 and also from LBG-3, while the [C II] emission is only concentrated at the location of the SMG as seen in the HST images. The zero velocity is given by the [C II] systemic redshift. Both AzTEC-3 and LBG-3 contribute to the main peak and to the extended tail (at more than 500 km s⁻¹) of the integrated Lyα emission, even if the tail is mostly dominated by the SMG.

Fig. 9. Left panel: Lyα (blue) and [C II] (red) profiles of the AzTEC-3+LBG-3 system in velocity space and normalized to the maximum. The [C II] spectrum is taken from Riechers et al. (2014) and it is associated to AzTEC-3 (see Fig. 7). Right panel: normalized [C II] profile (red), Lyα profile of the emission extracted at the position of the main Lyα peak associated to the SMG (blue and black curves), extracted from the position of the rest-frame UV knots of AzTEC-3 as seen in the F160W image (green), and extracted at the position of the main Lyα peak associated to LBG-3 (cyan). The extraction is done in an aperture of 0.7″ radius for the blue, green, and cyan profiles and increasing the S/N of the connected pixels in CubEx for the black profile. For simplicity, we call the spectra as onlySMG Lyα, onlySMG UV knots, and onlyLBG. The normalization of these spectra is performed with the same factor used to normalize the spectrum of the AzTEC-3+LBG-3 system in the left panel to show the different intensities. The normalization of the [C II] profile is also performed accordingly.

We separated the 1D Lyα spectrum of the SMG and of LBG-3 in two ways, with CubEx and extracting a fixed-aperture spectrum from the MUSE datacube. To make CubEx detect two separated (not blended) sources, we increased the S/N of the connected pixels from 3 to 5, the spatial and spectral S/N from 3 to 7, and decreased the minimum number of voxels from 30 to 5 in the detection parameters. This way CubEx is able to detect the brightest Lyα emission associated only to the SMG and that associated only to the LBG-3. Also, we extracted MUSE spectra within a 0.7″-radius aperture and obtained the spectra at the position of the brightest Lyα regions only coming from the SMG (onlySMG Lyα spectrum) and only from the LBG (onlyLBG spectrum). The profile of the Lyα emission just coming from the SMG presents a main peak which is fainter and broader than the main peak of the emission coming from the entire AzTEC-3+LBG-3 system (left panel of Fig. 9). The Lyα profile, obtained from an aperture including all the rest-frame UV knots of the SMG (onlySMG UV knots spectrum) shows a sharper slope on the red side. However, the overall shapes of the Lyα emissions are comparable.

Even if we focus on the brightest part of the Lyα emission only associated to the SMG (as the [C II] emission is), we see that the Lyα main peak is redshifted with respect to the [C II] by the same amount as the entire AzTEC-3+LBG-3 system.

It is worth noting that Riechers et al. (2014) estimated a star-formation rate per unit of area of 530 M_⊙ yr⁻¹ kpc² (see also Riechers et al. 2020, for a discussion), which is larger than the limit of 0.1 M_⊙ yr⁻¹ kpc⁻², satisfied by local starbursts and high-z Lyman Break galaxies to sustain starburst-driven outflows (Heckman 2001). Starburst-driven outflows could favor the escape of Lyα photons even from dusty interstellar media (e.g., Kunth et al. 1998; Verhamme et al. 2006), as that of the SMG, and they could play an important role in producing the observed Lyα surface brightness from AzTEC-3. At 34 ckpc (0.9″) from the star-forming region traced by the [C II] emission peak toward the southeast of the SMG (SE position in Fig. 7), we measure a Lyα surface brightness of ∼4.4 × 10⁻¹⁸ erg s⁻¹ cm⁻² arcsec⁻², indicating a L(Lyα) ∼ $\text{SB}(\text{Ly}\alpha )\times \text{4}\pi \text{0}.9^2\times \text{4}\pi {\text{D}}_{\mathrm{L}}^2(\text{z=5}.\text{3})\sim \text{1}.\text{3}\times {\text{10}}^{\text{43}}$ erg s⁻¹.

It is interesting to investigate what could be the main mechanism of the production of the Lyα photons that extend up to at least 30 ckpc. The molecular gas mass of 5.7 × 10¹⁰ M_⊙ (Riechers et al. 2020) and the SFR_FIR of 1100 M_⊙ yr⁻¹ imply that the starburst could be maintained for a time scale on the order of 50 Myr, as an upper limit assuming that no significant gas mass is lost due to the outflow itself. An outflow of a constant velocity of 800 km s⁻¹ would reach a maximum distance of 40 kpc in a time scale of 50 Myr and could have already reached part of that distance in a shorter time scale. This outflow could channel the escape of ionizing radiation, ionized, and neutral gas.

Moreover, the Lyα luminosity of 1.3 × 10⁴³ erg s⁻¹ at a distance of 34 ckpc could be explained in terms of recombination of atoms ionized by a starburst of a thousand solar masses per year. In fact, following Eq. (7) in Cantalupo (2017), the luminosity could be translated into a ionization luminosity of ∼1 × 10⁵⁴ photons s⁻¹ consistent with that of a starburst of 1000 M_⊙ yr⁻¹ (e.g., Leitherer et al. 1999). Also, following Eq. (5) from the same paper, it can be shown that, for a volume of 4/3π(34 ckpc)³, a T = 10⁴ K, and a filling factor of the interstellar medium clouds of 10⁻⁴ (e.g., Cantalupo et al. 2014), a cloud density of ∼3 atoms cm⁻³ would imply the observed L(Lyα) at 34 ckpc. At this density and temperature, recombination is a plausible scenario to explain the observed L(Lyα) and surface brightness.

The luminosity at 90 ckpc would imply a lower cloud density of less than 1 atom cm⁻³ that difficultly explains a recombination mechanism from the starburst. However, at this larger distance the HI scattering (free of dust absorption) could play a role in the case of N HI > 10²¹ atoms cm⁻². The radiative-transfer-model fit of the 1D integrated Lyα profile of the SMG and of the AzTEC-3+LBG-3 system can give insight on the N HI quantity. The gas at the peak of the Lyα emission and at 90 ckpc from the star-forming region could be the result of the interaction between the SMG and the mosaic_199 source (see below).

4.2. LBG-1

Among the star-forming galaxies with Lyα in emission detected in the MUSE datacube, we focus here on LBG-1, because it is the only galaxy, apart from AzTEC-3, for which we have a systemic redshift, inferred from the [C II] emission line (Riechers et al. 2014). In the next section, we consider two additional Lyα detections with photometric redshifts from the COSMOS2015 catalog (Fig. 3).

In Fig. 10, we show the emission detected by CubEx at a wavelength around 7661 Å, corresponding to Lyα toward LBG-1. This galaxy was studied in Capak et al. (2011), Riechers et al. (2014), Pavesi et al. (2019). Its known physical properties are reported in Table 2. Unlike the UV which clearly shows at least three components, the Lyα emission is smooth and encompasses the three UV components. The integrated L(Lyα) is equal to (15.9 ± 0.7)×10⁴¹ erg s⁻¹ ((5.5 ± 0.3)×10⁴² erg s⁻¹ if corrected by dust extinction, following Calzetti et al. 2000 and A_V = 0.2) and rest-frame EW(Lyα) = 2.0 ± 0.2 Å, by assuming the continuum magnitude (z_p = 23.71) from the COSMOS2015 database (the continuum is very noisy and affected by sky line residuals redwards of the Lyα wavelength in the MUSE spectrum).

Fig. 10. Upper left: signal-to-noise optimally extracted narrow-band image (Borisova et al. 2016) of the continuum-subtracted Lyα emission, rescaled to surface brightness units, and smoothed using a 2-pixel Gaussian kernel of LBG-1. The five white contours reveal the 0.8 to 2 × 10−18 erg s−1 cm−2 arcsec−2 levels, which correspond to 3 and 8σ in this image. The letters indicate reference positions described in the text. The small black segment indicates the 1″ scale. We also show the Lyα surface brightness contours overplotted to the F606W (upper right), F814W (lower left) ACS, and F160W WFC3 (lower right) HST images. The PSF of our MUSE data is 0.7″, corresponding to about 4 kpc or 27 ckpc at z = 5.3. The eight red contours correspond to the 3−10σ (1σ = 2.38 × 10−4 Jy beam−1) of the [C II] emission at 301.847−302.093 GHz, corresponding to −120 to 120 km s−1 from the peak emission. The synthesized beam size of the ALMA [C II] observations is 0.63″ × 0.56″ as reported in Riechers et al. (2014).

The size of the entire Lyα emission is 3.2″ × 3.2″ (about 20 × 20 kpc² or 125 × 125 ckpc²). As shown in Riechers et al. (2014), the [C II] emission at 301.974 GHz encompasses the three knots with a lower signal-to-noise tail toward the southwest (SW position in Fig. 10) as well. At MUSE resolution, we see that significant Lyα flux comes from the three components of LBG-1, maybe indicating that a low dust extinction is allowing Lyα photons to escape also close to the regions of star formation. In Fig. 11, we present the Lyα channel maps toward LBG-1. The figure shows that the Lyα emission is concentrated on the northern region of LBG-1 and that only at the longest wavelengths the emission comes only from the south (S and SW positions in Fig. 10). This could indicate that toward the south, LBG-1 is characterized by larger HI column densities and higher outflow velocities.

Fig. 11. Lyα channel map of LBG-1. Each panel corresponds to the indicated wavelength and velocities with respect to the systemic redshift inferred from the [C II] emission line. The white contours represent the isophotes at S/N = 4, 5, and 6. The red circle shows the F160W position of LBG-1 as in Fig. 1, drawn to drive the eye. The green contours in the third panel represent the contours of the continuum-subtracted Lyα surface brightness. The horizontal gray-color bar shows a signal-to-noise ratio from 3 to 5. We refer to Fig. 10 for the 1″ scale.

To increase the S/N and investigate the origin of the Lyα emission coming from the different directions of LBG-1, we slice the MUSE cube along the pseudo slits shown in the left side of Fig. 12, horizontal on the Lyα emission peak (first row), tilted by 110° on the Lyα emission peak (second row), tilted by ten degrees below the Lyα emission peak (third row), and tilted by ten degrees above the Lyα emission peak (fourth row). These slit orientations are chosen to isolate the Lyα emission coming from the star-forming knots of LBG-1 to that coming from above and below of them. We can see that at the position of the main Lyα peak, the emission extends from 7658 to 7672 Å. The highest velocity regions come from the center toward the southwest (S and SW position in Fig. 10). Toward the extreme southwest (SW position in Fig. 10), the Lyα emission reaches 7672 Å, corresponding to 400 km s⁻¹ with respect to the main peak wavelength. This value can be related to the gas kinematics, in a wrap of the material of the three knots and/or to high HI column densities (see Sect. 6). The kinematics of the gas could be related to the interplay of the gas in the merger of the three knots, as suggested by Riechers et al. (2014), the high column density could suggest that the merger is “wet” and/or there is a large gas reservoir supplying the star formation in the entire region. By slicing above the main peak, the Lyα emission is confined in wavelength.

Fig. 12. Left panels: pseudo slits oriented horizontally on the Lyα emission peak (first row), tilted by 110° on the Lyα emission peak (second row), tilted by ten degrees below the Lyα emission peak (third row), and tilted by ten degrees above the Lyα emission peak (fourth row). Middle panels: position-velocity diagrams obtaining slicing the continuum-subtracted MUSE datacube with a pseudo slit oriented as shown in the left panels. In the left y-axis, we indicate the wavelength and in the right y-axis the velocity with respect to the systemic redshift inferred from the [C II] emission line, the x-axis corresponds to the position along the pseudo slit in degrees (from the east to the west, except in the second row where it is from the south to the north). The white contours correspond to the 1, 2, 3σ level. Right panels: 1D spectra corresponding to the position-velocity diagrams of the middle panels. Wavelength is on the y-axis in the same range as in the middle panels and the flux normalized to the maximum is on the x-axis. The color coding of these spectra is like in Fig. 13.

In Fig. 13, we show the [C II] and Lyα profiles as a function of the velocity with respect to the [C II] systemic redshift. The integrated 1D spectrum of the Lyα emission line is composed by a narrower peak and a much broader component that reaches the continuum at 7680 Å. The FWHM of Lyα is larger than that of [C II]. This is not unusual in high-redshift Lyα emitting galaxies (e.g., Matthee et al. 2020a) and it could be related to the variety of HI kinematics properties that can condition the formation and escape of Lyα photons. Also, the tail of the Lyα emission could indicate that there is gas in a wide range of kinematic conditions and with different HI column densities around the three star-forming knots. It could be consistent with rotation among the three knots as well. In fact, the wavelength range of this tail corresponds to the feature visible in the second and third rows of Fig. 12 at large velocities and coming from the southwest. It is possible that this feature comes from the location of the [C II] emission and also from the region below the three knots. No metal lines are detected at enough signal to noise in the MUSE spectrum.

Fig. 13. Left panel: 1D profiles of the Lyα (blue) and [C II] (red) emissions toward LBG-1. The [C II] profile is taken from Riechers et al. (2014) and corresponds to the region outlined in Fig. 10 (red contours). Right panel: [C II] profile in red as in the left panel and Lyα profile obtained slicing the Lyα emission horizontally on top of LBG-1 (green), tilted by 110° on top of LBG-1 (cyan), tilted by ten degrees below LBG-1 (magenta), and tilted by ten degrees above LBG-1 (blue). The color coding of these spectra is like in Fig. 12. The two former spectra are the ones with higher signal to noise as can be seen by the smaller error bars. The thin black curve corresponds to the spectrum of the integrated Lyα emission as the blue curve of the left panel.

In Fig. 13b, we show the [C II] profile in velocity space together with the Lyα emission extracted slicing the Lyα image with pseudo slits oriented as in Fig. 12. These spectra are very noisy, but we can see that the highest peak of the integrated Lyα emission receives a contribution from the three knots, while the tail may not be originated from the northern region of LBG-1 (N position in Fig. 10). This could indicate the presence of a gas with higher velocity to the south and may suggest a merger of the three components. Alternatively, the gas could be characterized by larger N HI. By studying the Lyα profile in terms of radiative transfer models could inform about the two possibilities (see next section).

5. Lyα radiative transfer model

We use the most updated version of the FLaREON radiative transfer model (Gurung-López et al. 2019b) to quantify the properties of the interstellar, circum-galactic, and intergalactic (IGM) medium that can condition the shape and intensity of the Lyα emission in the AzTEC-3 protocluster. FLaREON was based on LyaRT (Orsi et al. 2012), a radiative transfer Monte Carlo code of Lyα emission, and predicted the Lyα line profile escaping a galaxy through different outflow configurations. Several outflow geometries were implemented, such as thin shell of HI gas (see also Verhamme et al. 2006; Gronke & Dijkstra 2016), and a galactic wind (Fig. 1 in Gurung-López et al. 2019a). In the new version of FLaREON, called zELDA, Gurung-Lopez et al. (2022) developed a new thin shell model in which the intrinsic galaxy spectrum contains a continuum in addition to a (Gaussian) Lyα emission line. The variable set delivered by zELDA includes the systemic redshift, z_in, the outflow expansion velocity, V_exp, the HI column density, N HI, and the dust optical depth, τ. Inflows are predicted such as the outflows, but with negative expansion velocities.

To be able to perform a quantitative comparison between our observed Lyα profiles and the ones predicted by zELDA, a Gaussian kernel representing the MUSE resolution is applied to the predicted profiles and the systemic redshift is provided. Also, the observed-spectrum sampling and signal to noise are taken care of in the observed frame (see Sect. 6.1 of Gurung-López et al. 2021, for details). The quantitative comparison involves obtaining the model that produces the best fit of the observed spectrum. The best fit is obtained with a Markov chain Monte Carlo (MCMC) approach. In each step of the chain, a line profile corresponding to the variables mentioned above is computed and compared with the observed spectrum. The MCMC takes into account the uncertainties in the observed spectrum. After the MCMC run, we consider the model corresponding to the 50th percentile of the probability distribution function of the variable as the best-fit profile and the 16th (84th) percentile as the model corresponding to the lower (upper) limit of each variable.

As shown in Gurung-Lopez et al. (2022), there is a grid of parameters where the MCMC chains can look for the best combination of parameters to reproduce the observed spectrum. Expansion velocities from 0 to 1000 km s⁻¹ and HI column densities from 10¹⁷ to 10^21.5 atoms cm⁻² are explored (their Sect. 2). However, the starting point of each chain is set by performing an initial optimization, which is an attempt to narrow the parameter space (their Sect. 4). The uncertainties of the best-fit zELDA parameters depend on the resolution, sampling, and signal to noise of the observed spectra (their Sect. 4). The zELDA performance is tested for a signal to noise larger than 5 at the wavelength of the main Lyα peak. The dust optical depth is poorly constrained even in the best signal-to-noise scenario, while the uncertainty on V_exp can increase by a factor of 25% when the signal to noise changes from ten to 5 and the uncertainty on N HI is almost four times larger. It means that by fitting only the Lyα profile with zELDA, we are not able to provide a significant estimation of the dust absorption, but we can aim at a reliable estimation of V_exp and N HI when the signal to noise of the observed Lyα spectrum is larger than ten.

According to the analysis in Gurung-Lopez et al. (2022), we can expect an anticorrelation between Lyα luminosity and N HI. This is explained considering that a gas with higher N HI would produce a larger number of scattering events and so a longer escaping path that can be associated to a larger dust attenuation and so a lower escape fraction of Lyα photons than in the case of lower N HI. However, the scattering events could also produce larger Lyα nebulae depending on the viewing angles.

We performed the zELDA fits of the spectra of the AzTEC-3+LBG-3 system (mosaic_1513), of LBG-1 (mosaic_1496), of the fixed-size aperture spectra extracted on top of the SMG Lyα emission peak (onlySMG Lyα), on top of the UV knots of the SMG (onlySMG UV knots), on top of LBG-3 (onlyLBG), on the bridge between SMG and LBG (bridgeSMGLBG), extracted from pseudo-slits oriented horizontally on LBG-1, tilted, tilted below, and tilted above LBG-1 (see Figs. 12 and 13). Also, we performed the fit for the sources mosaic_1548 and mosaic_1520 for which we have a signal-to-noise ratio larger than ten and an estimation of the physical parameters and photometric redshift from the COSMOS2015 catalog. We report the best fit values of the most reliable parameters, V_exp and N HI, in Table 3, together with the χ² of the best fit. We also report the ranges of the parameters contained within the 16th and 84th percentiles to show the extent of the parameter space.

5.1. Radiative transfer modeling of the AzTEC-3+LBG-3 system

In Fig. 14a, we show the models that best fit the observed spectrum of the AzTEC-3+LBG-3 system. The observed spectrum is shown in the vacuum framework as the zELDA models. By fixing the systemic redshift to that provided by the [C II] emission line, we estimated the MCMC model that best fits the data at wavelengths larger than the Lyα systemic wavelength (green curve in the figure). In fact, the zELDA models do not take into account the IGM absorption, but at z ≃ 5.3 the IGM could condition the spectrum at wavelengths bluer that Lyα. Also, we tried to correct the bluer side of the spectrum for the IGM effect by using the average prescription by Madau (1995) and run an MCMC chain of the corrected spectrum of the AzTEC-3+LBG-3 system (red curve in the figure). The best fit zELDA models in these two cases provided similar combinations of parameters. However, since the IGM could affect the blue side of the Lyα emission line all the way through the maximum of its red peak, we tried also a zELDA fit only of the reddest part of the Lyα emission line, at λ_vacuum > 7664 Å (blue curve in the figure). The spectrum of the AzTEC-3+LBG-3 system is composed by one main peak and an extended tail at λ_vacuum > 7668 Å. By masking the wavelengths bluer than the maximum of the Lyα peak, zELDA finds the best compromise between the red peak and the extended tail of the line as the best fit, which corresponds to models with a wide range of HI column densities (see Table 3) and V_exp up to 90 km s⁻¹.

Fig. 14. Left panel: observed-frame spectrum at the Lyα wavelength of the AzTEC-3+LBG-3 system in the vacuum. The observed spectrum is shown as data points with error bars and the smoothed spectrum as a black curve like in Fig. 9. The best-fit zELDA model that corresponds to the 50th percentile is shown in green and the green shaded area contains the models within the 16th and 84th percentiles obtained fitting the spectrum at wavelength larger than the systemic redshift. In blue, we show the models obtained fitting the spectrum only at the wavelength larger than the maximum. The red curve corresponds to the best fit in the wavelength range 7630−7700 Å for the AzTEC-3+LBG-3 spectrum corrected for the IGM absorption, by using the prescription of Madau (1995). Right panel: best-fit zELDA models for the spectra extracted in a 0.7″ aperture located on the position of the main Lyα emission associated to AzTEC-3 (upper left), the position of the UV knots of the SMG (upper right), the position of the main Lyα emission associated to LBG-3 (lower left), and the bridge between SMG and LBG (lower right). Black dots with error bars are the observed-frame spectra as shown in Fig. 9, the black lines are the smoothed spectra, the green (blue) curves and shaded areas are the best-fit models and the models within the 16th and 84th percentile of the model parameter space obtained fitting the spectra at wavelength larger than the systemic redshift (larger than 7669 Å). Vertical blue dashed lines indicate the Lyα redshift given by the AzTEC-3 [C II] detection. The zELDA fits are performed fixing the redshift to the systemic inferred by the [C II] emission peak.

In general, zELDA mainly accounts for the main peak of the Lyα emission line, which is dominant in intensity with respect to the extended tail. To identify the regions that most contribute to the extended tail at λ_vacuum > 7668 Å, we ran zELDA fits of the onlySMG Lyα, onlySMG UV knots, onlyLBG, and bridgeSMGLBG spectra (Fig. 14b). A main red peak and an extended tail are seen in the onlySMG spectra and we found a zELDA best fit for the main peak and one for the extended tail for them. For the onlySMG Lyα spectrum, the Lyα profile is consistent with zELDA models characterized by V_exp ≤ 50 km s⁻¹ and N HI on the order of 10²⁰ atoms cm⁻² for the main red peak and by V_exp ∼ 800 km s⁻¹ and N HI on the order of 3 × 10²⁰ atoms cm⁻² for the extended tail. The N HI value could be responsible for the scattering of Lyα photons up to 90 ckpc as mentioned in Sect. 4.1. On the contrary, in the region of the UV knots of star formation, the AzTEC-3 Lyα profile is consistent with zELDA models with V_exp < 100 km s⁻¹ and N HI < 10¹⁹ atoms cm⁻².

The best fit zELDA models of the bridgeSMGLBG and onlyLBG spectra are both characterized by V_exp ∼ 20 km s⁻¹ and N HI on the order of 10²⁰ atoms cm⁻². Therefore, the HI column density could be responsible for the escape of Lyα photons in the distorted region above LBG-3 (N position in Fig. 7) and in the region of interaction between AzTEC-3 and LBG-3. In this region of interaction, the gas could be turned into stars, produce star formation under favorable conditions, and allow the formation of new Lyα photons as well. In Fig. 16a, we present a cartoon to better visualize the combination of parameters inferred by zELDA for the AzTEC-3+LBG-3 system.

5.2. Radiative transfer modeling of LBG-1

In Fig. 15a, we show the results of modeling the Lyα profile of LBG-1. We performed the fit at wavelengths larger than the systemic redshift and at λ_vacuum > 7669 Å to account only for the tail. In the LBG-1 spectrum, the main red peak has intensity more comparable to the extended tail than in the case of the AzTEC-3+LBG-3 spectrum, making the best fit zELDA models broader on average to account for the tail. The best fit of the main peak is consistent with a model with V_exp < 30 km s⁻¹ and N HI = 5 − 10 × 10²⁰ atoms cm⁻². The extended tail is consistent with models with a large range of expansion velocities, up to 300 km s⁻¹, and HI column density up to 9 × 10²⁰ atoms cm⁻².

Fig. 15. Left panel: observed-frame spectrum at the Lyα wavelength of LBG-1. The observed spectrum in vacuum wavelengths is shown as black datapoints with error bar, the black curve is the smoothed spectrum. The green curve is the best-fit zELDA model that corresponds to percentile 50 and the green shaded area contains the models within percentiles 16 to 84. The blue curve corresponds to a zELDA fit at wavelengths larger than 7669 Å, obtained with the scope of finding a zELDA model for the extended tail at wavelengths larger than the main red peak. Right panels: zELDA fits of the spectra obtained collapsing the LBG-1 spectrum as in Fig. 13. In green, we show the best fit and the models within the 16th and 84th percentile obtained for wavelengths larger than the systemic redshift. The blue curves correspond to the best fit of the extended tail for the spectrum in the left panel. Vertical blue dashed lines indicate the Lyα redshift given by the LBG-1 [C II] detection, we fixed while performing the fits.

To investigate which regions of space and combination of parameters better reproduce the extended tail, we performed zELDA fits of the profiles obtained slicing the Lyα emission like in Fig. 12 and we show the results in Fig. 15b. Given the noisy spectra, we obtained a large uncertainty in the best fit models and their parameters. However, we can see that the best fit models of the slices containing Lyα emission coming from the N and E positions (included in the horizontal, tilted pseudo slit located on top of LBG-1, and the one above LBG-1, see Fig. 10) are consistent with HI column density up to 10²¹ atoms cm⁻² and V_exp < 30 km s⁻¹. The best fit is consistent with models of lower N HI and larger V_exp in the other position. This could support the idea of random distortion of the gas due to the merger of the three components toward the south of LBG-1. The combination of parameters inferred by zELDA for the LBG-1 region are shown in Fig. 16b.

Fig. 16. Qualitative representation of the combination of the zELDA parameters that best fit the Lyα emission of the AzTEC-3+LBG-3 system (left panel) and of the LBG-1 region (right panel). The size of the dots in the cartoon is proportional to Vexp and the color scale corresponds to the N HI value in a way that fainter color indicates lower N HI. The smallest and darkest dots show Vexp < 30 km s−1 and N HI ∼ 1021 atoms cm−2. The largest dot corresponds to Vexp up to 800 km s−1, the faintest color to N HI ∼ 1019 atoms cm−2.

5.3. Radiative transfer modeling of the Lyα emitting sources without systemic redshift

We ran zELDA leaving the redshift as a free parameters for the Lyα emitting sources in the AzTEC-3 environment without systemic redshift (Fig. 17). We focus on the sources with a counterpart in the COSMOS2015 catalog and with a photometric redshift consistent with z ∼ 5.3. The best fit parameters are shown in Table 3, where we also report the best fit systemic redshift value. As expected, the best fit redshift is consistent with z ∼ 5.3. The best fit zELDA models are consistent with galaxies with HI column densities up to 10²⁰ atoms cm⁻² and expansion velocities on the order of a few tens of km s⁻¹. This combination of parameters is consistent with the fact that these sources are characterized by low masses and moderate star-formation rates (see Table 2). In particular, the mosaic_1520 source is characterized by a star-formation rate of more than 100 M_⊙ yr⁻¹ and it is plausible that this rate is accompanied by an outflow with a velocity up to 60 km s⁻¹. Since the CubEx detection map shows a S/N larger than 5 from 7662 to 7675 Å, we propose an alternative zELDA solution for the spectrum of the mosaic_1548 source. In this solution, we found a combination of parameters that account for two peaks, a higher blue one at 7665 Å and a lower red one at 7672 Å. The best fit zELDA model suggests the presence of an inflow of a few tens of km s⁻¹ and a gas with a low column density up to 2 × 10¹⁹ atoms cm⁻². If confirmed, this solution indicates the presence of an inflow at the outskirt of the protocluster. However, the presence of an inflow, visible as a Lyα blue peak at z ∼ 5.3, would imply the presence of an ionized bubble in the vicinity of the region of the production of Lyα photons and that those photons survive the IGM absorption from z = 5 to z = 0.

Fig. 17. Observed-frame spectrum at the Lyα wavelength of mosaic_1548 (left), mosaic_1520 (right). The observed spectrum in vacuum wavelengths is shown as black data points with error bars, the black curve is the smoothed spectrum. The green curve is the best-fit zELDA model of a redshifted Lyα main peak that corresponds to percentile 50 and the green shaded area contains the models within percentiles 16 to 84. The vertical blue dashed lines indicate the Lyα redshift given by the 50th percentile model. For the left hand side spectrum, we also show in magenta a zELDA fit of the two visible peaks at 7665 and 7672 Å. The systemic wavelength corresponding to this fit is at 7668.1 Å and the model implies a blue peak higher than the red peak as in the case of an inflow (see Table 2 for the best fit parameters).

6. Discussion

In the 1.4 × 1.4 arcmin² region around the submillimeter galaxy AzTEC-3, we found ten Lyα emitting sources, including one extended emission toward AzTEC-3 itself and one toward the Lyman break galaxy, LBG-1. Significant [C II] detections were also observed at the positions of AzTEC-3 and LBG-1 (Riechers et al. 2014, 2020; Pavesi et al. 2019) and a significant CO molecular gas detection was detected at the positions of AzTEC-3 (Riechers et al. 2010). In the following section, we investigate the relation between Lyα, [C II], and CO emissions and we discuss the properties of the AzTEC-3 protocluster we can derive from them.

6.1. Dust and gas content revealed by Lyα, [CII], and CO detections in the AzTEC-3 environment

As described in Riechers et al. (2010, 2014, 2020), AzTEC-3 shows detections of the CO transitions from the Very Large Array, the NOrthern Extended Milllimeter Array, and the Plateau de Bure interferometer, and also rest-frame 1 mm continuum, [C II], and OH163 μm doublet emissions from an ALMA campaign and the COLDz survey.

The CO peak emission was observed to be offset from the HST ACS F814W image of the SMG by about 0.5″ (3 pkpc). This offset could be partially due to dust obscuration in the rest-frame UV image (Riechers et al. 2010). The dust obscuration could also explain the difference between the UV- and FIR-derived star-formation rates of AzTEC-3 (Capak et al. 2011), suggesting that the regions of most intense star formation are highly dust obscured. In fact, a dust mass on the order of 3 × 10⁸ M_⊙ was estimated from the SED fit of the optical-through-IR spectral energy distribution of the SMG (Riechers et al. 2014, 2020). Our MUSE observation shows a shift between the HST position of the SMG and the Lyα peak emission, which intensifies toward the southeast and the northeast. Lyα is not detected to the west. This could indicate that the dust on the main knots of star formation prevents the escape of Lyα photons on one side.

The molecular gas detections were in favor of a scenario of two gas components, a diffuse (1.3″ = 50 ckpc), low-excitation component and a denser (1″ = 40 ckpc), high-excitation component, where the diffuse gas has properties similar to those of normal high-z galaxies while the dense gas components has properties more similar to those of high-z FIR luminous quasars. However, as described in Riechers et al. (2010), the size of the CO emission was observed to be overall quite compact, 2.3 pkpc. Our MUSE observations show an even more diffuse HI gas, revealed by the Lyα emission, that extends up to 150 ckpc, embedding or blending with the HI gas of LBG-3. The zELDA modeling we performed on the Lyα profile informs about the presence of HI column densities up to 3 × 10²⁰ atoms cm⁻² above LBG-3 (Sect. 5).

The [C II] emission toward AzTEC-3 was also observed to be compact and its peak position to be consistent with that of CO (Riechers et al. 2014). The fact that the CO and [C II] peak emissions are spatially consistent implies that [C II] is tracing the position of the star-forming regions and the location of the molecular gas clouds rather than other gas phase regions of the SMG. The star-forming regions of AzTEC-3 would then contain some metals, unlike LBG-3 that lacks [C II] detection probably due to its low metallicity. The Lyα emission is much more extended than the star-forming regions traced by [C II], probably revealing the shape and distribution of the HI gas in the system, elongated above LBG-3 and extending southern than the SMG. The elongated shape could be the result of the interaction between AzTEC-3 and LBG-3.

At the compact position of the [C II] emission, F(Lyα) = (1.9 ± 0.7)×10⁻¹⁸ erg s⁻¹ cm⁻², rest-frame EW(Lyα) = 6.7 ± 1.6 Å, and L(Lyα) = (1.4 ± 0.5)×10⁸ L_⊙, giving a ratio of L(Lyα)/L([C II]) = (2.1 ± 0.7)×10⁻² (see also Riechers et al. 2020). Extracting the MUSE spectrum within a 0.7″-aperture on top of LBG-3, we estimate a F(Lyα) = (3.8 ± 0.6)×10⁻¹⁸ erg s⁻¹ cm⁻², a rest-frame EW(Lyα) = 9.5 ± 1.7 Å, and L(Lyα) = (2.8 ± 0.4)×10⁸ L_⊙. At this position no significant [C II] emission was detected and L([C II]) < 0.17 × 10⁹ L_⊙ (Riechers et al. 2014). So, on top of LBG-3, L(Lyα)/L([C II]) > 0.6.

We can now investigate the L(Lyα)/L([C II]) ratio in terms of its relation to star formation. A Lyα luminosity lower than the theoretical value and so a lower L(Lyα)/L([C II]) ratio can be usually explained by the presence of dust. We refer to the SFR versus L([C II]) empirical relation obtained by De Looze et al. (2014) to estimate the L(Lyα)/L([C II]) expected value for different kind of high-redshift star forming galaxies. We consider their Eq. (3) to relate the SFR and the luminosity of a SFR-sensitive emission line. Also, we assume that SFR is the total UV plus FIR star-formation rate and that it is equal to the SFR(Lyα) in the case all the emitted Lyα photons escape the galaxy (Fig. 18). We take the SFR(Lyα) expression from Kennicutt (1998), for which SFR = 10 M_⊙ yr⁻¹ implies an intrinsic L(Lyα) = 10⁴³ erg s⁻¹. By assuming the L([C II]) vs SFR relation of local starburst and high-z star-forming galaxies (Eqs. (7), (8), (11), (14), and (17) in De Looze et al. 2014), L(Lyα)/L([C II]) ∼ 0.3 − 0.7 at a SFR(UV+FIR) of about 1200 M_⊙ yr⁻¹ (yellow vertical line in Fig. 18). The L(Lyα)/L([C II]) theoretical ratio could decrease in the case of subsolar metallicity. A dust extinction corresponding to A_V = 0.5 could explain the L(Lyα)/L([C II]) ratio calculated for the spectrum extracted on top of the SMG, while the theoretical value is comparable with the ratio we calculate for the spectrum extracted on top of LBG-3.

Fig. 18. Left panel: total SFR vs [C II] luminosity from the empirical relations of De Looze et al. (2014). The red circle corresponds to the total SFR and L([C II]) of AzTEC-3. Middle panel: theoretical value of L(Lyα)/L([C II]) versus total SFR obtained assuming that SFR(Lyα) is equal to the total SFR and SFR(Lyα) is calculated with the calibration from Kennicutt (1998). This calculation is intended to give an idea of the theoretical value of L(Lyα)/L([C II]) at the location where the [C II] emission is concentrated. The vertical yellow line correspond to the total SFR of AzTEC-3 for all the possible values of L(Lyα)/L([C II]). Right panel: observed L(Lyα)/L([C II]) versus rest-frame Lyα equivalent width as shown in Harikane et al. (2018). The average relation for all the galaxies in that work is shown as a red dashed line. The gray symbols are their data points for the sources with a Lyα detection, excluding the upper limits. The red and black symbols correspond to AzTEC-3 (red circle), LBG-3 (black facedown triangle), and LBG-1 (black square).

Harikane et al. (2018) showed an anticorrelation between L([C II]) and rest-frame EW(Lyα) for galaxies at 5 < z < 7 (see the right panel of Fig. 18). Based on the EW(Lyα) we calculate, either the L([C II]) is much stronger in AzTEC-3, implying that AzTEC-3 is a source more metal rich than the galaxies studied in that work, or Lyα is more suppressed due to the higher dust content in AzTEC-3 (their Figs. 18–20) which is more likely the case given the nature of AzTEC-3. Also, a log(L([C II])/SFR_FIR) = 6.8 is lower than the value they calculated for a galaxy with a rest-frame EW(Lyα) lower than 30 Å. This is not surprising given the fact that the L([C II])/L_FIR ratio suggested the presence of a stronger-than-average, not normal, far-UV radiation field (Riechers et al. 2014, 2020).

Riechers et al. (2014) showed that the [C II] emission is compact over the entire velocity range, but also showed a blue low-significance wing toward LBG-3, that could correspond to either a tidal feature or a [C II] outflow. A blueshift of the OH163 μm doublet also supported this hypothesis. The presence of an outflow is plausible due to the intense AzTEC-3 star-formation rate. In fact, the SFR value and the size of the star-formation region as given by the [C II] emission determine a star-formation rate per unit of area larger than the limit expected for supporting starburt-driven outflows (Heckman 2001; Riechers et al. 2020). Modeling the Lyα profile, we found best fit models that are consistent with the presence of outflows from the UV star-forming regions of the SMG and the presence of high N HI where the Lyα emission is brighter, where, therefore, HI scattering could be efficient. This suggests that the gas kinematics around AzTEC-3 are the result of its ongoing starburst, and may be further shaped by its possible interaction with LBG-3.

The only other galaxy in the AzTEC-3 neighborhood with a significant detection of [C II] is LBG-1. Riechers et al. (2014) found that the [C II] emission is spatially resolved and covers all the three optical knots. The infrared SED fitting gave a dust mass less than 9 × 10⁷ M_⊙, which implies a dust extinction much lower (Capak et al. 2011) than in the case of AzTEC-3 and a SFR_FIR < 54 M_⊙ yr⁻¹, which makes LBG-1 consistent with being a main sequence galaxy at z = 5, composed of a young starburst and an underlying older population. At the compact position of the [C II] emission, F(Lyα) = (1.3 ± 0.8)×10⁻¹⁸ erg s⁻¹ cm⁻², rest-frame EW(Lyα) = 26 ± 11 Å (assuming a continuum flux density of 1 × 10⁻²⁰ erg s⁻¹ cm⁻² Å⁻¹), and L(Lyα) = (1.0 ± 0.6)×10⁸ L_⊙, giving a ratio of L(Lyα)/L([C II]) = 0.06 ± 0.04 (Pavesi et al. 2019), that is consistent with the theoretical value by assuming a dust reddening corresponding to A_V = 0.2 (see Table 2) and extinction law from Calzetti et al. (2000).

In addition to LBG-1, we detected eight other Lyα emitting galaxies in the AzTEC-3 field. They tend to be located in an elongated configuration (see Fig. 3). Some other objects with photometric redshift consistent with that of the SMG, but without a Lyα emission in our MUSE data, are also distributed around AzTEC-3 (Fig. 6). They could lack of Lyα emission due to a unfavorable amount and distribution of dust toward the line of sight, but they could also be lacking the HI gas reservoir that would shine in Lyα. It could have been consumed by star formation or by tidal stripping within the AzTEC-3 environment. Instead, the elongated distribution of the Lyα emitting galaxies could follow the elongated distribution of HI gas falling toward the SMG. More sensitive data would be needed to confirm this hypothesis.

6.2. Evidence of merger phenomena in the AzTEC-3 environment

There is evidence in the literature that supports the idea that compact SMGs could be the result of major mergers (e.g., Tacconi et al. 2006; Swinbank et al. 2008; Engel et al. 2010; Simpson et al. 2014). In the protocluster of our study, the high molecular gas mass (roughly 5 times the stellar mass) of AzTEC-3 and its SFR_FIR equal to 1100 M_⊙ yr⁻¹ (Riechers et al. 2014) suggest the presence of a heavily obscured starburst, possibly triggered by a major merger. The richness of the gas reservoir in the AzTEC-3 system (both dense and diffuse molecular gas, and diffuse atomic gas revealed by Lyα) supports the idea of a gas-rich merger.

The FIR luminosity surface density of AzTEC-3 was estimated to be 5 × 10¹² L_⊙ kpc² (Riechers et al. 2014) and it is at the limit for a starburst to be supported by radiation pressure. The [C II] to FIR luminosity ratio (L([C II])/L_FIR) and the no-detection in X rays (Riechers et al. 2020) suggested the presence of a strong radiation field, but gives no direct indication for the presence of an obscured AGN (Riechers et al. 2014). The merger of the SMG progenitors could have triggered the starburst we currently see. Also, the observed Lyα luminosity supports the idea that the Lyα photons are mainly produced by recombination of atoms excited by a starburst radiation field (see Sect. 6).

Based on the kinematics of our observations of Lyα emission, a fast outflow can depart from the starburst and extend to the south. We also detected a bridge of high column density gas between AzTEC-3 and LBG-3 that can be the result of the merger event between AzTEC-3 and LBG-3. The elongated Lyα emission north of LBG-3 suggests tidal interaction with the SMG gravitational potential. The gas could have been stripped and could have assumed a tidal structure as a consequence of a first passage between LBG-3 and AzTEC-3. The structure seen in the region to the south of the SMG suggests interaction between the SMG and the mosaic_199 source, a third phenomenon of interaction (Fig. 8) around AzTEC-3.

In addition to this, in the field there is LBG-1 that itself could be a merging system. The Lyα observations and modeling suggest the presence of a gas with expansion velocities of 50 up to 200 km s⁻¹ in the southern region of LBG-1 that could follow the merging rotation and an N HI overall on the order of 10²¹ atoms cm⁻² that could make a gas-rich merger. We do not have enough resolution to provide a more detailed description of the LBG-1 merger.

However, as proposed by Riechers et al. (2014), this system could be in an early stage of merger. The final stage could drive to a more intense star-formation rate than the one currently measured, but probably to a much fainter Lyα emission absorbed by the dust produced in the star formation merger-driven event (Yajima et al. 2013).

Other systems in the literature at z ∼ 6, such as CR7 (Matthee et al. 2020b) and Himiko (Ouchi et al. 2013) share properties with LBG-1. For both sources from the literature, the Lyα emissions were found to be elongated along the direction connecting the multiple components and the Lyα peaks were seen offset with respect to the brightest UV emissions. The offset could be related to a merger event (see also Jiang et al. 2013) and the elongation of the Lyα emission along the axis of the multiple components could trace the underlying gas distribution (Matthee et al. 2020b), which seems distorted toward the southwest in the case of LBG-1. Riechers et al. (2014) measured a L([C II])/L_FIR ratio of 3 − 9 × 10⁻³, consistent with the value for normal star-forming galaxies (see also Pavesi et al. 2019), and a L([C II])/L(CO) > 4600 indicative of a moderate radiation field. Also, the FWHM and the EW(Lyα) of LBG-1 are consistent with the values of normal star-forming galaxies rather than AGN.

6.3. Study of AGN activity in AzTEC-3

Our MUSE observations do not provide strong constrains on the AGN activity in AzTEC-3. However, by comparing the Lyα emission we detected with observations in the literature that support AGN activity, we found differences that point more to a merger-driven starburst rather than to an influential AGN.

Borisova et al. (2016) and den Brok et al. (2020) showed a large sample of Lyα nebulae around typeI and typeII AGN at z = 3, detected trough the CubEx software. Despite the surface brightness dimming, the radii of those nebulae were observed to be larger than 450 ckpc (more than 110 pkpc) and between 220 and 450 ckpc (50−100 pkpc and up to 300 pkpc with 480 pkpc for the Slug nebula), respectively, much larger than in our case. Also, the integrated Lyα luminosities are 1 order of magnitude brighter than the value of our entire system, even in the cases of obscured tori. Also, their 1D spectra show Lyα peaks that are redshifted more than 1000 km s⁻¹. The FWHM and the EW of the Lyα emission line we measured, even for the spectrum extracted just on top of the SMG, are consistent with the values measured for star-forming galaxies rather than AGNs (Henry et al. 2015; Matthee et al. 2020b). However, the luminosity values we measured can still be dominated by the effect of the dust of the SMG. In fact, correcting the Lyα flux by the extinction of A_V = 0.8 estimated for the starburst, the L(Lyα) emitted at the compact position of the SMG becomes comparable to the faintest sources studied in Borisova et al. (2016).

Emissions of Lyα, comparable in size to that of the AzTEC-3+LBG-3 system, were found around two star-forming galaxies at z = 6.5 and 6.6 (VR7 and CR7 Matthee et al. 2020a,b). However, their Lyα luminosities are more than 5 times brighter than in our system, implying that the SMG dust could allow the escape of fewer Lyα photons, even if the N HI conditions in the CGM could be similar and could produce an equally efficient scattering. For CR7, the AGN contribution was disfavored given the UV to IR luminosity ratio (Matthee et al. 2020b). In the case of AzTEC-3, Riechers et al. (2014) pointed out that a luminous AGN component was also disfavored by the non detection of the high excitation CO(16−15), even though not detecting this transition is not a hard constraint. Algera et al. (2021) found that, based on its radio emission, AzTEC-3 does not show a sign of AGN activity either and the MUSE spectrum does not present typical AGN features. Therefore, the data so far are not in favor of a scenario in which an AGN is powering the emission of AzTEC-3, even if a very obscured AGN, also a possible result of the major merger event, could still be present.

There are examples in the literature where other mechanisms of the production of the radiation field were proposed and that showed different properties in the Lyα emission in comparison to the AzTEC-3 system. Daddi et al. (2021) found evidence of gas accretion toward a massive system also containing an SMG at z = 2.91. The phenomenon of gas accretion was supported by a blue shifted component of the Lyα emission line. However, the Lyα emission they observed peaks in an empty region located at the center of the halo potential well and it extends over 1000 ckpc (300 pkpc), much larger than what we detect.

The SSA22 protocluster is an overdensity with more than one extended Lyα emitting source. The overdensity was detected as an excess of LBGs (Steidel et al. 2000) and more than 200 Lyα emitters (Hayashino et al. 2004), more than 30 Lyα nebulae (Matsuda et al. 2004), and more than 50 SMGs (Umehata et al. 2014) were discovered in the entire area. Both the Lyα emitters and the SMGs trace the densest peak of the protocluster. The extended Lyα sources include two giant Lyα blobs with sizes larger than 100 pkpc (Steidel et al. 2000). They were discovered to be located in the intersection of the three filaments, traced by the Lyα emitters, that characterize the protocluster. In agreement with the theory that galaxy formation at high redshift occurs along large-scale filamentary overdense regions, the intersection of filaments could evolve in low-redshift clusters and the giant Lyα blobs could be revealing cluster progenitors. The Lyα luminosity of the two giant Lyα blobs on the order of 10⁴⁴ erg s⁻¹ (about 20 times brighter than the Lyα emission associated to the AzTEC-3+LBG-3 system) and the association of one of the two to an SMG (LAB1, Chapman et al. 2001) suggested they may be progenitors of very massive galaxies near the center of a massive cluster (Matsuda et al. 2005) at z = 0. The star-formation rate estimated for the SMG associated to LAB1 is ∼1000 M_⊙ yr⁻¹ and the CO emission line implies a large amount of molecular gas (Chapman et al. 2004), comparable to the values of AzTEC-3. However, cooling flow could be a possible scenario for the Lyα emission (Chapman et al. 2001) also in SSAA2, due to the lack of an optical counterpart in LAB1, lack of both an optical and a submillimeter source in the other blob, and for the large and bright Lyα emission, unlike what we observe in the AzTEC-3 system.

6.4. Estimation of the mass of the AzTEC-3 protocluster and its fate at lower redshift

We found that six of the Lyα emitting sources detected in the AzTEC-3 environment, including AzTEC-3, LBG-3, and LBG-1, have a counterpart in the COSMOS2015 catalog and so we have an estimation of their stellar masses (from 5 × 10⁸ to 2 × 10¹⁰ M_⊙). The LBGs in the field studied in Capak et al. (2011) have stellar masses between 6.3 × 10⁷ and 2.5 × 10⁹ M_⊙. By summing all these mass values, we obtain a minimum value of the stellar mass in the protocluster of (5.8 ± 2.3)×10¹⁰ M_⊙. If we assume that the COSMOS2015 sources shown in Fig. 6 also belong to the AzTEC-3 protocluster, the minimum value of the stellar mass in the protocluster is (8.1 ± 2.4)×10¹⁰ M_⊙.

An estimation of the dark-matter halo mass of the protocluster can be obtained by assuming the relation between stellar mass and halo mass for the clusters studied in van der Burg et al. (2014). For a stellar mass of 5.8 × 10¹⁰ − 8.1 × 10¹⁰ M_⊙, we can calculate a minimum dark-matter halo mass of 4 × 10¹¹ − 8 × 10¹¹ M_⊙, which is consistent with the lower limit of dark-matter halo mass estimated in Capak et al. (2011). It is worth noting that the protocluster projected area can be as big as a 2 cMpc-radius circle (Capak et al. 2011) larger than the area probed by our MUSE observations. However, the stellar mass could be concentrated within 1 cMpc, since the center of mass of the protocluster is close to the location of AzTEC-3 and the two most massive sources of the field are about 600 ckpc apart. The relation from van der Burg et al. (2014) was obtained for clusters at z = 1 and there could be evolution between the properties of z ∼ 5 protoclusters and z = 1 clusters. Cen & Zheng (2013) provided an approximate equation to relate the observed Lyα luminosity of a Lyα nebula and the dark matter halo mass of its central galaxies. By following their Eqs. (2) and (4), we can give an estimation of the dark matter halo mass in which the AzTEC-3+LBG-3 Lyα nebula is located. The total Lyα luminosity of the AzTEC-3+LBG-3 system is (5 ± 1)×10⁴² erg s⁻¹. This luminosity implies a dark matter halo mass of (1.84 ± 0.04)×10¹² M_⊙ and a size of the nebula on the order of 8 arcsec² rescaled to z = 5.3, which is consistent with what we observe. Considering the simulation in Chiang et al. (2013), a protocluster with a halo mass of 10¹² M_⊙ could evolve into a cluster of 2 × 10¹³ M_⊙ by z = 2 and into a Fornax-type cluster at z = 0 with a typical mass of 2 × 10¹⁴ M_⊙. Chiang et al. (2013) also showed that z > 5 protoclusters with 10¹² M_⊙ halo masses span a region in the sky within an effective radius of about 5 cMpc. Therefore, we could be observing just the central, most massive part of the entire overdensity.

Submillimeter galaxies are thought to be high-z progenitors of present-day ellipticals that could be located in the core of present-day galaxy clusters. Swinbank et al. (2008) showed that SMGs at z = 2 could reside in a similar mass halo as SMGs as z > 4, but could be characterized by twice their stellar mass. To achieve that stellar mass, major mergers are a plausible explanation (Swinbank et al. 2008; Engel et al. 2010; Simpson et al. 2014; Dudzevičiūtė et al. 2020; Stach et al. 2021) and we have evidence of them in the protocluster around AzTEC-3. Unlike some other overdensites of galaxies at z > 4 from the literature, the AzTEC-3 environment contains only one submillimeter galaxy. As an example, the SPT 2349–56 protocluster contains 14 submillimeter galaxies at z = 4.3 and it is expected to evolve into a very massive cluster of more than 10¹⁵ M_⊙ at z = 0 (Miller et al. 2018). Also, the overdensity discovered by Oteo et al. (2018), containing at least ten dusty star-forming galaxies at z ≃ 4, is expected to evolve in a cluster as massive as the Coma cluster at z = 0.

7. Conclusions

In this work, we have analyzed the MUSE data of the environment around the AzTEC-3 submillimeter galaxy. We have made use of the CubExtractor (Cantalupo et al. 2019) software to reduce and analyze the data, as shown in Sects. 2 and 3. We found ten Lyα emitting sources, including an extended emission toward AzTEC-3 and LBG-3 with a total Lyα luminosity of (5 ± 1)×10⁴² erg s⁻¹ and another extended emission embedding the three components of LBG-1. The sources appear distributed in an elongated configuration of about 70″ in extent. Five of the sources have a counterpart in the COSMOS2015 catalog. The presence of the ten Lyα emitting sources around AzTEC-3 confirms that the region is an overdensity at z ∼ 5.3 (Capak et al. 2011, and Sect. 5). We calculated the center of mass of the protocluster which is close to the AzTEC-3+LBG-3 system, the most massive of the protocluster.

We studied the spectroscopic properties of the Lyα emission in the AzTEC-3 environment in detail and we compared with the zELDA radiative transfer model (Gurung-Lopez et al. 2022). We note that Lyα photons are emitted toward the southeast of the SMG, probably due to dust obscuration to the west. The Lyα emission of the SMG is blended with that of LBG-3 and shows a bridge between the two galaxies, which could reveal the gas distribution due to their interaction. The Lyα emission is elongated to the north of LBG-3 resembling a tidal feature due to the interaction with AzTEC-3. An elongated feature is also seen toward the south of the SMG that could be produced by the interaction with the mosaic_199 source. The presence of an outflow departing from the SMG is supported by the size of the star-forming region, the star-formation rate estimated from the [C II], and FIR observations (Riechers et al. 2014, 2020). It is plausible that this outflow cleared channels for the escape of Lyα photons and that the Lyα photons are produced by recombination of HI atoms based on the energetics of the Lyα emission (Cantalupo 2017, and Sect. 4). The HI scattering can explain the Lyα luminosity we observe at more than 90 ckpc. Also, the modeling of the Lyα line did not require any contribution from an AGN.

Based on the resolution of our MUSE observation, we found that the Lyα emission comes from the three components of LBG-1. This could be related to the low dust content in this galaxy. However, the emission is not symmetrical and could indicate the interaction of the three components. The Lyα emission line profile shows a main peak and an extended tail which is consistent with models of galaxies characterized by gas in a wide range of velocities up to 200 km s⁻¹. Overall the Lyα emission is consistent with models of galaxies characterized by N HI on the order of 10²¹ atoms cm⁻² that could indicate that the interaction of the three components is a gas-rich merger. The emission is redshifted with respect to the [C II] systemic redshift and the Lyα emission line is broader than [C II] in the 1D spectrum. This is not unusual for high-z Lyα emitters and could be related to the variety of gas kinematics and column densities that condition the escape of Lyα photons and instead do not affect the escape of [C II] photons. The radiative transfer fit of the Lyα of two other star-forming galaxies indicate that they are galaxies with HI column densities up to 10²⁰ atoms cm⁻² and HI expansion velocities on the order of a few tens of km s⁻¹.

Given the availability of CO and [C II] observations from previous campaigns (Capak et al. 2011; Riechers et al. 2010, 2014, 2020; Pavesi et al. 2018a, 2019) and the Lyα information from our MUSE dataset, we can discuss weather the environment can play a role in the stage of evolution of AzTEC-3. We have evidence of starburst-driven phenomena and interactions around AzTEC-3 that could support the idea that the gravitational potential of the SMG is accreting nearby galaxies (like LBG-3 and possibly mosaic_199) and accumulating gas. Due to the gravitational interactions, the submillimeter galaxy could increase its mass significantly by z = 2. With LBG-1, we could be assisting to another merging event in the AzTEC-3 protocluster, and the two galaxies could all merge by z = 0 (Riechers et al. 2020). We estimated that the dark matter halo mass of the protocluster is on the order of 10¹² M_⊙. This value is lower than that of other protoclusters studied in the literature that contain more than one submillimeter galaxy (e.g., Miller et al. 2018). However, a dark matter halo of 10¹² M_⊙ could evolve into a cluster of 2 × 10¹³ M_⊙ by z = 2 and into a Fornax-type cluster at z = 0 with a typical mass of 2 × 10¹⁴ M_⊙ (Chiang et al. 2013).

Acknowledgments

We thank Franz Bauer, Lodovico Coccato, Yara Jaffe, Evelyne Johnston, Sam Kim, Michael Maseda, Ian Smail, Frederic Vogt for useful discussions. L.G. is grateful to ETH for the use of computer facilities and to the CAS-CONICYT, CAS18016, fellowship. S.C. gratefully acknowledges support from Swiss National Science Foundation grant PP00P2_190092 and from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme grant agreement No 864361. H.D. acknowledges financial support from the Agencia Estatal de Investigación del Ministerio de Ciencia e Innovación (AEI-MCINN) under grant (La evolución de los cúmulos de galaxias desde el amanecer hasta el mediodía cósmico) with reference (PID2019-105776GB-I00/DOI:10.13039/501100011033) and from the ACIISI, Consejería de Economía, Conocimiento y Empleo del Gobierno de Canarias and the European Regional Development Fund (ERDF) under grant with reference PROID2020010107.

References

Appendix A: Protocluster members with Lyα in emission

We show here the surface brightness profiles of the protocluster candidates we have identified through their Lyα emission. Fig. A.1 to Fig. A.8 show the surface brightness of the detections in the form of optimized S/N narrow-band images for the sources listed in Table 2.

Fig. A.1. Optimally extracted narrow-band images of the Lyα emission mosaic_1548 (left panel) and contours in surface brightness units. The two white contours correspond to 3 and 4σ above the background, based on the S/N map provided by CubEx. In this case they correspond to 0.7 × 10−18 and 1.2 × 10−18 erg sec−1 cm−2 arcsec−2. Stamps of the source in the HST F606W and F160W images are shown in the middle and right panels, together with the narrow-band surface brightness contours. The source within the contours of the right panel is COSMOS2015_841844 and it has a 0.4″ and a 1″ axes in the F160W image. The size of the pictures are adapted to contain exactly the entire emission.

Fig. A.2. Same as Fig. A.1, but for the mosaic_1520 source. The three white contours correspond to 0.5 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2. The source inside the contours of the right panel is COSMOS2015_852474 and it has a 0.25″ radius in the F160W image.

Fig. A.3. Same as Fig. A.1, but for the mosaic_770 source. The two white contours correspond to 0.8 × 10−18 and 1.1 × 10−18 erg sec−1 cm−2 arcsec−2.

Fig. A.4. Same as Fig. A.1, but for the mosaic_1035 source. The two white contours correspond to 0.6 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2.

Fig. A.5. Same as Fig. A.1 for the P3_547 source. The two white contours correspond to 0.5 × 10−18 and 1.0 × 10−18 erg sec−1 cm−2 arcsec−2.

Fig. A.6. Same as Fig. A.1 for the P2_446 source. A source is visible in the HST F160W image in the center of the Lyα emission, but does not have a counterpart in the COSMOS2015 catalog. The two white contours correspond to 0.5 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2.

Fig. A.7. Same as Fig. A.1, but for the mosaic_414 source. A source is visible in the HST F160W image in the center of the Lyα emission, but does not have a counterpart in the COSMOS2015 catalog. The two white contours correspond to 0.5 × 10−18 and 0.7 × 10−18 erg sec−1 cm−2 arcsec−2.

Fig. A.8. Same as Fig. A.1, but for the mosaic_199 source. The two white contours correspond to 0.5 × 10−18 and 0.7 × 10−18 erg sec−1 cm−2 arcsec−2. A source is visible in the F160W image which is COSMOS2015_848724. Some low SN emission could be blended with the AzTEC-3+LBG-3 system (yellow contours).

All Tables

All Figures

Fig. 1. Left panel: mosaic of the entire area observed with MUSE, obtained collapsing the cube information in the 7600−7700 Å wavelength range (“mosaic”), which corresponds to the redshifted Lyα at z ≃ 5.3. We identified six regions characterized by similar background level, the center of the four pointings (P1, P2, P3, P4), the central vertical and horizontal (black rectangles) stripes of overlapping regions. The white square corresponds to the area zoomed in the right panel. Right panel: zoom of the region containing AzTEC-3 (upper red circle) and LBG-1 (lower red circle) in the HST F160W image. We show the position of LBG-3 as black square, the position of LBG-2 (1447523 in Capak et al. 2011 and COSMOS2015_849887 in Laigle et al. 2016) as blue circle, and the position of COSMOS2015_848724 as black circle. These sources are all at z ∼ 5.3. The yellow circle indicates the position of a knot of star formation which can also be part of AzTEC-3. It is worth noting that LBG-1 is located in the center of the mosaic, in the lower corner of the 20 h overlapping region.

In the text

Fig. 2. Detection fidelity as a function of signal-to-noise ratio. The fidelity is expressed as 1 − Nneg/Npos, where Nnegis the numbers of the sources detected in the datacubes multiplied by −1 and Npos in the number of the detections in the original datacubes, when we apply the list of detection criteria described in the text. The red stars and line correspond to the detections in the “mosaic”, while the other lines correspond to the individual pointings, magenta for P1, blue for P2, cyan for P3, and green for P4. The signal-to-noise ratio in the x-axis correspond to the ratio between the isophotal flux and its error as provided by CubEx. The shapes of the lines reflect the uncertainty in the fidelity measurement.

In the text

Fig. 3. RGB image composed of the HST images in F814W, F125W, and F160W filters covering the area of our MUSE observations. The overlaid contours represent the 3σ levels of the Lyα emission. Red contours correspond to the sources with counterparts in the COSMOS2015 catalog. Green contours correspond to the Lyα detections without counterparts. The inserts contain the normalized Lyα profiles (normalized flux density versus velocity) of the Lyα detections indicated in the title of the inserts. The wavelengths of the maximum of the line detected by CubEx is indicated with a vertical dashed blue line. The blue shaded areas correspond to the spectral pixel ranges of the lines detected by CubEx. The zero velocity corresponds to the systemic redshift of the [C II] detection of AzTEC-3 and it is indicated as a vertical red line in each insert.

In the text

Fig. 4. Left panel: 1D spectrum of the AzTEC-3+LBG-3 system in the Lyα wavelength range. The original-sampling spectrum is shown in blue with error bars, the one smoothed by 3 × 3 wavelength channels in black. The error spectrum comes from the rescaled variance cube. The magenta spectrum is the theoretical sky spectrum from Rousselot et al. (2000) in arbitrary units. Right panels: same as the left panel for LBG-1.

In the text

Fig. 5. HST filter transmission curves (ACS F606W in yellow, ACS F814W in cyan, WFC3 F105W in magenta) and representative spectra of star-forming galaxies at z ∼ 5.3 (Lyα absorber in red, Lyα emitter with a small, intermediate, and large EW(Lyα) in green, blue, black, respectively from Shapley et al. 2003). We corrected all the representative spectra for the intergalactic-medium absorption at z = 5.3 by assuming the formalism in Madau (1995), before using them.

In the text

Fig. 6. Location of the Lyα emitting sources detected in the AzTEC-3 environment. The ones with Lyα central wavelength larger (red) and smaller (blue) than that of the AzTEC-3 system are shown as small circles. The AzTEC-3 submillimeter and the LBG-3 galaxies are shown as yellow star and yellow diamond. Sources from the literature with photometric redshifts consistent with z ∼ 5.3 are also shown, five COSMOS2015 sources with 4.9 < zphot < 5.6 (cyan big circles), Lyman break galaxies listed in Capak et al. (2011), but without Lyα emission in our MUSE datacube (green squares), one CO detection from Pavesi et al. (2018a) at z = 5.3 (gray circle). The black cross is the center of coordinates of all the sources shown in the figure and the gray cross the center obtained without considering AzTEC-3. The blue is the center of coordinates only of the sources with stellar mass larger than 109 M⊙ and the red cross is the weighted mass barycenter of the Lyα emitting sources detected in the field. The green cross is the weighted mass barycenter of the sources with a stellar mass estimation but without considering AzTEC-3.

In the text

Fig. 7. Upper left: signal-to-noise optimally extracted narrow-band image (Borisova et al. 2016) of the continuum-subtracted Lyα emission toward the AzTEC-3+LBG-3 system, rescaled to surface brightness units, and smoothed using a 2-pixel Gaussian kernel. The narrow band is built considering the 13 wavelength channels of the detection. The 5 white contours reveal the 1.3 to 3.8 erg s−1 cm−2 arcsec−2 levels, which correspond to 3−10σ in this image. The black letters indicate the directions we refer to in the text (N = north, NW = northwest, S = south, SE = southeast, W = west of the SMG). The small black segment indicates the 1″ scale. We also show the continuum-subtracted Lyα surface brightness contours overplotted to the HST ACS F606W (upper right), F814W (lower left), and F160W (bottom right) images. The 0.5″-side square indicates the center of the F160W position of LBG-3. The PSF of our MUSE data is 0.7″, corresponding to about 4 kpc or 27 ckpc at z = 5.3. The eight red contours correspond to the [C II] emission (from 3σ to 10σ where 1σ = 2.38 × 10−4 Jy beam−1) at 301.671−302.142 GHz, corresponding to −230 to 230 km s−1 from the peak emission. The synthesized beam of the [C II] ALMA observations is 0.63″ × 0.56″ as reported in Riechers et al. (2014). The 3σ contour extends toward LBG-3.

In the text

Fig. 8. Lyα channel maps toward the AzTEC-3+LBG-3 system. Each panel corresponds to the indicated wavelength and velocities with respect to the systemic redshift inferred from the [C II] emission line. The white contours represent the isophotes at S/N = 4, 5, and 6. The red square corresponds to the F160W position of LBG-3 and the red circle to the location of the peak of the [C II] emission of AzTEC-3. The green contours in the fifth panel represent the contours of the continuum-subtracted Lyα surface brightness. The horizontal gray-color bar shows a signal-to-noise ratio from 3 to 5. We refer to Fig. 7 for the 1″ scale.

In the text

Fig. 9. Left panel: Lyα (blue) and [C II] (red) profiles of the AzTEC-3+LBG-3 system in velocity space and normalized to the maximum. The [C II] spectrum is taken from Riechers et al. (2014) and it is associated to AzTEC-3 (see Fig. 7). Right panel: normalized [C II] profile (red), Lyα profile of the emission extracted at the position of the main Lyα peak associated to the SMG (blue and black curves), extracted from the position of the rest-frame UV knots of AzTEC-3 as seen in the F160W image (green), and extracted at the position of the main Lyα peak associated to LBG-3 (cyan). The extraction is done in an aperture of 0.7″ radius for the blue, green, and cyan profiles and increasing the S/N of the connected pixels in CubEx for the black profile. For simplicity, we call the spectra as onlySMG Lyα, onlySMG UV knots, and onlyLBG. The normalization of these spectra is performed with the same factor used to normalize the spectrum of the AzTEC-3+LBG-3 system in the left panel to show the different intensities. The normalization of the [C II] profile is also performed accordingly.

In the text

Fig. 10. Upper left: signal-to-noise optimally extracted narrow-band image (Borisova et al. 2016) of the continuum-subtracted Lyα emission, rescaled to surface brightness units, and smoothed using a 2-pixel Gaussian kernel of LBG-1. The five white contours reveal the 0.8 to 2 × 10−18 erg s−1 cm−2 arcsec−2 levels, which correspond to 3 and 8σ in this image. The letters indicate reference positions described in the text. The small black segment indicates the 1″ scale. We also show the Lyα surface brightness contours overplotted to the F606W (upper right), F814W (lower left) ACS, and F160W WFC3 (lower right) HST images. The PSF of our MUSE data is 0.7″, corresponding to about 4 kpc or 27 ckpc at z = 5.3. The eight red contours correspond to the 3−10σ (1σ = 2.38 × 10−4 Jy beam−1) of the [C II] emission at 301.847−302.093 GHz, corresponding to −120 to 120 km s−1 from the peak emission. The synthesized beam size of the ALMA [C II] observations is 0.63″ × 0.56″ as reported in Riechers et al. (2014).

In the text

Fig. 11. Lyα channel map of LBG-1. Each panel corresponds to the indicated wavelength and velocities with respect to the systemic redshift inferred from the [C II] emission line. The white contours represent the isophotes at S/N = 4, 5, and 6. The red circle shows the F160W position of LBG-1 as in Fig. 1, drawn to drive the eye. The green contours in the third panel represent the contours of the continuum-subtracted Lyα surface brightness. The horizontal gray-color bar shows a signal-to-noise ratio from 3 to 5. We refer to Fig. 10 for the 1″ scale.

In the text

Fig. 12. Left panels: pseudo slits oriented horizontally on the Lyα emission peak (first row), tilted by 110° on the Lyα emission peak (second row), tilted by ten degrees below the Lyα emission peak (third row), and tilted by ten degrees above the Lyα emission peak (fourth row). Middle panels: position-velocity diagrams obtaining slicing the continuum-subtracted MUSE datacube with a pseudo slit oriented as shown in the left panels. In the left y-axis, we indicate the wavelength and in the right y-axis the velocity with respect to the systemic redshift inferred from the [C II] emission line, the x-axis corresponds to the position along the pseudo slit in degrees (from the east to the west, except in the second row where it is from the south to the north). The white contours correspond to the 1, 2, 3σ level. Right panels: 1D spectra corresponding to the position-velocity diagrams of the middle panels. Wavelength is on the y-axis in the same range as in the middle panels and the flux normalized to the maximum is on the x-axis. The color coding of these spectra is like in Fig. 13.

In the text

Fig. 13. Left panel: 1D profiles of the Lyα (blue) and [C II] (red) emissions toward LBG-1. The [C II] profile is taken from Riechers et al. (2014) and corresponds to the region outlined in Fig. 10 (red contours). Right panel: [C II] profile in red as in the left panel and Lyα profile obtained slicing the Lyα emission horizontally on top of LBG-1 (green), tilted by 110° on top of LBG-1 (cyan), tilted by ten degrees below LBG-1 (magenta), and tilted by ten degrees above LBG-1 (blue). The color coding of these spectra is like in Fig. 12. The two former spectra are the ones with higher signal to noise as can be seen by the smaller error bars. The thin black curve corresponds to the spectrum of the integrated Lyα emission as the blue curve of the left panel.

In the text

Fig. 14. Left panel: observed-frame spectrum at the Lyα wavelength of the AzTEC-3+LBG-3 system in the vacuum. The observed spectrum is shown as data points with error bars and the smoothed spectrum as a black curve like in Fig. 9. The best-fit zELDA model that corresponds to the 50th percentile is shown in green and the green shaded area contains the models within the 16th and 84th percentiles obtained fitting the spectrum at wavelength larger than the systemic redshift. In blue, we show the models obtained fitting the spectrum only at the wavelength larger than the maximum. The red curve corresponds to the best fit in the wavelength range 7630−7700 Å for the AzTEC-3+LBG-3 spectrum corrected for the IGM absorption, by using the prescription of Madau (1995). Right panel: best-fit zELDA models for the spectra extracted in a 0.7″ aperture located on the position of the main Lyα emission associated to AzTEC-3 (upper left), the position of the UV knots of the SMG (upper right), the position of the main Lyα emission associated to LBG-3 (lower left), and the bridge between SMG and LBG (lower right). Black dots with error bars are the observed-frame spectra as shown in Fig. 9, the black lines are the smoothed spectra, the green (blue) curves and shaded areas are the best-fit models and the models within the 16th and 84th percentile of the model parameter space obtained fitting the spectra at wavelength larger than the systemic redshift (larger than 7669 Å). Vertical blue dashed lines indicate the Lyα redshift given by the AzTEC-3 [C II] detection. The zELDA fits are performed fixing the redshift to the systemic inferred by the [C II] emission peak.

In the text

Fig. 15. Left panel: observed-frame spectrum at the Lyα wavelength of LBG-1. The observed spectrum in vacuum wavelengths is shown as black datapoints with error bar, the black curve is the smoothed spectrum. The green curve is the best-fit zELDA model that corresponds to percentile 50 and the green shaded area contains the models within percentiles 16 to 84. The blue curve corresponds to a zELDA fit at wavelengths larger than 7669 Å, obtained with the scope of finding a zELDA model for the extended tail at wavelengths larger than the main red peak. Right panels: zELDA fits of the spectra obtained collapsing the LBG-1 spectrum as in Fig. 13. In green, we show the best fit and the models within the 16th and 84th percentile obtained for wavelengths larger than the systemic redshift. The blue curves correspond to the best fit of the extended tail for the spectrum in the left panel. Vertical blue dashed lines indicate the Lyα redshift given by the LBG-1 [C II] detection, we fixed while performing the fits.

In the text

Fig. 16. Qualitative representation of the combination of the zELDA parameters that best fit the Lyα emission of the AzTEC-3+LBG-3 system (left panel) and of the LBG-1 region (right panel). The size of the dots in the cartoon is proportional to Vexp and the color scale corresponds to the N HI value in a way that fainter color indicates lower N HI. The smallest and darkest dots show Vexp < 30 km s−1 and N HI ∼ 1021 atoms cm−2. The largest dot corresponds to Vexp up to 800 km s−1, the faintest color to N HI ∼ 1019 atoms cm−2.

In the text

Fig. 17. Observed-frame spectrum at the Lyα wavelength of mosaic_1548 (left), mosaic_1520 (right). The observed spectrum in vacuum wavelengths is shown as black data points with error bars, the black curve is the smoothed spectrum. The green curve is the best-fit zELDA model of a redshifted Lyα main peak that corresponds to percentile 50 and the green shaded area contains the models within percentiles 16 to 84. The vertical blue dashed lines indicate the Lyα redshift given by the 50th percentile model. For the left hand side spectrum, we also show in magenta a zELDA fit of the two visible peaks at 7665 and 7672 Å. The systemic wavelength corresponding to this fit is at 7668.1 Å and the model implies a blue peak higher than the red peak as in the case of an inflow (see Table 2 for the best fit parameters).

In the text

Fig. 18. Left panel: total SFR vs [C II] luminosity from the empirical relations of De Looze et al. (2014). The red circle corresponds to the total SFR and L([C II]) of AzTEC-3. Middle panel: theoretical value of L(Lyα)/L([C II]) versus total SFR obtained assuming that SFR(Lyα) is equal to the total SFR and SFR(Lyα) is calculated with the calibration from Kennicutt (1998). This calculation is intended to give an idea of the theoretical value of L(Lyα)/L([C II]) at the location where the [C II] emission is concentrated. The vertical yellow line correspond to the total SFR of AzTEC-3 for all the possible values of L(Lyα)/L([C II]). Right panel: observed L(Lyα)/L([C II]) versus rest-frame Lyα equivalent width as shown in Harikane et al. (2018). The average relation for all the galaxies in that work is shown as a red dashed line. The gray symbols are their data points for the sources with a Lyα detection, excluding the upper limits. The red and black symbols correspond to AzTEC-3 (red circle), LBG-3 (black facedown triangle), and LBG-1 (black square).

In the text

Fig. A.1. Optimally extracted narrow-band images of the Lyα emission mosaic_1548 (left panel) and contours in surface brightness units. The two white contours correspond to 3 and 4σ above the background, based on the S/N map provided by CubEx. In this case they correspond to 0.7 × 10−18 and 1.2 × 10−18 erg sec−1 cm−2 arcsec−2. Stamps of the source in the HST F606W and F160W images are shown in the middle and right panels, together with the narrow-band surface brightness contours. The source within the contours of the right panel is COSMOS2015_841844 and it has a 0.4″ and a 1″ axes in the F160W image. The size of the pictures are adapted to contain exactly the entire emission.

In the text

	Fig. A.2. Same as Fig. A.1, but for the mosaic_1520 source. The three white contours correspond to 0.5 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2. The source inside the contours of the right panel is COSMOS2015_852474 and it has a 0.25″ radius in the F160W image.
In the text

	Fig. A.3. Same as Fig. A.1, but for the mosaic_770 source. The two white contours correspond to 0.8 × 10−18 and 1.1 × 10−18 erg sec−1 cm−2 arcsec−2.
In the text

	Fig. A.4. Same as Fig. A.1, but for the mosaic_1035 source. The two white contours correspond to 0.6 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2.
In the text

	Fig. A.5. Same as Fig. A.1 for the P3_547 source. The two white contours correspond to 0.5 × 10−18 and 1.0 × 10−18 erg sec−1 cm−2 arcsec−2.
In the text

	Fig. A.6. Same as Fig. A.1 for the P2_446 source. A source is visible in the HST F160W image in the center of the Lyα emission, but does not have a counterpart in the COSMOS2015 catalog. The two white contours correspond to 0.5 × 10−18 and 0.8 × 10−18 erg sec−1 cm−2 arcsec−2.
In the text

	Fig. A.7. Same as Fig. A.1, but for the mosaic_414 source. A source is visible in the HST F160W image in the center of the Lyα emission, but does not have a counterpart in the COSMOS2015 catalog. The two white contours correspond to 0.5 × 10−18 and 0.7 × 10−18 erg sec−1 cm−2 arcsec−2.
In the text

	Fig. A.8. Same as Fig. A.1, but for the mosaic_199 source. The two white contours correspond to 0.5 × 10−18 and 0.7 × 10−18 erg sec−1 cm−2 arcsec−2. A source is visible in the F160W image which is COSMOS2015_848724. Some low SN emission could be blended with the AzTEC-3+LBG-3 system (yellow contours).
In the text