© ESO, 2017

1. Introduction

For several decades it has been known that some galaxies have much lower surface brightnesses than others. Davies et al. (1994) demonstrated, using the ESO-Uppsala galaxy catalog (Lauberts & Valentijn 1989), how galaxy samples with fixed magnitude and size limits are biased by missing the low surface brightness galaxies due to limits in depth and sensitivity, resulting in a lower data quality. Historically, all galaxies with B-band central surface brightness of μ_0,B> 23 mag arcsec^-2 are considered as low surface brightness (LSB) galaxies (Impey & Bothun 1997). However, this definition is relatively broad and includes galaxies ranging from almost Milky Way sized galaxies to the smallest Milky Way satellite dwarf spheroidals (dSph). Early studies of LSB galaxies with detailed photometric measurements (Romanishin et al. 1983; Bothun et al. 1985, 1990; Sprayberry et al. 1995; Impey & Bothun 1989) concentrated mostly on relatively massive galaxies. These giant LSB galaxies, like Malin 1 (Impey & Bothun 1989), for which they measured a disk scale length of 55 kpc and a V-band central surface brightness of μ_0,V = 25.5 mag arcsec^-2, form an interesting class of objects in terms of galaxy evolution processes, as it is not well understood how they have managed to grow so massive without increasing their surface brightness. The appearance of such giant LSB galaxies has been proposed to be a result of their high angular momentum, and low star formation rate (Jimenez et al. 1998). These galaxies are predicted to be relatively rare and only appear in low density environments. In dense environments like the Fornax cluster, galaxies experience frequent tidal interactions with other galaxies and are affected by the cluster potential. This process, called harassment (Moore et al. 1998), tends to rip off material from the galaxies that enter deep into the cluster core, and eventually makes them denser (Smith et al. 2015).

In the end of 1980’s, large galaxy catalogs of the Virgo (Binggeli et al. 1985) and Fornax clusters (Ferguson 1989b) included also many dwarf sized (B-band absolute magnitude M_B> −18 mag¹) LSB galaxies, which greatly increased the number of known LSB galaxies. The automated photometry of Davies et al. (1988) allowed the detailed analysis of dwarf galaxies in these samples. Later studies, like the ones of Davies et al. (1990) or Sabatini et al. (2005), showed that dwarf LSB galaxies actually form the bulk of the population at the low luminosity end of the luminosity function. Most of these galaxies were found in clusters. In fact, galaxies with central V-band surface brightness of μ_0,V = 22−25 mag arcsec^-2 have been estimated to contribute significant amount (even 50%) of the light emitted by galaxies (Impey & Bothun 1997).

It is only recently that new instruments, such as OmegaCAM (Kuijken et al. 2002), Suprime-Cam (Miyazaki et al. 2002), DECam (Flaugher et al. 2015), and MegaCAM (Boulade et al. 1998), have made it possible to perform multi-band surveys with limiting V-band surface brightness down to μ_V ~ 28 mag arcsec^-2, over large areas in the sky. These new observations allow us to study also the distribution and properties of the dwarf LSB galaxies in statistically significant samples. Indeed, new imaging surveys like the Next Generation Virgo Survey (Ferrarese et al. 2012), have revealed a large number of LSB galaxies. Although the wide-field instruments attached to large telescopes have shown their efficiency performing deep surveys, other approaches have proven effective as well. One of them used the Dragonfly instrument (Abraham & van Dokkum 2014), which consists of a set of small 143 mm cameras, and was used when van Dokkum et al. (2015) discovered a large number of extended LSB galaxies in the Coma cluster, which they named ultra diffuse galaxies (UDG).

The UDGs discovered by van Dokkum et al. (2015) are defined to be extended (effective radius R_e>1.5 kpc) and faint (V-band absolute magnitude −16 mag<M_V<−13 mag), and have similar central surface brightnesses (μ_0,g>24 mag arcsec^-2) as the faintest dwarf galaxies. However, their effective radius can be even ten times larger. What makes these galaxies particularly interesting is that UDGs reside in the cluster environment where they appear in large numbers (Yagi et al. 2016; van der Burg et al. 2016, 2017; Wittmann et al. 2017). UDGs have now been found in all clusters where they have been searched for. For example, van der Burg et al. (2016) used an automated algorithm to find UDGs in clusters in the redshift range 0.044 <z< 0.063. They found that their abundance increases with increasing cluster halo mass, reaching ~200 UDGs in typical halo masses of M₂₀₀ ≈ 10¹⁵M_⊙. Recently UDGs have been reported also in some nearby galaxy groups (Merritt et al. 2016; Toloba et al. 2016; Crnojević et al. 2016; Román & Trujillo 2017b) and low density environments (Martínez-Delgado et al. 2016; Leisman et al. 2017) showing that these galaxies appear in all kind of galaxy environments.

In our study, we hereafter define all the galaxies with μ_0,r′> 23 mag arcsec^-2 as LSB galaxies, and the ones additionally having absolute r′-band magnitude M_r′> −19 mag as dwarf LSB galaxies. LSB galaxies that have an effective radius R_e> 1.5 kpc are called UDGs².

The formation mechanism of UDGs is still unclear. They have been suggested to form from medium mass (halo mass of 10¹⁰⁻¹¹M_⊙) galaxies as a result of strong gas outflows due to star formation feedback (Di Cintio et al. 2017), whereas Baushev (2016) suggested that UDGs can form via head-on collisions of gas-rich systems in the centers of galaxy clusters. UDGs have also been suggested to be the high spin tail of the typical dwarf elliptical (dE) galaxy population (Amorisco & Loeb 2016). Indeed, it is important to study the photometric properties of UDGs and LSB dwarf galaxies in different environments to see if there is a continuum between their properties.

The Fornax cluster, with a virial mass of 7 × 10¹³M_⊙ (Drinkwater et al. 2001a), is less massive than the Coma (1.4 × 10¹⁵M_⊙, Łokas & Mamon 2003) and Virgo clusters (1−3 × 10¹⁴M_⊙, McLaughlin 1999). However, in spite of its fairly low mass the Fornax cluster has a high fraction of early-type galaxies (E + S0 + dE + dS0)/all = 0.87 (Ferguson 1989a), and a central galaxy density similar to that of the more massive Virgo cluster. The core of the Fornax is also filled with hot X-ray emitting gas (Paolillo et al. 2002, and references therein), which makes the galaxies vulnerable to ram pressure stripping, which removes the cold gas from the galaxies. This implies that any galaxy which has spent a long time in the core of the Fornax cluster, should consist only of fairly old stellar populations.

So far only a few studies have mapped the LSB galaxy population in the Fornax cluster. Bothun et al. (1991) studied the properties of the LSB galaxies with central B-band surface brightnesses of μ_0,B> 23 mag arcsec^-2, and showed that there are tens of relatively metal-poor galaxies, which make a significant contribution to the faint end of the luminosity function. Due to their LSBs it is problematic to spectroscopically confirm their distances, and therefore often the cluster membership has been deduced from the clustering of these galaxies. However, the cluster membership of several LSB galaxies were spectroscopically confirmed in the sample of Drinkwater et al. (1999), which included galaxies with B-band total apparent magnitudes brighter than m_B< 19.7 mag (M_B< −11.7) and central surface brightnesses between 20 and 24 mag arcsec^-2. Also the study of Mieske et al. (2007), which used surface brightness fluctuation analysis to define galaxy distances, confirmed the cluster membership of several Fornax LSB galaxies. These studies support the idea that at least two thirds of the LSB galaxies in the area of Fornax are real cluster members. A recent study of Muñoz et al. (2015), performed with the DECam instrument, searched for new faint galaxies in the central parts of the Fornax cluster. Their observations reaching g′-band point sources down to 26.6 mag with S/N> 5, reveal more than hundred previously non-detected dwarf sized LSB galaxies. The faintest LSB dwarf galaxies in their sample have R_e ≈ 100 pc, which means that they have similar sizes as the Local Group dSph’s (McConnachie 2012, and the references therein).

In this paper, we perform a systematic search for LSB galaxies in the images of the Fornax Deep Survey (FDS), which is an ongoing survey using the VLT Survey Telescope (VST) at ESO/Cerro Paranal. It covers a larger field-of-view than any of the previous Fornax surveys with deep multi-band observations, and has similar depth as the Next Generation Virgo Survey (Ferrarese et al. 2012). The survey has already obtained several results, such as the discovery of an extended globular cluster population (D’Abrusco et al. 2016), the characterization of the extended stellar halo of NGC 1399 (Iodice et al. 2016), and the analysis of the merger system around NGC 1316 (Iodice et al. 2017). Since we have deep data in g′, r′ and i′-bands, we can determine also the colors of the galaxies.

We present a sample of LSB galaxies in the Fornax cluster, based on our analysis, and combined with those obtained in the previous studies. In Sects. 2 and 3 we describe the data used in this work, and briefly describe the reduction steps. In Sects. 4 and 5 we present the sample selection and the photometric measurements performed to obtain the structural parameters. We show the cluster-centric radial distributions (Sect. 6.1), orientations (Sect. 6.2) and colors (Sect. 7) of the galaxies. In Sect. 8 we discuss the results in the context of different formation theories of UDGs, and in Sect. 9 give the conclusions of this paper. For the Fornax cluster we use the distance of 19.95 Mpc (Tonry et al. 2001) corresponding to the distance modulus of 31.43 and scale of 0.0967 kpc arcsec^-2.

2. Data

We use the ongoing Fornax Deep Survey (FDS), which consists of the combined data of the Guaranteed Time Observation Surveys FOCUS (P.I. R. Peletier) and VEGAS (P.I. E. Iodice), dedicated to the Fornax cluster. Both surveys are performed with the ESO VLT Survey Telescope (VST), which is a 2.6-m diameter optical telescope located at Cerro Paranal, Chile (Schipani et al. 2012). The imaging is done with the OmegaCAM instrument (Kuijken et al. 2002), using the u′, g′, r′ and i′-bands, and 1° × 1° field of view. OmegaCAM consists of an array of 8 × 4 CCDs, each with 2144 × 4200 pixels. The pixel size is 0.21 arcsec and the average FWHM of the observations is ≈1 arcsec, so that the PSF is well sampled. For further information about the data see Iodice et al. (2016) and Peletier et al. (in prep.).

The observations used in this work were gathered in visitor mode runs during November 2013, 2014 and 2015 (ESO P92, P94 and P96, respectively). All the observations were performed in clear (photometric variations <10%) or photometric conditions. The observations in u′ and g′-bands were obtained in dark time, and those of the other bands in gray or dark time.

The observation area of FDS is divided into 1° × 1° fields (see Fig. 1) with some overlaps between the fields. The observations of each field were performed using offsets larger than 1 deg between consecutive exposures, always covering an adjacent field. During the sequence of multiple exposures, additional dithers of ≲10 arcmin width have been added to the 1 deg offsets with a fixed pattern for all fields. This ensures that the whole area has a uniform depth, meaning that whenever an exposure is dithered from the field center, and thus only partially covers the original field, there will be exposures in the adjacent fields with identical dithers providing a full coverage over all fields. This offset and dither strategy makes it possible to perform accurate sky subtraction without spending time for separate sky exposures (see Sect. 3.2 for details). The total exposure times in all fields are 11 000, 8000, 8000, and 5000 s in u′, g′, r′ and i′-bands, respectively. The total exposure times are divided into 150 s exposures meaning that each field is covered with a minimum of 75, 55, 55 and 35 exposures in u′, g′, r′ and i′, respectively.

The data used in this work cover an 4 deg² area centered on NGC 1399 (see Fig. 2). All the fields are imaged using the four bands. The limiting 5σ magnitudes (in AB system) for 1 arcsec² area are 27.6, 28.5, 28.5 and 27.1 mag in u′, g′, r′ and i′, respectively. In this work only g′, r′ and i′ bands are used, due to their deeper surface brightness limit and higher sensitivity for the relatively red galaxies in the Fornax cluster. The sources studied in this work have a central r′-band surface brightness of μ_0,r′ ≥ 23 mag arcsec^-2, so that most of them are not bright enough in the u′-band to detect them. In this work we analyze the fields 10, 11, 12 and 16 (see Fig. 1).

3. Data reduction

We have developed a pipeline to reduce FDS OmegaCAM data adapting the general OmegaCAM pipeline within the AstroWISE environment (McFarland et al. 2013). The pipeline used in this work is different from the one described in Iodice et al. (2016). However, the results of these two pipelines are consistent with each other (details will be given in Peletier et al., in prep.). The main reduction steps are provided in the next paragraphs.

3.1. Instrumental corrections

The instrumental corrections include removal of bias and correcting the sensitivity variations over the field of view (flat-field and illumination correction).

Bias was first defined from the overscan regions of the images, and the row-wise median values were subtracted from each science image. For not to add too much noise in the bias subtraction, we used the overscan corrected master bias image, obtained by median combining ten bias images taken each night.

After removing bias from the science images, they were corrected for the sensitivity variations over the focal plane by dividing the images by a master flat-field image. We adopted the flat-fielding method that was used by the Kilo Degree Survey (KiDS, de Jong et al. 2015): the master flat-field was achieved by first median-combining and normalizing eight dome flat-fields and eight twilight flat-fields, and then multiplying the averaged flat-fields with each other. High-frequency spatial Fourier modes are corrected using the dome flats and low frequencies using the twilight flats. This is based on the pre-assumption that the large-scale illumination of the twilight flat-field matches better the observational situation than that obtained from the dome. On the other hand, in the dome flats the S/N is high, which can be used to capture the pixel-by-pixel variations in the pixel sensitivity.

However, even after applying the flat-field correction small systematic flux variations remain across the instrument. These variations can be corrected by applying an illumination correction. We used the correction models made for the KiDS (see Verdoes Kleijn et al. 2013, for details). The models were made by mapping the photometric residuals across the CCD array using a set of dithered Landolt Selected Area (SA) field (Landolt 1992) observations, and fitting a linear model to the residuals. The images were multiplied with this illumination correction. The correction was applied after sky subtraction (see the next sub-section), to avoid the sky residuals being amplified by the illumination correction procedure.

3.2. Background subtraction and de-fringing

The images contain sky background flux composed of direct and scattered atmospheric emission, and scattered light of bright celestial sources. A careful removal of the atmospheric background light is essential when studying LSB objects such as UDGs. When considering a single image at low luminosity levels, we cannot directly tell which part of the light is diffuse light coming from the sources and which is background light. To bypass the problem, we can make an assumption that the pattern of background light stays constant if the telescope pointing direction is not changed by more than a few degrees. Due to the large dithers between the consecutive integrations the objects are not likely to appear twice in the same pixel, which allows us to produce background models by averaging a stack of images. The intensity of the sky changes thorough the night (1–10% between exposures), especially at the beginning and at the end of the night, which forces to scale the images used for the background model before combining them. The pattern of the scattered light changes also as a function of telescope pointing direction and the positions of the Sun and the Moon, so that any accurate model of those variations is not available.

A unique background model was made for each CCD of each exposure by scaling and stacking 12 consecutive dithered exposures (six before and six after the frame, whose background is modeled). First, SExtractor (Bertin & Arnouts 1996) was used to mask all stars and galaxies from all individual images (12 consecutive pointings). Specifically, we masked the objects with 5 pixels above the 5σ threshold, using a 50×50 pixel grid to estimate the background in the masking process. We ensured that we are not masking systematically any shapes (like vignetted edges) of the background by comparing a set of masks of background models. After the masking, the images were scaled with each other assuming that the shape of the background scales linearly with the total level of the background. The scaling factor s can be found by: $\mathit{s}\mathrm{=}\mathrm{Median}\left(\frac{{\mathit{m}}_{\mathrm{1}\mathit{,i}}}{{\mathit{m}}_{\mathrm{2}\mathit{,i}}}\right)\mathit{,}\hspace{0.17em}\mathit{i}\mathrm{=}\mathrm{1}\mathit{,}\mathit{...}\mathit{,}\mathrm{96}\mathit{,}$(1)where m_1,i is a set of medians measured within 96 90 × 90-pixel areas in image 1, used as a reference image (i.e., the image for which we are making the background model). Correspondingly, m_2,i is a set medians measured at the same locations in image 2, which we are scaling. Before combining the frames we exclude the images that are not suitable for obtaining the background model. We excluded frames, where more than 1/3 of the area was masked. Excluded were also frames which have a large scatter in $\frac{{\mathit{m}}_{\mathrm{1}\mathit{,i}}}{{\mathit{m}}_{\mathrm{2}\mathit{,i}}}$, since they have either large unmasked objects or have otherwise peculiar background. If only six or less frames were found to be useful for the background model (e.g., if an extended source fills the whole CCD being modeled, it is excluded), the scaling was done by using the median values of all 32 CCDs (instead of just one) and then using Eq. (1). After masking and multiplying the images with s, the selected frames were combined by taking the median of the stacks. This background model was then subtracted from the final image.

OmegaCAM has interference patterns (fringes) in i′-band images due to the internal reflections in the CCDs. Intensity of the fringe patterns is proportional to the total light coming to the CCD. As long as the background and the filter does not vary, the fringes have always the same shape since they are related to the properties of the CCD. Luckily, the fringes appear also in the background model and are subtracted with it. No other fringe correction was applied. The intensity of the fringe patterns in our images was lower than the masking threshold, which leaves them unmasked.

3.3. Weight maps

The pipeline also generates a weight frame for each exposure. The weight frame carries the information about the noise level, bad pixels, cosmic rays, satellite tracks, and saturated pixels in the image. The weight of a given pixel W_ij can be written as: ${\mathit{W}}_{\mathit{i,j}}\mathrm{=}\frac{\mathrm{1}}{{\mathit{\sigma }}^{\mathrm{2}}}\frac{{\mathit{F}}_{\mathit{ij}}}{{\mathit{I}}_{\mathit{ij}}}{\mathit{P}}_{\mathrm{hot}}{\mathit{P}}_{\mathrm{cold}}{\mathit{P}}_{\mathrm{saturated}}{\mathit{P}}_{\mathrm{cosmic}}{\mathit{P}}_{\mathrm{satellite}}\mathit{,}$(2)where σ² is the variance of the image measured from the raw frame, F is the flat-field pixel value, I is the illumination correction, and P_hot,P_cold,P_saturated,P_cosmic, and P_satellite, are the hot-, cold-, saturated-, cosmic ray-, and satellite-pixel maps, respectively (good → 1, bad → 0). The weight images were used for masking the unwanted pixels from the science images before they were stacked as a final mosaic.

Pixels are masked by giving a value of 0 to their fluxes, while for the other pixels a value of 1 is given. In hot-pixel maps the pixels which have high values in the bias images compared to the other pixels, are masked. In cold-pixel maps the pixels which deviate clearly (either high or low values) from the other pixels in the flat-field images are masked. Cosmic rays are detected from the images using the SExtractor cosmic ray detection algorithm (Bertin & Arnouts 1996). Satellite tracks are detected by first applying a Hough transform (Vandame 2001) to increase the linear patterns in the images: the lines consisting of more than 1000 pixels with intensity above the 5σ level relative to the background are then masked.

3.4. Astrometric and photometric calibrations

The first-order astrometric calibration was done by first matching the pixel coordinates to RA and Dec using the World Coordinate System (WCS) information from the fits header. Point source coordinates were then extracted using SExtractor and associated with the 2 Micron All Sky Survey Point Source Catalog (2MASS PSC, Skrutskie et al. 2006). The transformation was then extended by a second-order two-dimensional polynomial across the focal plane. SCAMP (Bertin 2006) was used for this purpose. The polynomial was fit iteratively five times, each time clipping the 2σ-outliers. The astrometric solution gives typically rms errors of 0.3 arcsec (compared to 2MASS PSC) for a single exposure, and 0.1 arcsec for the stacked final mosaic.

The absolute photometric calibration was performed by observing standard star fields each night and comparing their OmegaCAM magnitudes with the Sloan Digital Sky Survey Data Release 11 (SDSS DR11, Alam et al. 2015) catalog values. The OmegaCAM point source magnitudes were first corrected for the atmospheric extinction by subtracting a term kX, where X is airmass and k is the atmospheric extinction coefficient with the values of 0.182, 0.102 and 0.046 for g′, r′ and i′, respectively. The zero-point for a given CCD is the difference between the object’s corrected magnitude measured from a standard star field exposure and the catalog value. The zero-point for each CCD was kept constant for the whole night, only correcting for the varying airmass.

Fornax is poorly covered with stellar catalogs (in the optical) which could be used to check the accuracy of the photometric calibration. The American Association of Variable Star Observers Photometric All-Sky Survey catalog (APASS, Henden et al. 2012) is the only catalog with a large coverage over our observed Fornax fields. However, as the photometric errors of stars in this catalog with M_r′> 16 mag are as high as 0.05 mag, and because the photometric accuracy of our data is expected to be better than that, we did not use APASS for comparison. We made principal color analysis in a similar manner as was done for the Sloan Digital Sky Survey (SDSS) data by Ivezić et al. (2004). We did this test for the stacked 1° × 1° mosaics. We measured standard deviations of 0.035, 0.029, and 0.046 mag for the widths of the stellar locus principal colors s, w and x. The corresponding values for SDSS are 0.031, 0.025, and 0.042 mag. The offsets of stellar locii are −0.009, 0.003, and 0.009 mag for s, w and x, respectively. The typical rms scatter of the same offsets for SDSS are 0.007, 0.005, and 0.009 mag for s, w and x showing that our photometric accuracy is comparable to that of the SDSS. Zeropoint errors for SDSS are 0.01, 0.01, and 0.02 mag for g′, r′ and i′ (Ivezić et al. 2004). As the scatter in the stellar locii measured from our data is ≈10% larger than in SDSS, we estimate that our photometric errors are 0.02, 0.02, and 0.03 mag in g′, r′ and i′-bands, respectively.

3.5. Creating mosaics

After the astrometric and photometric calibrations, the images were sampled to 0.20 arcsec pixel size and combined using the SWarp software (Bertin 2010). Before combining the images cosmic rays and bad pixels were removed using the weight maps. Regardless of this removal of contaminated pixels, the resulting pixel distribution in the final mosaic were often non-Gaussian. To obtain better stability against outlier pixels, we decided to use median instead of mean when combining the mosaics. In order to achieve maximal depths in the images, the mosaics were combined from all the overlapping exposures. As a result the pipeline produces 1° × 1° mosaics with a 0.2 arcsec pixel resolution, and the corresponding weight images. The pixel values W_x,y for the final weight mosaic are obtained as (Kendall & Stuart 1977): ${\mathit{W}}_{\mathit{x,y}}\mathrm{=}\{\begin{array}{c}\\ \frac{\mathrm{2}}{\mathit{\pi }}{\left(\frac{{\sum }_{\mathit{k}}\sqrt{{\mathit{w}}_{\mathit{k}}}}{{\mathit{n}}_{\ne \mathrm{0}}}\right)}^{\mathrm{2}}\mathrm{(}{\mathit{n}}_{\ne \mathrm{0}}\mathrm{+}\frac{\mathit{\pi }}{\mathrm{2}}\mathrm{-}{\mathrm{1}}^{\mathrm{)}}\mathit{,}\hspace{0.17em}\mathrm{if}{\mathit{n}}_{\ne \mathrm{0}}\mathrm{iseven}\\ \frac{\mathrm{2}}{\mathit{\pi }}{\left(\frac{{\sum }_{\mathit{k}}\sqrt{{\mathit{w}}_{\mathit{k}}}}{{\mathit{n}}_{\ne \mathrm{0}}}\right)}^{\mathrm{2}}\mathrm{(}{\mathit{n}}_{\ne \mathrm{0}}\mathrm{+}\mathit{\pi }\mathrm{-}{\mathrm{2}}^{\mathrm{)}}\mathit{,}\hspace{0.17em}\mathrm{otherwise,}\end{array}$(3)where W_x,y is the weight of the median in a re-sampled mosaic pixel, ∑ _k is a sum over all the images that overlap with the pixel, w_k is the weight value in the image k, and n_{≠ 0} is the number of non-zero pixels.

4. Catalog of low surface brightness objects

4.1. Quantitative selection criteria

We aim to identify and classify the LSB galaxies in the selected Fornax fields, in particular the extended UDGs. According to our experience (see also Muñoz et al. 2015; and Müller et al. 2015), automatic detection using for instance SExtractor at LSB levels is at present not as reliable as the eye, so that the catalog was created by visually inspecting the images (done by AV). In future this catalog can be used as a control sample to test the completeness of any automatic detection method. Our sample includes diffuse sources fulfilling the criteria listed below, in other words we are deliberately excluding compact galaxies with the r′-band central surface brightness of μ_0,r′< 23 mag arcsec^-2. Most of the galaxies with surface brightnesses brighter than that are already identified in the Fornax Cluster Catalog (FCC; Ferguson 1989b) or can be easily found with automatic detection algorithms.

The selection criteria for the sources are:

• 1.

  Low surface brightness: the object has a central r′-band surface brightness of μ_0,r′ ≳ 23 mag arcsec^-2.

• 2.

  Extended: the object has a diameter of d₂₇ ≳ 10 arcsec (corresponding to 0.9 kpc) in the r′-band, at the surface brightness level of 27 mag arcsec^-2. So, we excluded small dwarf galaxies, and sources too faint for any reliable fitting of the surface brightness profiles. The Point Spread Function (PSF) for a source with a central surface brightness of μ_0,r′ = 23 mag arcsec^-2 has d₂₇< 4 arcsec, so that there is no danger to mix faint stars with objects.

• 3.

  Multi-band detection: the object can be recognized visually, and it has similar shapes in g′, r′ and i′-bands.

• 4.

  No contamination from bright sources: we excluded all the areas which have severe contamination from stellar halos (see Fig. 3) meaning that the overlapping halo is brighter than 24 mag arcsec^-2. This is to make sure that possible halo patterns are not confused with sources or cause biases to the photometry. In the vicinity of bright and extended galaxies like NGC 1399, we included only objects, which are located outside the 24 mag arcsec^-2 isophotes of the bright galaxy.

The identification criterion 1 was set to guarantee that also faint sources possibly omitted in the previous studies, are systematically identified. The objects with the lowest surface brightness in FCC have μ_0,B ≈ 24 mag arcsec^-2, which corresponds to μ_0,r′ ≈ 23 mag arcsec^-2. The galaxies fainter than this limit have been only rarely mapped before (Muñoz et al. 2015; Hilker et al. 2003; Bothun et al. 1991; Mieske et al. 2007; Kambas et al. 2000).

4.2. Accounting for imaging artefacts

Due to their LSB true objects can easily be confused with imaging artifacts, such as residuals from the background subtraction, or faint reflections from the instrument’s optics. The criterion 3 was set to filter out such false detections.

OmegaCAM, like many other wide-field instruments such as MegaCAM, is known to have strong halos (see Fig. 3) around bright stars, caused by reflections from the secondary mirror. These halos appear on the extension of the line connecting the bright source and the focal point of the CCD array. They are easy to identify as they are always associated with a bright star, and their brightness scales with that of the source. The criterion 4 ensures that these reflections will not bias our photometric analysis.

OmegaCAM is also known to have cross-talk between the CCDs 93-96 (see OmegaCAM user manual³ provided by ESO), which can lead a bright source (a star or a galactic nucleus) to appear as a faint ghost image in an adjacent CCD. The cross-talk can manifest either as positive or negative patterns, which have the same shape as the object causing that pattern. The negative crosstalk cannot be confused with the sources, but the positive crosstalk, ending up to 4% of the surface brightness of the source causing it, is more problematic. A crosstalk pattern may therefore have the same appearance as a faint diffuse source. However, as the crosstalk appears always in the same pixels in both CCDs, it is possible to identify that pattern by looking for a bright source with a similar shape within a distance of 7.5 arcmin. Even if a faint source appears in all three bands with a similar shape, it does not automatically exclude the possibility of being a crosstalk ghost. Indeed, in the vicinity of bright point sources the shapes and locations of the LSB sources always need to be compared with possible crosstalk sources.

Earlier works, such as that by Duc et al. (2015) have shown that Milky Way’s dust (cirrus) can be easily confused with LSB galaxies or streams. However, Iodice et al. (2016) showed that neither cirrus nor zodiacal light can cause contamination to the images in the Fornax cluster.

4.3. Producing the object catalog

The four fields used in this study were inspected visually to detect the sources (see Fig. 4). They were first identified from the 4 × 4 re-binned images and then from the unbinned r′-band images. Using both re-binned and unbinned images helped detecting LSB galaxies and structures in different intensity scales. As the objects were identified and selected visually, some of the smallest objects in our sample do not fulfil the criterion 2 after taking into account the effects of the PSF. However, as they are yet LSB galaxies, they are included in the analysis. We used the field definitions and running numbers to name the objects: for example, the sources found within the field 11 are called as FDS11_LSBn, where n is replaced with a running number.

After detection, the coordinates of the object were stored and postage stamp images (see Fig. 5 for an example) were cut in all three bands. The size of the image was adjusted manually to be at least eight times the isophotal (d₂₇) diameter of the object so that a sufficient number of background pixels are included, but as little as possible light is coming from the other sources. The maximum size was limited to 500 × 500 pixels (corresponding to 1.7 arcmin × 1.7 arcmin) to make the data easily editable. The postage stamps which were larger than the maximum size were rebinned to fit the limits.

After checking the original mosaic for possible crosstalk or false detections, masks were generated using SExtractor (Bertin 2010). All the sources were masked, and a 2D-plane with three degrees of freedom (intensity level and gradients along the x- and y-axes) was least square fit to the masked image and subtracted. This gives the first order approximation for the background level, and helps in judging whether a faint structure is real or not. We emphasize here that this model is not the final background model used for photometry, but only for editing the masks and identifying the object. If necessary, the masks were then modified, as in some cases the SExtractor masks were partly masking also the galaxy, or were not masking the PSF wings properly.

A total of 251 diffuse sources were initially identified from the four inspected fields. From this sample 17 objects are considered as imaging artifacts or they are too faint to be analyzed properly, leaving a sample of 234 objects. As discussed in the next section, 205 of these appeared to be galaxies.

4.4. Distinguishing galaxies from tidal structures

In this work we are particularly interested in the effects of the environment on the properties of LSB galaxies, and therefore want to exclude structures which most probably are tidal debris of an ongoing interaction. Therefore, the objects are classified either as galaxies or tidal structures based on their appearance. At this stage this distinction is qualitative and has an intrinsic uncertainty that is impossible to remove without spectroscopic data. We are aware of the possibility that some of the galaxies might have been born as a result of stripped tidal debris (Bournaud 2010), a possibility which is not ruled out in our approach.

In our classification galaxies can be identified as separate objects from their surroundings. They are structures that have distinguishable centers with excess light, and/or have apparent symmetry. Nevertheless, an irregular structure which is not connected to any other nearby sources will still be classified as a galaxy. Tidal structures have elongated or irregular shapes. They are typically connected to a pair or a group of galaxies that have disturbed appearance. They do not have a center with excess light. A structure that has a connecting bridge with another object is a galaxy if it has a clear center, but is otherwise classified as a tidal structure.

The classification criteria for both classes are listed in Table 1. As the objects appear in a variety of shapes it is not always immediately clear in which group an object should be classified. Therefore, a peculiar object is classified to the class where it fulfills at least two of the criteria. Examples of objects belonging to the two classes are shown in Fig. 5. From the total sample of 234 objects 205 are classified as galaxies and 29 as tidal structures. We compared these galaxies with the catalogs of Muñoz et al. (2015), Mieske et al. (2007), Bothun et al. (1991), and Ferguson (1989b). 59 of our galaxies are not included in any of those catalogs and are therefore new identifications.

5. Structural parameters and photometry

We used the calibrated r′-band postage stamp images to measure the effective radius (R_e), the apparent magnitude (m_r′), and the minor-to-major axis ratio (b/a), for all identified galaxies. Two different methods were used. In the first method we produced the radial light profile for each galaxy using azimuthally averaged bins, and then fit the 1D-profile with a single Sérsic function. In the second method, we used GALFIT 3.0 (Peng et al. 2010, 2002) for fitting the 2D flux distribution of the galaxy, again using a single Sérsic function, and if needed an additional PSF component was fit to the nucleus.

5.1. Azimuthally averaged surface brightness profiles

Radial profiles can be defined by fitting the isophotes with a series of ellipses leaving the center, axis ratio, and position angle as free parameters. However, since the S/N of the galaxies in our sample is low, the shape of the ellipses in this approach becomes unreliable in the galaxy outskirts. To prevent this from affecting the measured profiles, we decided to use fixed ellipticity for the elliptical annuli based on pixel value distribution moments. Also, as the galaxies we study typically do not have well defined central peaks, a special approach was needed for defining the centers.

The center of the galaxy and its shape were defined using pixel distribution moments. A detailed description of the method can be found in Appendix A. In the measurement of the profiles, we used a radial bin width of 4 pixels, or 0.8 arcsec (for comparison the FWHM of the PSF is ≈1.1 arcsec, see Fig. 6 below). The adopted bin width ensures that the bins are small enough to be able to capture the changes in the radial shape of the profile. The radial bins extend to the distance where the intensity level within the bin drops to 1/3 of the pixel-to-pixel background rms, which typically corresponds to μ_r′ ~ 28 mag arcsec^-2. All the masked pixels are rejected, and three times σ-clipped averages are used as the bin values. For the error of the bin we adopted the standard deviation of the non masked pixels within the bin, divided by the square root of number of the pixels.

At this point the images may still include some positive or negative residual sky, since the initial sky fit was done using SExtractor masks which often fail to cover the faint outskirts of the sources. For the final sky level we used the value measured at the radius of 4 R_e. The value of R_e was obtained from the cumulative light profile, and the sky level was measured from an 8 pixel wide galactocentric annulus, placed at r = 4 R_e. The annulus was divided into 20 azimuthal sectors, and a median of each sector was taken. Finally we used four times σ-clipped average of the medians as the residual sky value, which was then subtracted from each bin.

The sky corrected azimuthally averaged radial profiles were fit with a single Sérsic function using intensity units: $\mathit{I}\mathrm{\left(}\mathit{r}\mathrm{\right)}\mathrm{=}{\mathit{I}}_{\mathrm{e}}\exp \left[\mathrm{-}{\mathit{b}}_{\mathit{n}}{\left(\frac{\mathit{r}}{{\mathit{R}}_{\mathrm{e}}}\right)}^{\frac{\mathrm{1}}{\mathit{n}}}\mathrm{-}\mathrm{1}\right]\mathit{,}$(4)where R_e is the effective radius, I_e is the surface brightness at R_e, and n defines how peaked the Sérsic profile is. The parameter b_n depends on n as Γ(2n) = 2γ(2n,b_n), where Γ and γ are the complete and incomplete gamma functions, respectively (Sersic 1968; Ciotti 1991). While least-square fitting the Sérsic function, the radial bins were weighted with their inverse variances. The values for R_e, n and the total apparent magnitude m_r′ were obtained from the 1D-Sérsic fit. Those values were also used as the initial values of the Sérsic profile in the GALFIT models.

5.2. GALFIT models

GALFIT has been successfully used in several works to model the 2D light-distributions of bright galaxies (see e.g., Peng et al. 2010; Salo et al. 2015; Hoyos et al. 2011) and also those of faint galaxies (Janz et al. 2012, 2014). However, Muñoz et al. 2015 claim that for the LSB galaxies not all of their fits using GALFIT converged. Nevertheless, in this study we have successfully fit all our sample galaxies with GALFIT, to obtain R_e, ${\mathit{m}}_{\mathit{r}}^{\mathrm{\prime }}$, n and b/a. Most likely our success stems from using good initial parameters in GALFIT obtained from the 1D fits.

GALFIT is capable of fitting several components simultaneously, taking into account the effects of the PSF and proper weighting of the data. The weights created during the data reduction were used for obtaining the σ-images needed for the GALFIT fits. The pixel value σ_i,j for the σ-image is calculated as follows: ${\mathit{\sigma }}_{\mathit{i,j}}^{\mathrm{2}}\mathrm{=}\frac{\mathrm{1}}{{\mathit{W}}_{\mathit{i,j}}}\mathit{,}$(5)where W_i,j corresponds to the value of the weight image in pixel coordinate i,j.

While fitting with GALFIT it is possible to convolve the models with a given PSF-image. In order to take into account the variations of the PSF, we made a separate model for each field. We used SExtractor to detect and select the stars to be used for the model. SExtractor has a parameter CLASS_STAR, which indicates the probability of an object to be a star (see Bertin & Arnouts 1996, for details). We selected the objects with CLASS_STAR> 0.8. We also excluded saturated and faint stars from the stack by using SExtractor’s automatic aperture photometry MAG_AUTO. As the stars brighter than 15 mag are typically saturated, we included only the stars with 16 mag <MAG_AUTO< 19 mag. A pixel area 101 × 101 pixels was cut around each star, and normalized by the total flux within it. All the cut images were then median combined to get an average PSF model. As the background has been subtracted in the images already in the reduction, we do not apply further background subtraction at this point. Any possible contamination from the nearby objects in the PSF – stack is averaged out when the stack is median combined. Figure 6 shows an example of the stacked model (for field 11) as well the best fit Gaussian and Moffat (Moffat 1969) models. We decided to use the stacked model, as particularly the Gaussian function fail to fit the faint outermost parts of the PSF. We did not take into account the PSF FWHM variations within the 1° × 1° fields, as we found them to be less than 10% (consistent with de Jong et al. 2015).

Initially, we fit all the images using a single Sérsic function as a model for the galaxy, and a plane with three degrees of freedom (intensity level and gradients along the x- and y-axes) as the sky component. The parameters for R_e, m_r′, n, b/a and the center coordinates obtained with the 1D fitting method (explained in Sect. 5.1), were used as input parameters in GALFIT. All the parameters except for the galaxy center, were fit as free parameters.

In some of the fits the center of the galaxy was initially defined incorrectly. This can be seen as asymmetric residuals, so that part of the galaxy has positive residual values and the other part has negative values. In such cases we fixed all the other fit parameters except for the galaxy center, and let GALFIT to refit the center of the Sérsic profile. After this the center was fixed again and the other Sérsic parameters were fit again. This procedure reduced the fit residuals significantly.

Some of the galaxies show also a peaked nucleus (see the 4th row in Fig. 7) which is clearly a separate component in the galaxy center. In such cases, we manually place an additional PSF-component to fit the nucleus. The PSF-component is added to improve the overall fit of the galaxy rather than aiming for the detailed analysis of the nuclear star clusters. We fit the PSF-component using a fixed center position, and leave the total magnitude of the nucleus as a free parameter to fit.

Using the steps described before, our fits converged for all the galaxies in our sample. In the end, we examined the fits for systematics in the residuals, and corrected the ones where the center of the galaxy was wrong or the nucleus was missed. To ensure that the sky components are fit correctly, the radial surface brightness profiles and the cumulative radial profiles were inspected. Specifically we checked that GALFIT does not mix the sky component with the Sérsic profile, which would give unrealistically large effective radii and total magnitudes to the galaxies. The opposite (absorbing galaxy light to sky) is unlikely, since all our images had large areas of background sky not covered by the galaxy.

5.3. Comparison of the 1D and 2D methods

Figure 8 shows a comparison of the Sérsic n-values, axis ratios (b/a), total r′-band magnitudes (m_r′), and effective radii (R_e) obtained with the two methods. Both methods give similar distributions for the Sérsic n values, although the scatter is significant with rms = 0.47. The effective radii and magnitudes are well in agreement in the two measurements (the lower panels of 8), except for the four outliers. The outliers are explained due to the different fitting of their background levels in the 1D and 2D methods. Only for the axis ratios b/a a systematic shift appears between the two methods (see upper right panel), so that the values defined using the distribution moments are systematically closer to unity than the ones obtained using GALFIT. By inspecting the residual images, we concluded that the axis ratios obtained by GALFIT resemble more the actual shapes of the galaxies (see e.g., the second row of Fig. 7).

As expected, the position angles are similar when both methods show small b/a values. The galaxies which show a large difference between the elliptical shape measured with GALFIT and the distribution moments, are often located near to other sources. GALFIT seems to be more stable in the presence of such disturbances. Therefore, for the analysis we decided to use the values measured with GALFIT. The histogram for R_e values used in the analysis is shown in Fig. 9, and the histograms for magnitudes (m_r′), mean effective surface brightness⁴ (), axis ratios (b/a), and Sérsic n-values are shown in Fig. 10.

We conclude that both of these methods can be used for studying LSB galaxies. However, GALFIT is more accurate in obtaining the axis ratios. Also, as it is capable of correcting for the effect of the PSF, it should be used when the intrinsic shapes of these galaxies are analyzed. When comparing the background estimation of these methods, GALFIT has slight advantage as it uses all the non-masked pixels and allows the background to have gradients. However, our observation that the background level of GALFIT can be somewhat degenerated with the outer parts of the Sérsic-profile is clearly an issue that should be acknowledged when this method is used.

5.4. Accuracy of the photometric measurements and completeness of the UDG detections

5.4.1. Comparison to Muñoz et al. (2015)

We checked how the completeness of the galaxy sample obtained by us compares with that of Muñoz et al. (2015). Since the two samples have a different spatial coverage, we limited the comparison to field 11 (see Fig. 1) which is covered by both studies. There are 96 and 62 galaxies located in that field in the samples of Munoz et al. and this study, respectively. 52 of those galaxies are common to both studies. The sample of Munoz et al. has 44 objects that are not included in our sample, which is explained by our selection criteria given in Sect. 4.1. When these criteria are taken into account, 43 of the 44 galaxies appearing only in the sample of Munoz et al. are excluded⁵ from our sample. The remaining object was classified as a tidal structure by us. In that same field, there are ten objects that are not in the sample of Munoz et al., but appear in our sample.

We also checked how the parameters obtained by us compare with those given in Muñoz et al. (2015), for the 126 sources identified in both studies. In both studies the parameters are obtained using GALFIT. Figure 11 (right panels) shows the differences for the galaxies as a function of the surface brightness, and Fig. 12 shows a comparison of Sérsic n, R_e and m_i′ between the two measurements. In order to convert the i′-band magnitudes of Muñoz et al. (2015) to r′-band, we used the median r′−i′ aperture color⁶ of ⟨ r′−i′ ⟩ = 0.3 mag from the FDS data, measured within R_e. The measured offsets and their standard deviations are Δm_i′ = 0.3 with σ = 0.3 mag, ΔR_e/R_e = 0.0 with σ = 0.2, and Δn = 0.0 with σ = 0.2. We find that our R_e and n values are well in agreement with those of Muñoz et al. (2015). However, there is a small offset in the total magnitudes, so that the values measured by us are 0.3 mag brighter.

5.4.2. Tests with mock galaxies

In order to test the completeness of our faint galaxy detections, and the reliability of the parameters obtained for them, we added mock-galaxies to our images. The detection efficiency is tested for UDGs by embedding mock UDGs with their parameters adopted from Mihos et al. (2015) and van Dokkum et al. (2015) to the science images (see Appendix B for details). Additionally, we tested the photometric accuracy of our measurements for the UDGs and LSB dwarfs using additional ~150 mock galaxies with parameters typical for the galaxies in the sample of Muñoz et al. (2015). The parameter ranges of the used mock galaxies are shown in Appendix B.

The detection efficiency for UDGs is tested by embedding mock UDGs with parameters adopted from 16 representative UDGs from Mihos et al. (2015) and van Dokkum et al. (2015), to the r′-band science images of all the studied fields one by one. They were distributed randomly across the image and their coordinates were stored. After embedding the mock galaxies, the image was inspected and the sources were identified (see Sect. 4). In this test, we failed to find only one UDG, which should have been detected according to our selection criteria. This UDG was not detected as it was overshadowed by the diffuse light of a nearby star and the outskirts of a galaxy. The results of this test for different fields are listed in Appendix B. The test shows that our data and the visual detection is efficient (≈92% efficiency) in detecting UDGs such as those presented in van Dokkum et al. (2015) and Mihos et al. (2015).

We made GALFIT fits for all the mock galaxies described in Appendix B. The photometry was perforemd as described in Sect. 5.2 Fig. 11 collects the differences in m_r′, R_e, b/a, and Sérsic n, between the original and measured values (red and black dots), for all identified mock galaxies. Any possible systematic shifts are negligible in all studied parameters. The measured offsets are Δm_r′ = 0.00 mag, ΔR_e/R_e = 0.01, Δ(b/a) = 0.00, and Δn = −0.01.

To characterize the typical uncertainties of the fit parameters, we tabulate the 1σ-deviations in Fig. 11 as a function of the mean effective surface brightness. Similarly as Hoyos et al. (2011), we fit these intrinsic standard deviations (σ), which we assume to follow a Gaussian distribution, with a simple linear function: ${\log }_{\mathrm{10}}\mathrm{\left(}\mathit{\sigma }\mathrm{\right)}\mathrm{=}\mathit{\alpha }\mathrm{\times }\mathit{\mu ̅}\mathrm{e}\mathit{,}{\mathit{r}}^{\mathrm{\prime }}\mathrm{+}\mathit{\beta ,}$(6)where α and β are free parameters. The fit results are listed in Table 2, and the error estimates for the individual galaxies are given with their other photometric parameters in the Appendix D.

6. Locations and orientations of LSBs within the Fornax cluster

6.1. Radial number density profile

The locations of the galaxies identified by us in the four Fornax cluster fields are plotted over the combined i′, r′, and g′-band image in Fig. 2. As explained in Sect. 4.1, we masked all the areas covered by stellar halos or bright extended galaxies. Then a cluster-centric radial surface density profile was made by counting the number of objects in radial bins and dividing those numbers by the non-masked area within each bin. The number density profile and the corresponding cumulative profile are shown in Fig. 13. We used NGC 1399 as the center of the cluster, since the hot intra-cluster X-ray gas is centered to it (Paolillo et al. 2002). Also, the smoothed FCC galaxy number density distribution peaks on top of NGC 1399 (Drinkwater et al. 2001a).

We make a comparison between the radial distribution in the cluster of the LSB dwarf galaxies identified by us, and that of the FCC galaxies (see Fig. 13). For the FCC galaxies we used those classified as “confirmed” or “probable cluster members” in Ferguson (1989b). It appears that the LSB galaxies of this study are less centrally concentrated than the more luminous FCC galaxies. The Kolmogorov-Smirnov (K-S) test gives a p-value of 0.032 for the assumption that the two distributions are from the same underlying distribution, indicating that the difference is statistically significant (p < 0.05). In principle there can be a bias in this comparison, because the FCC galaxies are identified also on top of the halos of bright stars and galaxies, which areas were excluded in our study. However, such bias would affect our result only if the galaxies in the central parts of the cluster were more concentrated to the halos of bright galaxies than to the surrounding fields.

We further divided the FCC into bright galaxies with mean effective surface brightness mag arcsec^-2, and to faint galaxies with mag arcsec^-2. By comparing the radial distributions of the galaxies in these two bins shows that the bright FCC galaxies are more centrally concentrated than the galaxies in our sample (K-S test p-value = 0.014). When comparing the radial distributions of the faint FCC galaxies with our sample galaxies, the K-S test gives a p-value of 0.32. This p-value means that these two distributions are not statistically different, which is expected as these two samples have several galaxies in common.

6.2. Orientations

The orientations for the individual galaxies in our sample are shown in Fig. 14, where over-plotted are also the locations of the FCC galaxies in the same field. The relative orientations with respect to the cluster center and the closest FCC galaxy with M_r′ < −18 mag⁷ are plotted in Fig. 15. However, as the galaxies with b/a ~ 1 may cause additional noise to the orientation plots, thus blurring possible underlying dependencies, only galaxies with b/a < 0.9 are considered. It appears that when including all the galaxies up to b/a = 0.9, there is no statistically significant preferred alignment, neither toward the closest bright galaxies (p-value = 0.657, for retaining the hypothesis that the alignments are random), nor toward NGC 1399 (p-value = 0.060). However, the galaxies with b/a < 0.7 show a weak preferred alignment toward their bright nearby galaxies, for which a K-S test gives a p-value of 0.031.

7. Colors of the sample galaxies

We measured the aperture magnitudes in the g′, r′ and i′-bands for the galaxies in our sample, using the background (GALFIT) subtracted and masked images. However, in this paper we only analyze the g′−r′ color, since in these bands the data are the deepest, and give accurate colors for most objects. We used elliptical apertures defined by the parameters obtained from our r′-band GALFIT fits (center coordinates, and R_e,r′ used as the major axis of the aperture). Instead of using total magnitudes aperture colors were obtained. This is to minimize systematic errors from the sky background determination. As in Capaccioli et al. (2015), we estimate the errors σ_g′−r′ for g′−r′ colors as: $\begin{array}{ccc}{\mathit{\sigma }}_{{\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}}^{\mathrm{2}}\mathrm{=}{\mathit{\sigma }}_{\mathrm{ZP}\mathit{,}{\mathit{g}}^{\mathrm{\prime }}}^{\mathrm{2}}\mathrm{+}{\mathit{\sigma }}_{\mathrm{ZP}\mathit{,}{\mathit{r}}^{\mathrm{\prime }}}^{\mathrm{2}}& & \\ \mathrm{+}{\left(\frac{\mathrm{2.5}}{{\mathit{I}}_{{\mathit{g}}^{\mathrm{\prime }}}\ln \mathrm{10}}\right)}^{\mathrm{2}}\mathrm{(}{\mathit{\sigma }}_{\mathit{I,}{\mathit{g}}^{\mathrm{\prime }}}\mathrm{+}{\mathit{\sigma }}_{\mathrm{sky}\mathit{,}{\mathit{g}}^{\mathrm{\prime }}}{\mathrm{)}}^{\mathrm{2}}\mathrm{+}{\left(\frac{\mathrm{2.5}}{{\mathit{I}}_{{\mathit{r}}^{\mathrm{\prime }}}\ln \mathrm{10}}\right)}^{\mathrm{2}}\mathrm{(}{\mathit{\sigma }}_{\mathit{I,}{\mathit{r}}^{\mathrm{\prime }}}\mathrm{+}{\mathit{\sigma }}_{\mathrm{sky}\mathit{,}{\mathit{r}}^{\mathrm{\prime }}}{\mathrm{)}}^{\mathrm{2}}\mathit{,}& & \end{array}$(7)where I_g′ is g′-band mean intensity within the aperture, and σ_I,g′, σ_sky,g′ and σ_ZP,g′ are the errors for the surface brightness, the sky, and the photometric zero point in g′-band, respectively. I_r′, σ_I,r′, σ_sky,r′ and σ_ZP,r′, are the corresponding quantities in r′-band. For the mean intensity we assumed simple Poissonian behavior, so that ${\mathit{\sigma }}_{\mathit{I,}{\mathit{r}}^{\mathrm{\prime }}\mathit{/}{\mathit{g}}^{\mathrm{\prime }}}\mathrm{=}\sqrt{{\mathit{I}}_{{\mathit{r}}^{\mathrm{\prime }}\mathit{/}{\mathit{g}}^{\mathrm{\prime }}}\mathit{/}\mathrm{(}\mathit{GAIN}\mathrm{\times }\mathit{n}\mathrm{)}}\mathrm{\times }\mathit{GAIN}$, where n is the number of pixels within the aperture. I, σ_I and σ_sky are given in flux units, whereas σ_ZP are in magnitudes.

The g′−r′ colors of the galaxies as a function of their total absolute r′-band magnitudes M_r′ are shown in the upper panel of Fig. 16, with the typical errorbars of the colors shown below the points. Plotted separately are the nucleated and non-nucleated LSB dwarfs (with R_e< 1.5 kpc), and the UDGs (with R_e> 1.5 kpc). Pearson’s correlation coefficient for the points shows a negative correlation ρ = −0.33 ± 0.04, and a linear fit gives the relation g′−r′ = −0.04( ± 0.01) × (M_r′ + 12)−0.48( ± 0.09). In the color–magnitude relation the UDGs are among the brightest galaxies, but follow the same relation with the dwarf LSB galaxies in our sample. The measured slope of the color–magnitude relation of our sample is the same as that measured for the Virgo dwarf ellipticals by Kim et al. (2010)d(g′−r′)/dM_r′ = −0.04 ± 0.01, and for the Virgo red sequence by Roediger et al. (2017)d(g′−r′)/dM_r′ = 0.3−0.4.

The g′−r′ colors of the galaxies as a function of the mean effective surface brightness in the r′-band are shown in the lower panel of Fig. 16. We find that the colors correlate also with the surface brightness becoming redder with increasing surface brightness. Pearson’s correlation coefficient for the points is ρ = −0.26 ± 0.05, and a linear fit gives the relation . A more thorough discussion of the colors of galaxies in the FDS catalog will follow in a future paper.

8. Discussion

The motivation of this work is to study the properties of the LSB galaxies in the Fornax cluster, and how they compare with those observed in other clusters or galaxy groups. We are particularly interested in the UDGs, using a threshold surface brightness and size typical for the previously identified UDGs in clusters.

8.1. Concept of an UDG in the literature

To conduct a meaningful comparison between the UDGs in the Fornax cluster and in other galaxy environments, it is important to make sure that the objects we are comparing are selected similarly. The definition of UDGs, adapted from van Dokkum et al. (2015) for the Coma cluster, was that they are galaxies with R_e> 1.5 kpc, and stellar mass of M_{∗ ,L} ≈ 10⁷M_⊙, or −16.2 mag <M_r′< −13.2 mag⁸. Works published earlier than that might contain a few similar galaxies (see e.g., Sandage & Binggeli 1984; Bothun et al. 1991; Conselice et al. 2002), in which case they were typically called as LSB galaxies. Since the largest UDGs found so far have R_e ≈ 10 kpc (Mihos et al. 2015), and the smallest ones overlap with the typical dE galaxies, it is possible that some of the UDGs form the low mass tail of the dEs with atypically large effective radii, and some of them form a genuinely distinct population. To study this, in the following we analyze separately the properties of the small UDGs with 1.5 kpc <R_e< 3.0 kpc (i.e., with typical sizes of UDGs in Coma), and large UDGs with R_e> 3 kpc.

A comprehensive collection of UDG studies in the literature has been presented by Yagi et al. (2016). However, not all of these works have sufficient image depth and the same measurements given as in this study. Also, most of these works contain very few UDGs. Here we discuss only those works to which we can make comparisons easily, without any auxiliary assumptions about the shapes or colors of these galaxies. The most complete available UDG samples have been made for the Coma cluster by Koda et al. (2015, included in Yagi’s collection), and for galaxy clusters at larger distances by van der Burg et al. (2016). Both of these studies used SExtractor to generate object lists, and GALFIT to fit Sérsic profiles to the galaxies.

8.2. Comparison of UDGs in Fornax and in other environments

We found nine UDG candidates in Fornax within the 4 deg² search area, of which five have R_e< 3 kpc, and four have R_e> 3 kpc. Three of these galaxies appear also in the sample of Muñoz et al. (2015), two of them are detected in Mieske et al. (2007), five appear in the FCC, and four are detected by Bothun et al. (1991). Additionally, two of the UDGs appear in Lisker et al. (2017), but in that work FDS11_LSB1 is not considered as a galaxy. We identified all the UDGs in Muñoz et al. (2015) that were located within the area of our study. We also identified “FDS11_LSB30”, classified as UDG in Muñoz et al. (2015), but because of its small size (R_e< 1.5 kpc) it was not classified as UDG. Two of our UDGs are new detections. Figure 17 shows the r′-band stamp images of the detected UDGs, with their structural parameters and cluster centric distances listed in the upper right corner. The structural parameters of these UDGs are given in Table 3.

8.2.1. Sizes

The sample of Yagi et al. (2016) includes 288 UDGs (the total number of galaxies is 854 galaxies) residing in the Coma cluster, of which 267 have R_e< 3 kpc, and 21 are larger than 3 kpc. Most of these UDGs have R_e ~ 1.5 kpc, and the number of UDGs drops rapidly with increasing effective radius. The largest UDG in Coma has R_e = 6.1 kpc, whereas in Fornax the largest one has R_e ~ 10 kpc, which is considerably larger. The Coma UDGs have r′-band magnitudes⁹ between −16.8 mag <M_r′< −11.8 mag, whereas in our sample for Fornax the magnitude range is −15.8 mag <M_r′< −13.5 mag. The narrower magnitude range in Fornax is explained by the smaller sample size, as these two samples have similar medians and standard deviations (see Table 3).

The size-magnitude relations of the UDGs in the Fornax and Coma clusters (van Dokkum et al. 2015; Yagi et al. 2016) are shown in the upper and lower left panels of Fig. 18, respectively. It appears that in the Coma cluster the division between UDGs and LSB dwarfs at 1.5 kpc is artificial meaning that they form a continuous distribution in the size-magnitude parameter space. The same is not that apparent from the size-magnitude relation of the Fornax galaxies, where at least the two largest UDGs are clearly outliers.

UDGs appear also in the Virgo cluster, although no systematic search of them has been done. Gavazzi et al. (2005) measured structural parameters of a sample of early-type galaxies in the Virgo cluster, and found 14 galaxies with 1.5 kpc <R_e< 3.0 kpc and mag arcsec^-2, which according to our criteria would be classified as UDGs. However, their study does not reach similar image depths as the deepest images obtained for the Coma and Fornax clusters, which probably explains why they do not find many UDGs larger than 3 kpc. Particularly the large UDGs in our study have low effective surface brightnesses being fainter than mag arcsec^-2. However, Mihos et al. (2015) find three large UDGs (R_e ~ 3−10 kpc) in the central parts of the Virgo cluster. Although these findings do not comprise a complete sample, they already demonstrate that UDGs in Virgo can be as large as the largest UDGs in the Fornax cluster.

The observation that the largest UDGs in Coma are smaller than in Fornax or Virgo clusters, most likely is a detection bias related to the fact that the UDG identifications both in Yagi et al. (2016) and van Dokkum et al. (2015) are made using SExtractor. van der Burg et al. (2016) tested the detection efficiency of SExtractor using artificial LSB galaxies: they found that the detection efficiency is less than 0.5 for the UDGs with R_e ~ 3 kpc and mag arcsec^-2, and further drops toward lower surface brightnesses and larger effective radii. Thus, if large UDGs like the ones detected in this study exist in Coma, most of them would not have been detected using automatic methods.

In the middle panels of Fig. 18 we show the normalized number distributions of the effective radii (of a magnitude- and size-limited sample) of the LSB galaxies in Coma and Fornax clusters. The galaxies are chosen so that neither sample is limited by its selection criteria: we selected the galaxies within two cluster core radii from the cluster centers corresponding to 450 kpc and 700 kpc in the Fornax (Ferguson 1989a) and Coma clusters (Kent & Gunn 1982), respectively. Additionally, we required these galaxies to have R_e> 1 kpc, mag arcsec^-2 and −18 mag <M_r′−12 mag (shown with the blue area in the left panels). The histograms are normalized by the number of bright galaxies¹⁰ (N_bright) with M_r′< −17 mag in the selected area. These histograms highlight the difference between Fornax and Coma, the large UDGs in the former being clearly detached from the rest of the LSB population. We also plot the relation observed by van der Burg et al. (2016), which tells that the number of UDGs with given R_e decreases as n[dex^-1] $\mathrm{\propto }{\mathit{R}}_{\mathrm{e}}^{\mathrm{-}\mathrm{3.4}\hspace{0.17em}\mathrm{\pm }\hspace{0.17em}\mathrm{0.2}}$, where n[dex^-1] is the number of UDGs within a logarithmic bin. In the Fornax cluster, this relation clearly underestimates the number of large UDGs. This is not surprising as the sample of van der Burg et al. (2016) is known to miss many such galaxies, since they are using SExtractor. Using Monte Carlo modeling and assuming the sizes of the UDGs in Fornax to follow the van der Burg (2016) size distribution, we find that the probability of four or more UDGs out of nine having R_e> 3 kpc is p = 0.01. However, the number statistics alone is not sufficient to tell if the distribution of our sample is different from the one found by Van der Burg et al. A sample that is not limited by the number statistics (as is our sample) nor missing the large UDGs (like the ones using SExtractor are) is clearly needed to understand the total contribution of large UDGs to the total galaxy populations in clusters.

8.2.2. Number of UDGs

Taking into account the homogeneity of the data and our tests made with the mock galaxies, we should be able to detect all the galaxies down to mag arcsec^-2. Given the fact that we are excluding 20% of the area to avoid possible source confusion, we are probably missing 1−3 UDGs, which would be bright enough to be detected in case that they were not overlapping with other sources. Adding these missed galaxies to the nine identified UDGs we get surface number density of ν_UDG = 25 ± 8 UDGs Mpc^-2, which compares to ν_UDG ≈ 64 UDGs Mpc^-2 in the Coma cluster (N(UDGs) = 98 within the innermost 700 kpc in Yagi et al. 2016). While normalizing these numbers with the surface densities of the bright galaxies¹¹ in the studied areas, we get $\frac{{\mathit{\nu }}_{\mathrm{UDG}}}{{\mathit{\nu }}_{\mathrm{bright}}}\mathrm{=}\mathrm{0.7}\mathrm{\pm }\mathrm{0.2}$ and 0.45±0.05 for Fornax and Coma clusters, respectively. This shows that the normalized surface number density of UDGs in the Fornax cluster is larger than that in the Coma cluster.

There exists a correlation between the virial mass (M₂₀₀) of the cluster and the number of UDGs within the virial radius (r₂₀₀; van der Burg et al. 2016, see Fig. 19). For a cluster halo mass of 7 × 10¹³M_⊙, corresponding to that of the Fornax cluster (Drinkwater et al. 2001a), the expected number of UDGs within r₂₀₀ is 10–20. If we assume uniform density of UDGs and extrapolate it up to the virial radius of Fornax r₂₀₀ (0.7 Mpc, or 2.2°, Drinkwater et al. 2001a), we get an upper limit of N = 42 ± 12 UDGs. However, the sample of van der Burg et al. (2016) excludes galaxies that have mag arcsec^-2, or R_e> 7 kpc. Therefore, in order to make a fair comparison to their study we need to drop two out of the nine identified UDGs in our sample. Taking this into account leads to the upper limit of N = 33 ± 9 UDGs inside r₂₀₀, which brings the Fornax cluster to the relation by van der Burg et al. (see Fig. 19). For comparison, the Coma cluster has a mass of M = 1.4 × 10¹⁵M_⊙ (Łokas & Mamon 2003), and has 288 identified UDGs, which also matches well with the relation by van der Burg et al. (2016).

8.2.3. Shapes and orientations

The stamp images in Fig. 17 show that the large UDGs in our sample are less symmetric and more elongated than the smaller ones (see also the upper right panel of Fig. 18). Their shapes can be caused by tidal interactions, but in principle they can also be inclined disks or prolate spheroidals as suggested by Burkert (2017). Some UDGs have earlier been identified as disrupted early-type dwarfs, like HCC-087 in the Hydra I cluster (with M_V = −11.6 mag and R_e = 3.1 kpc) studied by Koch et al. (2012). Koch et al. were able to reproduce the observed s-shaped morphology of this galaxy by modeling its gravitational interaction with the cluster center. Also several other studies have identified signs of tidal disruption in UDGs (Mihos et al. 2015; Merritt et al. 2016; Toloba et al. 2016; Wittmann et al. 2017). Obviously, at some level tidal interactions are shaping the morphology of the LSB galaxies in clusters.

In the Coma cluster the UDGs are preferably elongated toward the cluster center (Yagi et al. 2016), which is not the case for the nine UDGs in the Fornax cluster (see Fig. 14). The number statistics are not good enough to conduct a conclusive analysis, but we found that some of the UDGs in Fornax are elongated toward their nearby dwarf galaxies (M_r′> −19 mag). In this sense an interesting pair is FDS11_LSB1 and FDS11_LSB2 (see Fig. 20), which galaxies are located at a projected distance of 15 kpc from each other. Both galaxies point toward a spectroscopically confirmed Fornax dE, FCC252 (Drinkwater et al. 2001b), which has a B-band total magnitude of M_B = −15.2 mag (Ferguson 1989b). FCC252 itself does not show any signs of tidal interaction. Another UDG showing signs of galaxy interaction is FDS16_LSB85 (see Fig. 4), which has a tidal tail pointing toward the nearby LSB dwarf, FCC 125 (Ferguson 1989b, or FDS 16_LSB50 in our sample). FCC 125 has M_r′ = −13.3 mag assuming that it resides at the distance of Fornax, although it has no spectral confirmation of the cluster membership. On the other hand, the four smallest UDGs with round and symmetric appearance do not show any signs of tidal interactions. In order to test if the elongated shapes of the large UDGs are resulting from tidal interactions, a larger sample is needed.

The properties of UDGs in our Fornax sample, and of those in the Coma cluster given by Yagi et al. (2016), are compared in Table 3. The mean axis ratio in Coma is ⟨ b/a ⟩ = 0.73 ± 0.16, the small (R_e< 3 kpc, ⟨ b/a ⟩ = 0.75), and large (R_e> 3 kpc, ⟨ b/a ⟩ = 0.69) UDGs having similar axis ratios. However, in our sample in Fornax the large UDGs are more elongated (⟨ b/a ⟩ = 0.69 and ⟨ b/a ⟩ = 0.38, for small and large UDGs, respectively).

While comparing the b/a-values between the small UDGs and the dwarf LSB galaxies (R_e< 1.5 kpc), we don’t find any significant differences, neither in the Fornax nor in the Coma clusters. However, the large UDGs in Fornax have a significantly different b/a-distribution compared to the one of LSB dwarfs (K-S test p-value = 0.0005, for the b/a distributions being the same). The same comparison between the two types of galaxies in Coma shows no significant difference (p-value = 0.24, see also the right panels in Fig. 18). We conclude that in the Fornax cluster the large UDGs are significantly more elongated than the dwarf LSB galaxies, but the same is not obvious in Coma. We find no difference between the axis ratio distributions of the LSB dwarfs and small UDGs.

As expected for the dwarf LSB galaxies (e.g., Bothun et al. 1991), the values of the Sérsic index are below unity for the UDGs, both in the Coma and the Fornax clusters. In fact, the n-values are similar in the Fornax (⟨ n ⟩ = 0.8 ± 0.2, n = 0.4−1.2) and in the Coma (⟨ n ⟩ = 0.9 ± 0.2, n = 0.4−2.0) clusters. The galaxies in both samples are fit using GALFIT, the only difference in the fitting approach being that Yagi et al. allowed the central PSF component of the galaxies to be off-centered from the Sérsic component.

8.2.4. Colors

The colors of UDGs set constraints for their formation mechanisms. The UDGs in Coma (Koda et al. 2015) have been shown to have similar colors as the normal red sequence dEs of the same luminosity. Also, the measured slope of the color–magnitude relation of our LSB sample matches well with the one of dE:s in Virgo cluster, which supports also them being mostly quiescent red-sequence galaxies. The mean color of the Coma UDGs is ⟨ g′−r′ ⟩¹² = 0.67 mag, which color is the same for the large and small UDGs. This is comparable to ⟨ g′−r′ ⟩ = 0.59 mag (with the range of 0.2–1.0) that we obtain for the UDGs in Fornax (see red triangles in Fig. 16). The only exception is FDS11_LSB1, which has g′−r′ = 0.18 mag, being a clear outlier toward blue colors. The observation that the UDGs do not clearly deviate from the colors of the LSB dwarf galaxies within the same luminosity range is consistent with them having similar origin.

8.3. How to explain the origin of UDGs?

8.3.1. Explanations in the literature

Van Dokkum et al. (2015) first pointed out that UDGs form a population that is continuous with the dwarf galaxies in the size-magnitude relation, and therefore can simply be a diffuse end tail in that relation. However, they do not think that UDGs originate from the same progenitors as the typical dwarf galaxies in clusters, since processes like harassment (Moore et al. 1998; Smith et al. 2015) and tidal stirring, rather make the galaxies more compact than diffuse. Their suggestion was that UDGs could be failed Milky Way mass (halo mass of ~10¹²M_⊙) galaxies which lost their gas during their in-fall into the cluster environment, and therefore were not able to form a large amount of luminous matter, otherwise typical for their observed size. This means that compared to their sizes, these galaxies might have massive dark matter halos. However, the number of globular clusters (GC) around these galaxies (Amorisco et al. 2016; Beasley & Trujillo 2016) rather suggest that they are Large Magellanic Could (LMC) type galaxies with halo mass of ~10¹⁰M_⊙). Also, the similar cluster-centric distributions of dwarf galaxies and UDGs found by Roman & Trujillo (2017a) and by van der Burg et al. (2016), is problematic in the interpretation of van Dokkum et al. This is because due to the large dark matter halos dynamical friction should have made these originally Milky Way sized galaxies to sink deeper into the cluster potential well, which is not observed.

Amorisco & Loeb (2016) suggested that UDGs are dwarf galaxies with especially high original angular momentum. In this scenario, the high angular momentum makes the UDGs more flattened and extended than the typical dEs are. Otherwise the formed galaxies are expected to be similar, and to appear in similar environments as the other dwarf galaxies. Their model predicts that the sizes of the UDGs increase with increasing angular momentum, which makes the largest UDGs more disk-like (elongated when inclined) than the smaller ones. This prediction is well in agreement with our observation that the largest UDGs are more elongated. However, the analysis of Burkert (2017) shows that the b/a-distribution of UDGs is more compatible with them being prolate spheroidals than disks. In Amorisco & Loeb (2016) the baryonic physics is not modeled, but they were able to model the relation between the cluster halo mass and the total number of UDGs as seen by van der Burg et al. (2016).

On the other hand, Di Cintio et al. (2017) suggested that UDGs are dwarf sized galaxies, which are extended due to their feedback driven gas outflows which give rise to an expanded stellar component. They modeled baryon physics of isolated galaxies, and for comparison with observations selected objects which fulfil the criteria of typical UDGs. Their models predict average B−R colors of 0.77 ± 0.12 mag (or g′−r′ = 0.5 mag) for these galaxies. The predicted color is 0.13 mag bluer than the mean observed value for the Coma UDGs, and 0.07 mag bluer than that the UDGs in the Fornax cluster in our sample. This discrepancy between the models and observations might be reduced if the galaxies were simulated in the cluster environment, where also external quenching takes place. An important prediction from their model is that the UDGs should mostly form outside the clusters, and also contain gas if it is not externally removed. This is well consistent with the observations of the ALFALFA survey (Leisman et al. 2017), where 115 HI rich isolated UDGs were studies. Additionally, Román & Trujillo (2017b) find that the colors of the UDGs located in the periphery of group environments have bluer colors (g−i ~ 0.45) than the UDGs in the clusters (g−i ~ 0.76), which is also consistent with this formation scenario. One discrepancy between the models of Di Cintio et al. (2017) and the observations is that the effective radii in the simulated galaxies vary between 1 kpc <R_e< 3 kpc, whereas the observed R_e can be as large as ~10 kpc. This discrepancy can simply be a sampling issue of the simulations, or alternatively the largest UDGs have a different origin.

Baushev (2016) suggested that UDGs can form via straight-on collisions of two massive galaxies, which penetrate each other with large relative velocities. This collision has little effect on the stellar content, but will rip out the cold gas in these galaxies. The collision rate strongly depends on the number density of galaxies, and high relative velocities are needed to prevent merging of the colliding systems. Therefore, this model is not good for explaining UDGs in galaxy groups nor in the field, where the relative velocities of galaxies are smaller. However, such collisions can hardly be prevented in galaxy clusters, where besides major mergers, also dwarf-sized galaxy collisions are expected. Since the strength of galaxy-galaxy interactions is stronger in low velocity encounters, the dwarf galaxies in groups are more likely to experience strong tidal forces than the ones in clusters, which could explain the large sizes of UDGs in the lower mass environments (e.g., Merritt et al. 2016). As these dwarf-dwarf interactions in clusters and groups are not well studied in simulations, further simulations of the importance of these tidal encounters are needed.

It seems that both the high angular momentum origin, and feedback induced gas outflows, can reproduce most of the observed properties of UDGs at present. However, at this moment, we also should seriously consider the contribution of the tidal interactions, which could be the explanation of the most extended UDGs.

8.3.2. Is the environment important for the formation of UDGs?

The Coma and Fornax clusters are clearly different environments, which should affect the observed properties of UDGs, if their evolution is driven by their environment. The virial mass of Coma is 1.4 × 10¹⁵M_⊙, and the virial radius is 2.9 Mpc (Łokas & Mamon 2003), while the corresponding values for the Fornax cluster are 7 × 10¹³M_⊙ and 0.7 Mpc (Drinkwater et al. 2001a). This means that although Coma is 20 times more massive than Fornax, it is less dense. The velocity dispersion of the Coma main group galaxies is σ = 1082 km s^-1 (Colless & Dunn 1996), which is nearly three times larger than the one of the Fornax cluster galaxies with σ = 374 km s^-1 (Drinkwater et al. 2001a). Therefore galaxy-galaxy interactions are expected to be stronger in the Fornax than in the Coma cluster.

As discussed above, the average magnitudes, g′−r′ colors and Sérsic n-values of the UDGs are similar in the Coma and Fornax clusters. The preferred alignment toward the cluster center that appears among the Coma UDGs, but not among those in the Fornax cluster, might suggest that the galaxy-cluster potential interaction might be more important in Coma.

On the other hand, UDGs have been found also in lower density environments outside the clusters. Four tidally disturbed UDGs have been found in the NGC 5485 group (Merritt et al. 2016), one UDG in a filament of the Pisces-Perseus supercluster (Martínez-Delgado et al. 2016), one UDG in the NGC 253 group (Toloba et al. 2016), and one as a satellite of Centaurus A (Crnojević et al. 2016). All these galaxies are large having effective radii R_e = 2.5−5 kpc. The size distribution of the low density environment UDGs seems to be different from the one in clusters, but again this can be a bias of cluster samples missing the large UDGs. However, recently Wittmann et al. (2017) visually inspected the UDG population in the core of the Perseus cluster (M_vir = 8.5 × 10¹⁴M_⊙, Mathews et al. 2006), but did not find any UDGs larger than R_e> 4.1 kpc. This observation might indicate that the difference in the effective radii distributions is not only an observational bias. Otherwise, there seems to be no structural difference between the UDGs in the group environments and the ones in clusters.

Recently, the observations of the isolated UDGs showed that they contain HI gas (Trujillo et al. 2017; Leisman et al. 2017), and posses bluer colors than the ones in clusters (Román & Trujillo 2017b; Leisman et al. 2017). Additionally, Leisman et al. (2017) observed that these HI rich UDGs may exist in high angular momentum halos. These observations are consistent with the scenario where UDGs are formed in dwarf-sized dark matter halos outside the clusters.

To conclude, from the observational (photometric) point of view we do not find systematic differences in the structure of the bulk of the UDGs in different galaxy environments. However, in Fornax, Virgo and group environments there are elongated large UDGs, which have not been found in higher mass clusters like Coma or Perseus. The UDGs in the isolation and outskirts of groups possess bluer colors than the ones in groups and clusters, which might be an indication of environmental effects.

8.4. LSB dwarfs in FDS

Most of the galaxies in our sample are not UDGs but rather dwarf-sized LSB galaxies. The effective radii of these galaxies range from 120 pc to 1.5 kpc (an upper limit by definition), and the total r′-band magnitude between −8.9 mag >M_e,r′> −15.2 mag, which means that we are reaching the sizes of the largest Milky Way satellite dSph galaxies. For comparison, Ursa Minor has R_e = 181 pc and M_V = −8.8 mag, and the Fornax dSph has R_e = 710 pc and M_V = −13.4 mag (McConnachie 2012). Having in mind that the dividing line between the LSB dwarfs and the other dwarf galaxies is physically not very clear, it is interesting to compare their radial distributions in the Fornax cluster. As the LSB dwarfs appear also in galaxy groups, we can compare the Fornax LSB dwarfs with those appearing in the Centaurus group, where a sample comparable to ours exists.

8.4.1. Radial distribution

Dynamical friction (Chandrasekhar 1943) transfers kinetic energy from massive galaxies to smaller galaxies, making the massive galaxies to sink deeper into the cluster potential well. This mass segregation takes place in time scales shorter than Hubble time (White 1977). Thus, assuming that all galaxies spend a similar amount of time in the cluster, the distribution of massive galaxies should be more centrally concentrated. Harassment should also make the galaxies themselves more centrally concentrated near the cluster center. For a dynamically relaxed cluster we would then expect that also among the dwarf galaxies the more massive and higher surface brightness galaxies preferentially appear in the central regions of the cluster. However, according to the cosmological simulations like the Millennium simulations by Boylan-Kolchin et al. (2009), clusters grow hierarchically by accreting galaxy groups, which in reality makes the situation more complicated than that.

To study if the above mentioned processes are relevant in the Fornax cluster we studied the radial number density profiles of the galaxies. We found that the bright FCC galaxies indeed are more centrally clustered than the fainter galaxies in our sample (see Fig. 13). In the same figure plotted are also ultra compact dwarf galaxies (UCDs) from the sample of Gregg et al. (2009), which is complete up to 0.9° of the Fornax cluster. UCDs are compact dwarf galaxies, with sizes similar to GCs or nuclei of dwarf galaxies. It appears that the dwarf galaxy concentration in the central regions of the Fornax cluster increases with increasing compactness of the object, so that the most concentrated are the UCDs, and the least concentrated the LSB dwarfs.

It seems that LSB dwarfs are the most numerous type of galaxies in Fornax, everywhere except in the area within 0.3° from NGC 1399 (only the area within 5 arcmin from NGC 1399 was masked). A similar phenomenon has been observed also in several other clusters (van der Burg et al. 2016; Wittmann et al. 2017). In principle this could be a bias due to difficulties in detecting the LSB galaxies in the central regions of the clusters, but it is also possible that the galaxies entering too close to the cluster center get tidally disturbed and finally accreted to the central cD galaxy. Most probably we can exclude the first possibility, because bright stars, which are the main reason for possible overlapping with the LSB galaxies, are not clustered into the cluster center. We see a drop also in the number of bright FCC galaxies which should be easily visible throughout the Fornax cluster.

8.4.2. Comparison with the Centaurus group

In the following we compare the properties of the Fornax LSB dwarfs with those in the Centaurus group, located at the distance of 4 Mpc (Jerjen et al. 2000). The main body of the Centaurus group consists of four bright galaxies (brighter than M_r′< −17 mag). The photometric measurements for the dwarf galaxies in this group in the g′- and r′-bands are provided by Müller et al. (2017). The images reach the V-band surface brightness of 28 mag arcsec^-2. With that limiting surface brightness they should be able to find all similar LSB dwarfs as detected by us in the Fornax cluster. The Centaurus group is located at suitable distance so that the analysis can be done in a similar manner as in this paper.

The size-magnitude, color–magnitude and Sérsic n-magnitude relations of the LSB dwarf galaxies in the Fornax cluster, and in the Centaurus group are shown in Fig. 21. The Centaurus group sample is complete for the sources which have diameters larger than 28 arcsec (0.6 kpc at 4 Mpc distance) at a surface brightness of 28 mag arcsec^-2. The effective radii of these galaxies range between 0.10 kpc <R_e< 1.82 kpc, and the r′-band absolute magnitudes between −7.9 mag >M_r′> −17.2 mag. To eliminate selection effects from the comparison, we removed from the Centaurus sample the galaxies with mag arcsec^-2, as these galaxies would not be included in our sample.

The g′−r′ colors and the values of Sérsic n of the Fornax and Centaurus LSB dwarf galaxies, are plotted as a function of the total r′-band magnitude in Fig. 21. The number histograms of the two parameters are shown in the small right panels of both figures. It appears that the LSB dwarf galaxy colors in both environments are fairly similar. Only difference is that the average colors of galaxies in the Centaurus group become bluer and the effective radii smaller than in Fornax with increasing magnitude. Also, the scatter in the colors of the Centaurus group LSB dwarfs is clearly higher. Notable is that there is no difference in the colors, sizes or Sérsic indices of the faintest LSB galaxies (M_r′ ≳ −12 mag).

The values of Séric n parameters of these galaxies are also similar. In both environments the Sérsic n value drops from n ~ 1 to 0.5 with decreasing galaxy magnitude. A similar trend is found also by de Rijcke et al. (2009) for Perseus, and by Misgeld et al. (2009) for the Hydra I cluster, which has a virial mass of 4.4$\begin{array}{c}\mathrm{+}\mathrm{1.1}\\ -1.0\end{array}\mathrm{\times }{\mathrm{10}}^{\mathrm{14}}{\mathit{M}}_{\mathrm{\odot }}$, thus being ≈6 times more massive than the Fornax cluster.

Indeed, there are no differences in the properties of the smallest LSB dwarf galaxies (M_r′ ≳ −12 mag) in the Fornax cluster and the Centaurus group, regardless of the very different type of environments that they represent. This is consistent with the simulations (Zolotov et al. 2012; Oñorbe et al. 2015), which predict that strong background UV – radiation and star formation feedback in the early epoch of the universe suppresses the accretion of the cold gas into the smallest dwarf galaxies. Due to their low mass, these galaxies are inefficient in accreting more cold gas later in their evolution, whereas the more massive ones can continue accreting gas (if available) and forming stars.

9. Summary and conclusions

New deep g′, r′ and i′-band images have been obtained for the central region of the Fornax cluster using the OmegaCAM instrument attached to the VST telescope at ESO. Four 1° × 1° fields of the data, covering the innermost 4 deg² of the cluster, have been analyzed in this work. The images, reaching a surface brightness of ~28 mag arcsec^-2 (for S/N = 5 over 1 arcsec²) in the r′-band, are used to compile a catalog of 205 LSB galaxies (μ_0,r′> 23 mag arcsec^-2), of which galaxies 59 are new detections. These detections complement the previous catalogs of dwarfs in the Fornax cluster, namely the Fornax Cluster Catalog by Ferguson (1989b), the Next Generation Fornax Survey catalog by Muñoz et al. (2015), and the list of extended LSB structures in the Fornax cluster by Lisker et al. (2017). Galaxies are separated from tidal structures, and for these galaxies the photometric parameters are derived, by fitting Sérsic functions to the azimuthally averaged light profiles, and by fitting the full 2D flux distributions of the galaxies using GALFIT. The reliability of the two methods is tested, and the completeness of the UDGs is evaluated by adding artificial mock galaxies to the images. Also, g′−r′ aperture colors within the effective radii of the sample galaxies have been obtained.

The following conclusions are obtained:

• 1.

  Based on visual inspection we identify nine galaxies fulfillingthe definition for UDGs by van Dokkumet al. (2015). Three of thesegalaxies are new detections. The number density of UDGsnormalized with the number of bright galaxies, is found to behigher in Fornax than in the Coma cluster (i.e. 0.7±0.2 and 0.45±0.05, respectively). However, this can be a detection bias of the Coma sample in a sense that not all UDGs are detected in Coma.

• 2.

  We made photometric measurements using GALFIT models and azimuthally averaged luminosity profiles. These two methods give consistent results in terms of effective radii (R_e), total r′-band magnitudes (m_r′), and Sérsic n-values. We find that the GALFIT models are more reliable for obtaining the ellipticities (b/a), and are therefore used in the analysis of this paper.

• 3.

  Assuming that the UDGs in our sample are located at the distance of Fornax they have −15.8 mag <M_r′< −13.5 mag and 1.6 kpc <R_e< 11 kpc, which are typical sizes of UDGs found in other environments. However, we find a larger fraction of large UDGs (R_e> 3 kpc) than was found in the Coma cluster (Yagi et al. 2016), and in the distant clusters (van der Burg et al. 2016). As those studies, both using SExtractor, are not as effective as our visual inspection in detecting large UDGs, we can not exclude the possibility of the difference being a selection bias.

• 4.

  The five small UDGs with R_e< 3 kpc detected in the Fornax cluster by us, are similar to the typical LSB dwarfs in terms of their magnitude, color and shape, being consistent with them having common origin. The four largest UDGs (with R_e> 3 kpc) appear as outliers having ten times larger effective radii than the typical LSB dwarfs of the same galaxy luminosities. Additionally, the large UDGs in our sample are significantly more elongated (0.2 <b/a< 0.4) than the typical LSB dwarfs, in our sample and in Coma, and they also show signs of tidal interactions.

• 5.

  Contrary to the UDG orientations in Coma, we find that the LSB dwarfs in the Fornax cluster are randomly oriented, without any preferences neither toward the cluster center, nor toward their nearby bright galaxies. However, the most elongated galaxies show a preferred alignment toward their nearest bright galaxies.

• 6.

  We find that the LSB galaxies in the center of the Fornax cluster follow a color–magnitude relation with a slope, which is same as the one of the red sequence in the Virgo cluster (Kim et al. 2010; Roediger et al. 2017). The UDGs follow the same relation as the smaller LSB dwarfs, although there is one clear outlier toward blue.

• 7.

  Our observations are in agreement with both the high-spin and the strong feedback models of UDG formation, where they are seen as an extension of the dwarf galaxy population. However, the large and elongated UDGs in our sample are likely indications that the tidal forces can also produce some of these UDGs, at least their largest, most elongated representatives.

• 8.

  The LSB dwarf galaxies in our sample have magnitudes in the range of −15.4 mag <m_r′< −8.9 mag (with the mean of −11.4 mag) and effective radii between 0.14 kpc <R_e< 1.5 kpc (with the mean values of 0.5 kpc). Their colors and profile shapes were compared with the ones of the LSB dwarf galaxies in the Centaurus group (Müller et al. 2017): they were found to be similar for the faintest LSB galaxies (M_r′ ≳ −12 mag). However, the LSB galaxies in the Centaurus become bluer and smaller with increasing magnitude, when compared to the ones in the Fornax cluster.

• 9.

  We find that the distribution of LSB dwarf galaxies in the Fornax cluster is less centrally concentrated than the one of the bright galaxies with M_r′< −17 mag. Also, the number density of LSB galaxies drops at a cluster-centric distance of r = 0.6° (180 kpc) inward, which can be an indication of tidal disruption of galaxies that enter close to the cluster center. The observed drop in the number of the LSB galaxies in the center of the Fornax cluster is consistent with the findings in other clusters, showing that it is a common phenomenon in galaxy clusters (Wittmann et al. 2017; van der Burg et al. 2016).

---

Acknowledgments

A.V. acknowledge the support from grant from the Vilho, Yrjö and Kalle Väisälä Foundation. J.F.-B. and R.P. acknowledge support from grant AYA2016-77237-C3-1-P from the Spanish Ministry of Economy and Competitiveness (MINECO). A.V., E.L. and H.S. acknowledge the support from the academy of Finland (grant No. 297738). This paper has benefited strongly from discussions within the SUNDIAL ITN network, a EU Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 721463. We would also like to thank the anonymous referee, whose detailed comments and suggestions helped to significantly improve this paper.

References

Appendix A: Pixel value distribution moments.

The center and initial shape parameters of a galaxy are measured by calculating the first and second moments of the galaxy’s pixel value distribution. The pixels from which the moments are calculated are selected by manually placing an elliptical aperture over the galaxy that follows the apparent shape of the galaxy and covers the visible part of the galaxy. If we mark x- and y-coordinates and intensity of a pixel i as x_i,y_i and I_i, respectively, we can calculate the first moments of pixel value distribution as follows $\begin{array}{ccc}\overline{\mathit{x}}& \mathrm{=}& \frac{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}{\mathit{x}}_{\mathit{i}}}{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}}\mathit{,}\\ \overline{\mathit{y}}& \mathrm{=}& \frac{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}{\mathit{y}}_{\mathit{i}}}{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}}\mathrm{·}\end{array}$(A.1)The first moments correspond to the weighted center of the pixel value distribution. The second order moments $\begin{array}{ccc}\overline{{\mathit{x}}^{\mathrm{2}}}& \mathrm{=}& \frac{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}{\mathit{x}}_{\mathit{i}}^{\mathrm{2}}}{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}}\mathrm{-}{\overline{\mathit{x}}}^{\mathrm{2}}\mathit{,}\\ \overline{{\mathit{y}}^{\mathrm{2}}}& \mathrm{=}& \frac{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}{\mathit{y}}_{\mathit{i}}^{\mathrm{2}}}{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}}\mathrm{-}{\overline{\mathit{y}}}^{\mathrm{2}}\mathit{,}\\ \overline{\mathit{xy}}& \mathrm{=}& \frac{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}{\mathit{x}}_{\mathit{i}}{\mathit{y}}_{\mathit{i}}}{{\sum }_{\mathit{i}}{\mathit{I}}_{\mathit{i}}}\mathrm{-}\overline{\mathit{x}}\hspace{0.17em}\overline{\mathit{y}}\mathit{,}\end{array}$(A.2)describe the spread of the distribution along the axes. They can be transformed to semi-minor axis b and semi-major axis a as $\begin{array}{ccc}{\mathit{a}}^{\mathrm{2}}& \mathrm{=}& \frac{\overline{{\mathit{x}}^{\mathrm{2}}}\mathrm{+}\overline{{\mathit{y}}^{\mathrm{2}}}}{\mathrm{2}}\mathrm{+}\sqrt{{\left(\frac{\overline{{\mathit{x}}^{\mathrm{2}}}\mathrm{-}\overline{{\mathit{y}}^{\mathrm{2}}}}{\mathrm{2}}\right)}^{\mathrm{2}}\mathrm{+}{\overline{\mathit{xy}}}^{\mathrm{2}}}\mathit{,}\\ {\mathit{b}}^{\mathrm{2}}& \mathrm{=}& \frac{\overline{{\mathit{x}}^{\mathrm{2}}}\mathrm{+}\overline{{\mathit{y}}^{\mathrm{2}}}}{\mathrm{2}}\mathrm{-}\sqrt{{\left(\frac{\overline{{\mathit{x}}^{\mathrm{2}}}\mathrm{-}\overline{{\mathit{y}}^{\mathrm{2}}}}{\mathrm{2}}\right)}^{\mathrm{2}}\mathrm{+}{\overline{\mathit{xy}}}^{\mathrm{2}}}\mathit{,}\end{array}$(A.3)and the position angle θ$\tan \mathrm{\left(}\mathrm{2}\mathit{\theta }\mathrm{\right)}\mathrm{=}\mathrm{2}\frac{\overline{\mathit{xy}}}{\overline{{\mathit{x}}^{\mathrm{2}}}\mathrm{-}\overline{{\mathit{y}}^{\mathrm{2}}}}\mathrm{·}$(A.4)

Appendix B: Mock galaxy parameters

The mock galaxies are generated by using 2D Sérsic functions with the chosen structural parameters and random position angles. Also, the Poisson noise is added to the galaxies, and the models are then convolved with the OmegaCAM PSF (see Sect. 5.2 for details) before adding them to the science mosaics.

The parameters of the UDG mock galaxies used for testing the detection efficiency are given in Table B.1. Also the indications of the detections in different fields are given in Table B.1. The additional ~150 mock galaxies used for testing the photometric accuracy have m_r′ = 15−24 mag, R_e = 5−20 arcsec, Sérsic n = 0.3−2 and b/a = 0.2−1. The differences between the

input model and measured parameters for all the mock galaxies are shown in the Fig. 11.

Appendix C: Transformations between photometric filters

To transform the Subaru Suprime-Cam B and R magnitudes into the r′- and g′-band values we used the transformations from Yagi et al. (2010). The r′ band magnitude m_r′ is ${\mathit{m}}_{{\mathit{r}}^{\mathrm{\prime }}}\mathrm{=}{\mathit{m}}_{\mathit{R}}\mathrm{-}\mathrm{0.0188}\mathrm{+}\mathrm{0.1492}\mathrm{\times }\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)\mathrm{-}\mathrm{0.0128}\mathrm{\times }{\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)}^{\mathrm{2}}\mathit{,}$(C.1)where m_R is the apparent magnitude in Suprime-Cam R-band. Also, the Suprime-Cam B−R color can be transformed into g′−r′ color as follows: $\mathit{B}\mathrm{-}\mathit{R}\mathrm{=}\mathrm{0.00440}\mathrm{+}\mathrm{1.32386}\mathrm{\times }\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)\mathrm{+}\mathrm{0.0010734}\mathrm{\times }{\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)}^{\mathrm{2}}\mathit{.}$(C.2)For the transformations between the Johnson U,B,V,R and I and SDSS g′, r′ and i′ colors we used the equations given in the SDSS website by Lupton (2005¹³): ${\mathit{m}}_{\mathit{V}}\mathrm{=}{\mathit{m}}_{{\mathit{r}}^{\mathrm{\prime }}}\mathrm{+}\mathrm{0.4216}\mathrm{\times }\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)\mathrm{-}\mathrm{0.0038}\mathit{,}$(C.3)where m_V and m_r′ are the apparent magnitudes in V- and r′-filters, respectively, and g′−r′ is g′−r′ color. Similarly: ${\mathit{m}}_{\mathit{B}}\mathrm{=}{\mathit{m}}_{{\mathit{r}}^{\mathrm{\prime }}}\mathrm{+}\mathrm{1.33130}\mathrm{\times }\left({\mathit{g}}^{\mathrm{\prime }}\mathrm{-}{\mathit{r}}^{\mathrm{\prime }}\right)\mathrm{+}\mathrm{0.2271}\mathit{,}$(C.4)where m_B is the apparent magnitude in B-band.

Appendix D: List of galaxies

We list the properties of the galaxies in our sample In Table D.1.

All Tables

All Figures