© ESO 2022

1. Introduction

In the last decade, the deployment of wide field imaging facilities with large collecting areas has improved our ability to study low surface brightness (LSB) galaxies¹. Several deep surveys, such as the Next Generation Virgo Survey (NGVS; Ferrarese et al. 2012), the Kilo Degree Survey (KiDS; de Jong et al. 2015), the VST Early-type GAlaxy Survey (VEGAS; Capaccioli et al. 2015), and the Fornax Deep Survey (FDS; Iodice et al. 2016; Venhola et al. 2018, hereafter V18), have gathered large amounts of data that reveal previously unseen LSB galaxies (Muñoz et al. 2015; Koda et al. 2015; van der Burg et al. 2016; Chamba et al. 2020; Prole et al. 2019). These galaxies, having an extremely diffuse structure and existing in dense environments, allow us to test our galaxy formation theories in a so far unexplored and unique parameter space. Ultimately, the number and properties of these galaxies will allow us to probe their formation mechanisms and can be a benchmark for our cosmological models.

So far, we have not reached any lower limit in surface brightness (SB) where we stop finding new galaxies (see, e.g., Fattahi et al. 2020). In addition to not knowing how many galaxies we fail to detect, spectroscopic instruments cannot reach similar depths as effectively as the imaging instruments. This limits our understanding of LSB galaxies – the confirmation of galaxy cluster membership by radial velocities is made difficult, and thus the contribution of the LSB galaxies to the galaxy luminosity function (LF), and their properties in general, is not well known. As there exists a luminosity-SB relation for galaxies (Binggeli et al. 1984; Kormendy & Bender 2012), most of the galaxies that have low SBs are low-mass dwarfs, but massive LSB galaxies also exist (Romanishin et al. 1983; Bothun et al. 1985; Impey & Bothun 1989, 1997).

Massive LSB galaxies are typically blue, slowly star-forming spiral galaxies (McGaugh & Bothun 1994). In the case of dwarfs, there are both blue star-forming and red quiescent LSB dwarf galaxies (Leisman et al. 2017; Román & Trujillo 2017). Low surface brightness dwarfs follow the morphology-density relation (Dressler 1980), such that the red LSB dwarfs are mostly in dense environments and the blue LSB dwarfs in sparse ones (Leisman et al. 2017; Román & Trujillo 2017). In this study we concentrate on the analysis of the low-mass LSB galaxies in the Fornax cluster. Thus, most of the galaxies are expected to be quiescent and gas poor.

A subgroup of LSB dwarf galaxies called ultra-diffuse galaxies (UDGs; Sandage & Binggeli 1984; van Dokkum et al. 2015), which are defined by having large effective radii (R_e > 1.5 kpc), has been studied in several environments in order to understand what causes their large sizes and low SBs (Di Cintio et al. 2017; Zaritsky et al. 2019; Merritt et al. 2016; Mancera Piña et al. 2018; van der Burg et al. 2016; Mihos et al. 2015; Venhola et al. 2017; Wittmann et al. 2016; Iodice et al. 2020; Leisman et al. 2017). These studies have found little evidence of UDGs constituting a distinct population of dwarf galaxies (Conselice 2018; Koda et al. 2015; Chilingarian et al. 2019; Mancera Piña et al. 2019; Lim et al. 2020); they instead point to them being the diffuse end in the continuum of dwarf galaxy properties.

Efforts to try to explain the origin of UDGs have led to progress in understanding different external and internal mechanisms that make some dwarf galaxies of a given mass have lower SBs than others. Ultra-diffuse galaxies and other LSB dwarfs tend to form naturally in modern simulations: The large angular momentum of dark matter (DM) halos and gas leads to large galaxies with low SBs (Amorisco & Loeb 2016; Tremmel et al. 2020). The dependence between the rotation of the DM halos and the density of the environment might lead to an increased fraction of LSB dwarf galaxies in high density environments (Tremmel et al. 2020). However, star formation suppression due to feedback can also cause the low SB of the low-mass galaxies (Di Cintio et al. 2017, 2019; Jiang et al. 2019), and not all simulations find a dependence between the SB and the rotation of the DM halo or the stellar body of low-mass galaxies (Jiang et al. 2019; Di Cintio et al. 2019). Thus, it seems that there are different mechanisms that can lead to the formation of LSB galaxies independent of the density of their formation environment.

According to simulations, some LSB dwarf galaxies also form during and after the infall of galaxies into a cluster. Quenching of the star formation by ram-pressure stripping (RPS) during the infall of dwarf galaxies leads to passively aging stellar populations. After the brightest and most massive stars end their nuclear burning phase, the SB of the galaxies fades over time if no new stars are born. This leads to some dwarf galaxies with high SB during their infall to a denser environment fading into LSB dwarfs and UDGs (Tremmel et al. 2020; Jiang et al. 2019). This process also leads to the SB-age relation predicted by simulations (Tremmel et al. 2020). As the galaxies are dragged toward the center of the cluster over time due to tidal friction, galaxies that have fallen into a cluster early are now likelier to appear as LSB galaxies near the center of the cluster relative to the recently in-fallen population (Tremmel et al. 2020; Jiang et al. 2019). Hence, the fraction of LSB dwarf galaxies with respect to non-LSB dwarfs should increase toward the cluster center (Tremmel et al. 2020). In addition to the decrease in SB due to the aging of the stellar populations, the surface density (SD) of galaxies that travel near the cluster center during their pericenter can also decrease by as much as 30% due to the tidal field of the cluster (Tremmel et al. 2020; Sales et al. 2020). This decrease in the SD happens when DM is stripped from the halo of the galaxies and, as a result, the stellar body of the galaxy expands due to the changes in the gravitational potential.

From an observational point of view, a lot of work has been done in order to identify UDGs and other LSB dwarfs. Their distribution and abundance has been analyzed within and outside of groups and clusters. The number of UDGs is found to be proportional to the mass of the cluster (or group), and they appear to follow a cored spatial distribution in the cluster similar to that of other dwarf galaxies (van der Burg et al. 2016; Mancera Piña et al. 2018; Wittmann et al. 2016). The concentration of the UDGs’ light distribution and their projected shapes are very similar to those of dwarf galaxies of the same mass but higher SB (Mancera Piña et al. 2019). They also have a bimodality in their color distribution similar to other dwarf galaxies (Román & Trujillo 2017). From spectroscopic studies of small samples of UDGs, we know that their stellar populations and kinematics are similar to those of other dwarfs (Chilingarian et al. 2019; Ruiz-Lara et al. 2018).

Regardless of the effort in identifying UDGs and other LSB dwarfs and comparing their properties with other galaxies in other environments, less work has been done to set the LSB dwarfs in the context of a complete dwarf galaxy population in any given environment. This is clearly necessary in order to understand how the LSB dwarfs are different from other galaxies. With a complete dwarf galaxy population, it is straightforward to test whether we find the observable relations found in the simulations, for example, an age-SB relation, a correlation between the cluster-centric distances and sizes of the galaxies, or differences between the distribution of UDGs and other dwarfs.

Recently, Lim et al. (2020) studied the UDGs in the Virgo cluster within a complete galaxy sample and showed that they are a normal diffuse end tail of the cluster dwarf galaxy population. Contrary to what was thought based on previous studies (e.g., Mancera Piña et al. 2018), they also found that the UDGs are more centrally clustered in the cluster than other dwarfs, in agreement with simulation predictions (Sales et al. 2020). Their study clearly demonstrated how important it is to place UDGs in the context of a complete galaxy population.

In order to generate a useful galaxy catalog for the abovementioned purpose, we need to identify and quantify galaxies in a robust, homogeneous manner. Many galaxy catalogs compiled from the data of new surveys are produced by identifying the galaxies either visually or automatically using Source Extractor (SE; Bertin & Arnouts 1996). The problems related to those methods are well known and have been pointed out by, for example, Akhlaghi & Ichikawa (2015) and Venhola et al. (2017). At present, many better methods have become available, such as Max-Tree Objects (MTO; Teeninga et al. 2016), NoiseChisel (Akhlaghi & Ichikawa 2015), and ProFound (Robotham et al. 2018). A quantitative comparison between the novel detection algorithms by Haigh et al. (2021) showed that, when optimized for detection completeness and accuracy of the measured parameters, MTO slightly outperforms NoiseChisel and others. These improved algorithms have not yet been widely used; for example, the statistical studies of UDGs (Yagi et al. 2016; van der Burg et al. 2016; Mancera Piña et al. 2018) are all based on catalogs obtained with SE. However, some new studies have already started using, for example, MTO (Prole et al. 2019; Chamba et al. 2020; Müller & Schnider 2021). With the new detection algorithms, we can reduce the systematic biases in the LSB samples and, moreover, obtain more complete catalogs from the currently available data sets.

In the ground-based observations of LSB dwarfs, seeing often sets a limit for the minimum size of the objects that we can separate from background galaxies. Hence, studying a nearby galaxy cluster allows us to resolve small galaxies best. Being the second closest cluster to us and being covered with deep, publicly available survey data, the Fornax cluster is an excellent environment for studying the faintest dwarfs. The basic properties of the Fornax cluster are listed in Table 1.

In this study we apply MTO to detect LSB dwarf galaxies in the Fornax cluster. A few studies have mapped the LSB galaxy population in the Fornax cluster before: Bothun et al. (1991), Hilker et al. (1999, 2003), Mieske et al. (2007), Muñoz et al. (2015), Venhola et al. (2017), Eigenthaler et al. (2018), Ordenes-Briceño et al. (2018), and V18. All of these studies identify objects either visually or by applying SE and are mostly concentrated on the central area of the cluster. In this work we aim to extend the FDS dwarf galaxy catalogs in order to obtain a homogeneous and complete sample of dwarf galaxies. Venhola et al. (2017) visually studied the LSB galaxy population in the 4 deg² area around the center of the cluster from the FDS data. The faintest objects in that catalog reach ${\overline{\mu }}_{e,r^{\prime }}$ ≈ 28 mag arcsec⁻² and have sizes of up to R_e ≈ 10 kpc, making it the most complete compilation in that area of the Fornax cluster. V18 published a spatially complete catalog of dwarfs for the whole 26 deg² area of the FDS using SE for the detection, but that catalog misses many LSB galaxies. Throughout the paper, we refer to these works as the core LSB catalog (Venhola et al. 2017) and the Fornax Deep Survey Dwarf Catalog (FDSDC; V18). In this work we aim to increase the LSB completeness of the FDSDC to a depth similar to that of the core LSB catalog.

In this study we first describe the FDS and a simulated data set (Sect. 2) that we use for the quality assessment of our object detection (Sect. 3). We apply MTO to detect LSB galaxies in the full FDS data set and fit light profiles of the detected galaxies using GALFIT (Peng et al. 2010) (Sect. 4). The resulting LSB candidates are separated into cluster members and background objects (Sect. 5), and the distribution and properties of the cluster LSB galaxies are analyzed (Sect. 6). We then discuss how the newly identified LSB dwarf population relates to the previously known dwarf populations in the Fornax cluster and other environments and simulations in Sect. 7, and we summarize our results in Sect. 8.

Throughout the paper we use the distance of 20.0 Mpc for the Fornax cluster, which corresponds to the distance modulus of 31.51 mag (Blakeslee et al. 2009). At this distance, 1 deg corresponds to 0.349 Mpc and 1 arcsec corresponds to 100 pc.

2. Data and existing catalogs

We aim to detect as many as possible bona fide LSB dwarf galaxies from the FDS data set. Moreover, in order to test the completeness and biases of our detection algorithm, we require a data set with a known ground truth and similar image quality to the FDS. For that purpose we generated artificial images that resemble the FDS images. These data sets are described in the following subsections.

2.1. The Fornax Deep Survey data

The ESO VLT Survey Telescope (VST; Schipani et al. 2012) observations of the FDS cover a 21 deg² area centered on the Fornax main cluster in the u′,g′,r′ and i′ bands, and an additional 5 deg² area centered on the infalling Fornax A subgroup, observed in the g′, r′, and i′ bands. The data are collected with the OmegaCAM instrument, which is a 1 deg² field-of-view 32-CCD camera with a 0.21 arcsec pixel size². The reduction and calibration steps of the data are described by V18.

The photometric uncertainties of the data were estimated by V18, and they are 0.04, 0.03, 0.03, and 0.04 mag in u′,g′,r′ and i′, respectively. Fields are covered with a nearly homogeneous depth with the 1σ limiting SB over 1 pixel (0.2 arcsec × 0.2 arcsec) of 26.6, 26.7, 26.1 and 25.5 mag arcsec⁻² in u′,g′,r′ and i′, respectively. When averaged over 1 arcsec² those limits correspond to limiting SBs of 28.3, 28.4, 27.8, 27.2 mag arcsec⁻² in u′,g′,r′, and i′, respectively.

The FDS provides two photometric galaxy catalogs of dwarfs and separate measurements for the giant galaxies (Iodice et al. 2019; Spavone et al. 2020; Raj et al. 2019, 2020). The FDSDC, which contains all the dwarf galaxies with semimajor axis a > 2 arcsec in the whole FDS, was generated by V18. The 50% completeness limit of this catalog is at the r′-band mean effective SB of ${\overline{\mu }}_{e,r^{\prime }}$ ≈ 26 mag arcsec⁻². Venhola et al. (2017) presented an LSB galaxy catalog that was made by visually identifying LSB galaxies from the 4 deg² in the center of the cluster. For the next steps we assume that this core LSB catalog is complete within the depth limits of the data, as it was made by inspecting the images manually and selecting all LSB objects. In this work we utilize the core LSB catalog and FDSDC for testing the detection completeness obtained when using MTO and for the base catalogs in the analysis.

2.2. Simulated images

For generating the mock images, we used the background noise and point spread function (PSF) model of the r′-band FDS field 11 described by V18. The artificial observations were produced by first generating a blank image with a size of 10 000 × 10 000 pixels (corresponding to approximately a quarter of a single FDS field).

We then injected point sources representing stars to the image, using an apparent luminosity distribution drawn from an exponential distribution N ∝ exp(m_r′/3) between object magnitudes of 12 mag < m_r′ < 27 mag. This distribution and the number density of the stars was selected to be similar to that found from the FDS images based on a SE detection of stars in the Field 11. We then embedded Sérsic models at random positions to the image as mock galaxies, which we consider to be representative for both the cluster and background galaxies. The number of background and cluster galaxies were 4000 and 50, respectively, per image. The structural parameter ranges of the mock galaxies were matched to the values found for the cluster and background galaxies³ in the catalogs of V18, but the SB distribution was extended down to ${\overline{\mu }}_{e,r^{\prime }}$ = 31 mag arcsec⁻².

After embedding all the objects in units of electron counts, the image was convolved with the OmegaCAM r′-band PSF (V18), and the Poisson noise was applied. Finally, Gaussian background noise was added to the pixels, and the pixel values were divided by the gain factor. We generated 110 such mock images, which we used to assess the detection completeness of MTO and the accuracy of the photometric measurements (see Sects. 3.2 and 4).

We acknowledge that these simple mock images cannot reproduce all the complexity of the real data (e.g., different galaxy morphologies, globular clusters in Fornax galaxies, reduction artifacts, PSF variations, reflections, satellite tracks etc.), which might lead to some biases in the analysis of the completeness limits. To ensure that we nevertheless obtain good estimates for the completeness limits we also test the detection with real images (using the core LSB catalog). However, the main benefit of using the simulated data is that we can be certain about the ground truth of the images, and thus we are able to objectively quantify the accuracy of the detections and the photometry.

3. LSB object identification

We use MTO for identifying LSB objects. MTO is an object detection program that utilizes maximum trees (Salembier et al. 1998; Souza et al. 2016) for identifying structures in an image and keeps track of the hierarchical structure of the identified objects. This approach allows the program to deal with nested objects (i.e., objects that appear on top of each other), which is especially important in the cluster environment where the object density is high.

Haigh et al. (2021) have shown that MTO outperforms other common source detection programs in completeness and purity. In their paper, Haigh et al. used a set of simulated and real images with known ground truths to test NoiseChisel, Profound, SE, and MTO. They optimized the programs using a combined completeness and area score, and found that MTO obtains highest score for both measures. They also found that MTO does not require a tuning of parameters when changing from a data set to another, whereas for other algorithms the optimal set of parameters changed between the data sets. A caveat of MTO is that it is much slower than SE: It takes about 20 min to analyze one FDS field (21 kpix × 21 kpix) with MTO using a typical desktop computer, whereas with SE it takes less than half a minute. However, as the entirety of the FDS data is only tens of gigapixels, the slower processing time is not a problem. As Haigh et al. have done these comparisons in detail (see also Venhola 2019, Ph.D. thesis), we do not perform further comparisons between the other different detection algorithms, instead focusing on testing the performance of MTO using the simulated and original FDS data and comparing it with the results obtained when using SE⁴.

3.1. Detection of LSB galaxies using MTO

The algorithm of MTO has been described in detail by Teeninga et al. (2016). However, for completeness we give a short overview of the detection process here.

MTO assumes that an input image, with a gain of g, consists of three different components (all in fluxes): Gaussian background noise with variance g⁻¹B, a Poisson distributed signal g⁻¹O, and other noise sources R, such as readout noise. The background is assumed to be flat over the whole image, thus being well approximated by the mean of the pixel values of the pixels devoid of object signal. The five main steps of the algorithm are as follows.

The first is background estimation: The image is first divided into 64 × 64 pixel (12.8 × 12.8 arcsec²) tiles. In each tile statistical tests are made to test the Gaussianity and flatness of the tiles. The Gaussianity test is done using the D’Agostino-Pearson K²-statistic. Then the tiles are tested for absence of gradients (flatness) by comparing the means within the different tile quadrants with each other using a t-test. Depending on whether any tiles pass the background tests, the common tile size is increased or decreased by a factor of two. The background size is then selected to be the largest scale in which tiles pass those tests. The tiles passing these two tests are then selected to represent the background, and the mean value of those tiles is subtracted from the image. Background variance g⁻¹B is also measured from the selected tiles. The background becomes the first node of the tree.

The second is the identification of significant branches: The image is thresholded with increasing threshold levels, and at each level the maximum connected structures are identified. The significance of each structure is tested by calculating its power⁵, and it is considered as a significant branch if its power exceeds the limit α. We used the default value for α that rejects the detections that are less significant than 5σ. The hierarchical structure of the branches is then stored into a tree.

The third is the labeling of the branches: The branches in the tree are labeled so that the branches that belong to the same object obtain the same label. In the case of nodes (i.e., several branches sharing the same parent), the branch with the most power obtains the same label as its parent, and all the other branches obtain new labels.

The fourth is the generation of the segmentation map: A segmentation map is produced by projecting the tree into a plane with same dimensions as the input image. The labels for the segmentation map are taken from the highest level branch that is present in the given pixel.

The fifth is parameter extraction: Structural parameters for each object are measured from the input image. These parameters are calculated using the pixels that have the corresponding label in the segmentation image. For nested objects, the intensity level of the parent node is subtracted from the object’s pixel values before calculating the parameters. This works as a local background estimation and thus prevents the measured SB of the nested objects from being biased by their parent objects. Parameters that we use in this work are the object center coordinates that are calculated from the flux weighted mean positions of the pixels. Total magnitudes are calculated as a sum of the flux in those pixels. For the measure of size, we use the length of semiminor and semimajor axes, which are calculated from the pixel value distribution moments (see Appendix A.1).

For a given input image, MTO outputs a segmentation map and an object list that includes the locations of the detected objects and their magnitude, size, and median SB.

3.2. Detection completeness

In this section we assess the detection completeness and parameter extraction accuracy of MTO using artificial and real images (described in Sects. 2.1 and 2.2). In detail, we test the detection completeness of MTO and the accuracy of the output parameters of the detected objects. The latter test is crucial despite the fact that we will later perform a more elaborate photometry for the objects since the MTO magnitudes and effective radii of the detected objects will be used when the objects of interest are selected among the large number of detections.

3.2.1. Completeness with mock galaxies

We ran MTO for the 110 simulated fields (see Sect. 2.2) and compared the detections with the known locations and parameters of the simulated galaxies (i.e., the ground truth). In order to have a comparison for the quality assessment, we also ran the detections using SE. For SE, we optimized the detection parameters by running the SE-detection for simulated images using different parameters and chose the set of parameters that produced the highest F − score. Like Haigh et al. (2021), we decided to use F − score for the optimization as it takes into account both completeness (the proportion of objects that are detected) and precision (the proportion of detections that can be matched to real objects). F − score is defined as follows:

$$\begin{array}{c}\hfill F-score=2\times \frac{precision\times completeness}{precision+completeness}.\end{array}$$(1)

The parameter values we obtained using the optimization are similar to those obtained by Haigh et al. (2021), although we used larger background size than they did. Details of the set of parameters tested and the used parameters are listed in the Appendix A.2.

When counting the number of detections, we defined a mock galaxy to be detected if there is an object in the output catalog with semimajor axis a > 2 arcsec and a > 0.1 × R_e, in located within half R_e from the center of the embedded mock galaxy⁶. In the case of nested galaxies or overlapping stars, this rule leads to some false positive detections, but by inspecting the detections we found that this effect is minor (few percent).

Completeness of the SE and MTO detections are shown as a function of the input r′-band mean effective SB (${\overline{\mu }}_{e,r}$) and effective radius (R_e) in the upper panels of Fig. 1. Compared to SE, MTO extends the detection completeness both toward larger and smaller R_e and toward fainter SB. The improvements in the detection of galaxies with lower SB and larger size are expected due to the improved background estimation of MTO with respect to SE and the continuous thresholding (Teeninga et al. 2016). The improvement in the completeness of objects with small R_e is caused by the selection criteria of objects: more accurate detection of object outskirts leads to more accurate size estimations for the objects. If an object with low SB and small R_e is detected only in its inner parts, the resulting size is underestimated and the object is then excluded by the selection limits due to its small size.

Fig. 1. Results of detection completeness tests. Detection completeness of SE (left panels) and MTO (right panels) as a function of the input r′-band mean effective SB (${\overline{\mu }}_{e,r}$), effective radius (Re), and axis ratio (b/a). Upper and middle panels: results for the artificial images and the lower panels for the known core LSB catalog galaxies (Venhola et al. 2017). The gray scales in the upper and middle panels correspond to the completeness indicated by the color bar. The green and blue contours show the 75% and 50% completeness limits, respectively. Those same lines are also shown in the lower panels. In the lower panels, squares show the detected galaxies and red triangles show the non-detections.

We tested the detection completeness also against the projected shape of the galaxy (see Fig. 1). We find no preferable selection of elongated nor round galaxies. In the Appendix A.3, we show the detection completeness as a function of all the input structural parameters (Fig. A.2). We found no significant biases except that the very large and very small R_e galaxies are poorly detected for ${\overline{\mu }}_{e,r^{\prime }}>25.5$ mag arcsec⁻².

As we know the input effective radii and magnitudes of the mock galaxies, we can compare those with the parameters extracted by MTO and SE. With MTO and SE we did not measure the effective radii but rather the semimajor axes based on second-order moments of the pixel value distribution of the objects (see Appendix A.1). This means that we need to compare those two different measures of size in order to understand how well the extents of the objects are measured. We note that for typical dwarf galaxies with exponential profiles, R_e ≈ a (see Fig. A.1). In Fig. 2, we show the comparison of the input versus output values of both methods as a function of the input SB of the objects. The trends for the two methods are qualitatively similar showing a decrease in the extracted object sizes and apparent magnitudes toward lower SBs. Both of these methods underestimate the sizes and measure too faint apparent magnitudes for the lowest SB objects, but SE underestimates the values more severely. In the case of MTO, the systematic underestimation of luminosity goes from 0.1 mag to 1 mag in the ${\overline{\mu }}_{e,r^{\prime }}$ range from 24 to 27 mag arcsec⁻². It is also important to take into account the fact that MTO detects approximately twice the number of objects the SE detects, which means that its trends get affected by objects that SE fails to detect. To take this into account, in Fig. 3 we show the comparison of the parameters obtained by SE and MTO for the objects that both programs have detected. Clear trends are visible both in object sizes and apparent magnitudes, so SE measures fainter magnitudes and smaller sizes for the objects than MTO. Those differences become especially significant for the objects fainter than ${\overline{\mu }}_{e,r^{\prime }}$ > 24 mag arcsec⁻² since there is ≈30% systematic difference in the sizes and 0.5 mag difference in the apparent magnitudes.

Fig. 2. Accuracy of parameters for initial detections. Upper panels: ratio between the measured semimajor axis length and the input effective radii (aout/Re, in) for the mock galaxies as a function of their mean effective SB in the r′ band (${\overline{\mu }}_{e,r^{\prime }}$). Lower panels: difference between the input magnitudes (min) and the measured magnitudes (mout). Left and right panels: comparisons when using SE and MTO, respectively. The blue (MTO) and red (SE) lines correspond to the running means of the measurements with a filter size of $\mathrm{\Delta }{\overline{\mu }}_{e,r^{\prime }}$ = 1 mag arcsec−2. Shaded areas around the running means show the median absolute deviation from the means with the corresponding colors. The gray and black symbols in the right panels correspond to the objects that are only detected using MTO and to the objects detected using both methods, respectively.

Fig. 3. Comparison of the extracted parameters between SE and MTO. Left panel: ratio between the circularized radii of the objects obtained by SE (${\sqrt{\mathit{ab}}}_{\mathrm{SEx}}$) and by MTO (${\sqrt{\mathit{ab}}}_{\mathrm{MTO}}$) as a function of the input r′-band mean effective SB (${\overline{\mu }}_{e,r^{\prime }}$). Right panel: difference between the apparent magnitudes obtained with SE (mSEx) and MTO (mMTO). The red lines show the running medians of the measurements with a filter size of $\mathrm{\Delta }{\overline{\mu }}_{e,r^{\prime }}$ = 1 mag arcsec−2.

In summary, we found that MTO obtains approximately one magnitude deeper completeness limits than SE. Additionally, it measures the size and luminosity parameters of the galaxies more accurately than SE. Despite the fact that MTO improves the accuracy of extracted magnitudes with respect to SE, it still measures magnitudes that are 1–2 mag too faint for many objects (lower-right panel of Fig. 2), and thus more accurate photometric measurements are needed for the selected objects. At least two factors contribute to the bias in the MTO photometry: The outer parts of the objects that are nested with brighter objects will be associated with the brighter ones as no models are assumed for the objects. Additionally, the outer parts of the faintest objects are lost in the noise.

3.2.2. Completeness with the FDS data

In order to test the detection completeness also with real data and confirm the completeness limits that we obtained, we use the core LSB catalog. This catalog is assumed to reach the depth limits of the FDS data, as it contains all the LSB galaxies that can be seen in the images by visual inspection.

In order to perform a quantitative test, we run MTO and SE for the 4 deg² area in the center of the Fornax cluster that is covered both by the core LSB catalog and the FDSDC. We then compare the object lists produced by each of the methods with those catalogs. We adopted the same rules for counting the detections as we did with the artificial galaxies. In the lower panels of Fig. 1, we show the R_e and ${\overline{\mu }}_{e,r^{\prime }}$ distribution of the LSB galaxies, and indicate the objects detected by each of the methods in the two different panels.

MTO and SE detected 205 and 176 objects, respectively, of the 244 LSB galaxies. Similarly as was demonstrated with the simulated data, we found also with the real data that MTO was able to increase the completeness limit of the detections toward lower SBs, and both toward larger and smaller effective radii. The completeness limits obtained with the mock images match well those obtained using real detected galaxies in the ${\overline{\mu }}_{e,r^{\prime }}-R_e$-parameter space. As can be seen in the lower panels of Fig. 1, most of the missed objects are faint dwarfs with small effective radii. This is expected since we showed (Fig. 2) that the underestimation of galaxy sizes becomes more severe toward lower SBs.

In the bottom right panel of Fig. 1, some (N = 8) galaxies with ${\overline{\mu }}_{e,r}<26$ mag arcsec⁻² and R_e > 4 arcsec are missed by MTO. In principle, such galaxies should be easily detectable in the images. We inspected why they are not identified in our test. In Fig. 4 we show postage stamps of such galaxies and the corresponding segmentation maps. All of those galaxies are detected by MTO, but do not pass our test, as the center of the detection is too much off from the dwarf in the FDS catalogs. This effect is caused by the mixing of the diffuse outskirts of the objects. Based on their size and SB, all these galaxies would have been selected to our catalog (described in Sect. 5). This indicates that the completeness obtained with our test is slightly underestimated.

Fig. 4. Objects missed due to offset in center coordinates. Examples of relatively bright objects that were not considered as detected with MTO by our test in Sect. 3.2.2 since the center of the detected object is too far from the dwarf center. Left panels: r′-band postage stamps, and the right panels: cutouts from the segmentation maps. The target objects are shown with the small crosses.

In summary, our analysis in this and in the previous subsection showed that using MTO we can detect the LSB dwarfs with more than 50% completeness down to ${\overline{\mu }}_{e,r^{\prime }}$ ≈ 26.5 mag arcsec⁻² in the FDS data, which is approximately one magnitude deeper than with SE. Additionally, MTO pushes the 75% detection limits near the level of the 50% completeness limits of the FDSDC.

4. Photometry

In order to obtain reliable photometric measurements for the identified galaxies, we used GALFIT 3.0 (Peng et al. 2010) to fit Sérsic functions to the 2D galaxy light distribution. Since MTO identifies all the structures in the images very accurately and labels them in the segmentation maps, we were able to automatize the photometric measurements using the information from those maps. The eight steps in the photometric measurements were as follows.

The first step was cutting the postage stamps: Postage stamp images for all the available bands from the science mosaics and weight mosaics were cut. The images were cut according to the central coordinates of objects found by MTO, and cut radii were selected to be ten times the semimajor axis lengths found by MTO. Sigma images were also made accordingly from the weight images following the workflow described by V18.

The second was classifying secondary objects and background: Since the FDS data are deep and the cluster is a dense galactic environment, the postage stamps usually include many galaxies in addition to the main object. Those objects are first selected based on the MTO segmentation maps and their sizes and magnitudes are also collected into a list. Using the segmentation map also the background is identified. As all the pixels in the images belong to some branch of the max-tree (at least the root), the background is selected as the object that has the lowest median SB among the objects that cover at least 20% of the total image area. The background pixels are then fit with a linear plane and the goodness of fit is estimated by calculating the normalized chi-squared value, ${\chi }_{\nu }^2$. Based on our experience, we decided to use ${\chi }_{\nu }^2<2$ as the limiting value for a linear sky model. In the case of nonlinear background, we treat the lowest segmentation branch in the image as a normal object and fit it with a Sérsic profile.

The third was classifying secondary objects: It is not purposeful to fit all the secondary objects with GALFIT, since the rapidly growing number of fit parameters can make the fitting unstable and many of the secondary objects are small, which means that they can be masked. However, especially in the inner parts of the cluster many objects appear nested on the sky and have to be fitted simultaneously in order to obtain accurate parameters for them. Thus we need to select a limited amount of objects to the GALFIT modeling. At first, we select all the secondary objects that have⁷m_r′ < m_r′,main + 3 mag and have total magnitude m_r′ < 23 mag. From those objects we keep the ones that are bright (m_r′ < 15 mag), extended (a > 2 arcsec), or overlapping with the main target (d < 3R_e). All the remaining objects that are not selected for the model or identified as the background are selected for masking. Additionally, the centers of bright stars (m_r′ < 15 mag) are selected for masking⁸, although their outer parts will be fitted with a Sérsic profile. At last, in order to keep the number of models in the GALFIT modeling reasonable we limit the maximum number of Sérsic profiles to five. In cases where there are more than five objects that are initially selected for the model we remove the faintest objects from the model and mask them.

The fourth was creating an input model and a mask: The objects selected for masking are masked based on the MTO segmentation maps. The GALFIT model is initialized as follows: The background is modeled with a linear plane with three degrees of freedom (x- and y-gradients and mean) and all the objects selected for the model are modeled with Sérsic profiles whose initial magnitude and effective radius are taken from the MTO output lists and axis ratio b/a and Sérsic index n are set to one. All the parameters are fitted simultaneously, but we apply the following limits for the Sérsic components in order to avoid divergence in the fitting routine: We limit the center of the Sérsic profile to be within 3R_e from the detection center, the effective radii to be between 0.1R_e, in < R_e, out < 10R_e, in (R_e, in is the initial effective radius), Sérsic index to be between 0.3 < n < 10, and the magnitude to be within 2.5 mag from the MTO input value.

The fifth was fitting: The input model is fed to GALFIT with the corresponding sigma image and PSF model. We use the PSF models generated by stacking stars described by V18. The parameters of the objects obtained from GALFIT fitting are then collected to a table.

The sixth was identifying the nucleus: Some dwarf galaxies have an unresolved star cluster in their center, which needs to be fitted with a separate PSF component in order to obtain a good fit. We compare the Gini coefficients of the PSF and the galaxy center within the innermost 2 arcsec in order to identify nuclei. The galaxy center is adopted from the Sérsic model fit performed with GALFIT (step 5) and paraboloid fitting is used in the central regions to adjust the center. If the Gini coefficient of the galaxy center is at least 40% of that of the PSF step 5 is performed again using Sérsic + PSF as a input model for the galaxy. More details of the detection of the nucleus are given in Appendix A.5.

The seventh involved aperture magnitudes, the residual flux fraction, and the concentration parameter: Similarly as V18, we also measure aperture magnitudes within R_e using an elliptical aperture with b/a ratio adopted from the GALFIT model. We also calculate the residual flux fraction

$$\begin{array}{c}\hfill RFF=\frac{{\mathrm{\Sigma }}_{i=1}^{n_r<R_p}|dat{a}_i-mode{l}_i|-0.8{\sigma }_i}{F_{r<R_p}},\end{array}$$(2)

where n_r < R_p is the number of pixels within the Petrosian radius (R_p). R_p is defined to be the radius where galaxy’s SB equals 1/5 of the mean SB within that radius. F_r < Rp is the total flux within the R_p and σ_i is the sigma image value at pixel i. The term |data_i  −  model_i| corresponds to the residual at a given pixel, and the following term takes into account the expected value for a normal-distributed noise, normalizing the perfect residual to zero. We also calculate the concentration parameter:

$$\begin{array}{c}\hfill C=5\times {log}_{10}\frac{R_{80}}{R_{20}},\end{array}$$(3)

where R₈₀ and R₂₀ are the radii that enclose 80% and 20% of the total Petrosian luminosity of the galaxy, respectively.

The eighth was generating residual and color images: In order to visually classify the identified objects based on their morphology, we also generate color composite images of them using the g′,r′, and i′-band FDS images. We also generated residual images of each object, by subtracting the GALFIT model from the original image after first transforming each into units of mag arcsec⁻². Inspecting the residuals in this manner gives a similar weight to residual in each intensity scale and it allows us to easily spot any inner structures such as spiral arms or bars in the galaxies that can give crucial information about the nature of the objects in the classification. An example of the different photometric steps is shown in the Fig. 5.

Fig. 5. GALFIT photometry. Example of the steps done in order to obtain the photometric parameters. First, the sigma-, science- and segmentation-image postage stamps centered on the object of interest are cut (left panels). The objects are selected for the mask and the GALFIT model (the green and red symbols in the second column picture from the left, respectively). A mask is generated (the upper panel of the third column from the left) using the segmentation map, and the initial model (the lower panel of the third column from the left) is built using the MTO object locations, magnitudes, and sizes. We note the large fuzzy object in the input model above and to the left of the target, which corresponds to the extended source outside the image field of view: The object is placed within the image in the input model, but it is allowed to move outside the image edges during the fitting. GALFIT is then used for model fitting. The right column panels from top to bottom show the fitted model with and without the noise with the mask overlaid and the residuals after subtracting the model from the data.

In order to quantify the uncertainties of the parameters obtained from the photometry, we tested the accuracy of our photometric pipeline with the mock images described in Sect. 2.2. We ran the photometric pipeline for the identified mock galaxies similarly as for real images and compared the obtained parameters of the individual galaxies with the corresponding input values. The comparison between the input and output values as a function of the input SB is shown in the upper panels of Fig. 6.

Fig. 6. GALFIT parameter accuracy. Comparison between the input and output values of the structural parameters of the mock galaxies. The upper panel shows the differences between the input and output magnitudes (mr′), effective radii (Re), Sérsic indices (n), and axis ratios (b/a) as a function of input mean effective r′-band SB (${\overline{\mu }}_{e,r^{\prime }}$). The black symbols show individual measurements, and the blue line shows the running mean with a 2 mag arcsec−2 filter size. The corresponding standard deviations are shown with the dashed blue lines. The lower panels show the magnitude of the standard deviations between the input and output values. The blue curves show the fits of Eq. (4) to the data.

In order to estimate the uncertainties in the measured parameters of the real galaxies, we measured the standard deviation, σ, of the differences between the input and output values of the measured parameters as a function of the output SB. Similarly as Hoyos et al. (2011), we fit a linear model to the logarithm of the standard deviations:

$$\begin{array}{c}\hfill {log}_{10}(\sigma )=\alpha \times {\overline{\mu }}_{e,{r^{\prime }}^{}}+\beta ,\end{array}$$(4)

where α and β are fitted. The relations between the mean effective SB in r′ band and the standard deviations of the difference between the input and output values of the different parameters are shown in the lower panels of Fig. 6. The fit parameters of the curves in the lower panels are shown in Table 2.

The scatter between the input and measured values increases with decreasing SB. There are also small systematic trends in the GALFIT parameters with decreasing SB, so the measured magnitudes become slightly fainter and effective radii slightly smaller than the input values, but these trends are much smaller than the scatter. We use these fits for estimating the uncertainties of the parameters in the final catalog. The accuracy of our pipeline is similar to that obtained by V18, who also used the FDS data, but performed the photometry⁹ manually. This result shows that automated analysis of the LSB galaxies can obtain similar accuracy as the manual one when the accurate segmentation maps of the MTO are used for generating the masks and input models.

5. Preparation of the catalog

In order to detect LSB galaxies from the full FDS data set, we ran MTO for all the 26 FDS fields. For the detection, we used the same stacked g′+r′+i′ band mosaics that were also used for generating the FDSDC in V18. As a result from the detection, we obtained the locations, magnitudes, and the minor- and major-axis radii of the objects. We then used the stellar halo masks generated by V18, which were used to mask the bright stars from the data, and removed all the detections that were within the masked areas.

We then selected the objects with a > 2 arcsec and a median SB of the detected pixels fainter than 23 mag arcsec⁻². The SB limit guarantees that we mostly detect LSB galaxies that were incompletely detected in the FDSDC. We acknowledge the fact that when using this relatively high SB limit our initial detection lists partly overlap with FDSDC. However, this is not a problem since we subsequently set further limits based on GALFIT parameters, in order to select the objects for the final LSB galaxy catalog, and also remove any duplicate detections of the galaxies. As a result we obtain a sample of 7270 LSB candidates, which is an order of magnitude more than we currently know.

In addition to galaxies, MTO detects many extended objects that are not galaxies, such as reflection rings around bright stars or artifacts from the image reduction. Additionally, there are background galaxy groups that are located within a common LSB envelope, which gets associated with some of the galaxies, making its median SB low and its size large (see Appendix A.4). We were able to remove most of the reflection rings (see Appendix A.4) by using masks generated by selecting bright stars (m_r′ < 16 mag) from the American Association of Variable Star Observers’ Photometric All Sky Survey catalog (APASS; Henden et al. 2012) and excluding sources, which reside in the areas near them. Around the bright stars, we masked the areas where the SB of the star is μ_r′ < 25.5 mag arcsec⁻². For that task, we used the PSF model of V18. Applying these masks removes ≈3% from the survey area, but as the masks cover areas around saturated and bright stars, those areas would have been non-usable in any case. After excluding the masked objects, we inspected all objects and manually removed all the remaining false positives from the sample (N_removed = 1234).

We then ran the automated photometry for all the remaining galaxies (N = 6036) and visually checked all the model fits. The galaxies that were not fitted optimally were flagged. Flagged galaxies were then refitted after modifying their masks and/or input models. Only 41 (=0.7%) galaxies needed refitting.

As our intent is to produce a homogeneous extension to the FDSDC, we followed the steps done by V18 for selecting the likely cluster galaxies. First, we applied the same color, SB, and concentration parameter cuts as Venhola et al. (2018). We excluded the galaxies that are redder than the brightest cluster galaxies (g′−r′> 0.95 or g′−i′> 1.35). We also excluded the small high-SB galaxies whose SB deviates by at least three standard deviations from the mean of the known cluster galaxies’ SB distribution (see Fig. 7). Finally, we excluded the very concentrated (C>3.5) galaxies with faint apparent magnitude.

Fig. 7. Parametric cuts used for the selection of LSB candidates. The panels show g′−r′ colors (top panel), SBs (middle panel), and the concentration parameter, C (bottom panel of the galaxies), as a function of their r′-band apparent magnitude, mr′. The selected galaxies (black symbols) and the galaxies rejected for being too red (red symbols), having too high of a SB (green symbols), and a too high C (blue symbols) are indicated with the different symbols. The black lines in the panels show the selection cuts (see the text and V18).

After applying the parametric selection, we are left with 1417 galaxies that pass all the selection criteria. We then crossmatch those galaxies with the galaxies detected previously in the FDSDC. We remove galaxies whose centers are within 5 arcsec from the FDSDC galaxies in order to eliminate the duplicates. This removes 403 galaxies from the sample, leaving 1014 galaxies. The radius of 5 arcsecs radius is adequate as that corresponds to approximately 500 pc at the distance of the Fornax cluster, and the projected separations of the FDSDC galaxies are much larger than that. We also crossmatched our remaining 1014 galaxies with the background galaxies that were identified during the creation of the FDSDC by V18. V18 already classified those galaxies morphologically as background galaxies, so we do not perform that task for the common galaxies again. We use a search radius of 3 arcsec when searching for duplicate background galaxies, as their sizes are smaller and number density is larger than for the FDSDC galaxies. After removing the 419 duplicates, we are left with 595 previously unidentified LSB candidates.

We inspected the 595 LSB candidates visually using color images generated from the FDS g′-, r′-, and i′-band seeing-selected data¹⁰, and residual images, and removed from the sample the galaxies (N = 320) that show a clear spiral structure or a bulge not typically seen in low-mass galaxies. We classified the other galaxies as either likely cluster galaxies (N = 226) or galaxies with unclear morphology (N = 49). We used the same parametric classifications as V18 for the uncertain cases. We excluded the red (g′−r′> 0.45) uncertain objects with high RFF as likely background galaxies (N = 10) and accepted the others as likely cluster members (N = 39). We did not perform further morphological classifications to early- and late-type galaxies due to the LSB nature of the objects, which makes such a classification uncertain.

Since we aim to use the catalog to analyze the effects of the environment on the evolution of dwarf galaxies, we identified dwarfs whose morphology shows signs of interactions. This classification was done visually (by Aku Venhola) from the images generated by the photometric pipeline. We gave the objects four different classes: (i) “regular,” no signs of interactions in the outskirts; (ii) “possibly disturbed,” some irregularities in the outskirts; (iii) “disturbed,” clear tidal tails or arms in the outskirts; and (iv) “unclear,” some nearby objects or SB of the object prevents the classification. We did this classification for all the FDSDC galaxies and additional galaxies identified in this work. Examples of the different morphologies are given in Fig. 8.

Fig. 8. Tidal morphology. Examples of the different morphological labels given for the galaxies based on visual inspection. The galaxy in the middle panel shows slight irregularities in its outskirts and is thus classified as slightly disturbed. The galaxy in the bottom panel shows clear tidal arms, which justifies its classification as clearly disturbed.

After all the selection cuts we are left with 265 new LSB dwarfs in the Fornax cluster. When these galaxies are combined with the FDSDC (N = 556), we have a total of 821 dwarf galaxies in the area of the FDS. We publish the parameters and classifications of the LSB dwarfs we have found along with those of the FDSDC galaxies as a separate electronic table. An extract of the longer table is presented in the Appendix B.

6. Analysis of the LSB galaxies

6.1. Luminosity function

The new LSB galaxies extend the FDSDC toward fainter total magnitudes and lower SBs. Effectively the extension shifts the peak of the detected LF 1 mag deeper than FDSDC. The magnitude and mean effective SB distribution of the LSB extension galaxies are shown with red symbols in Fig. 9. As we have defined the completeness limits for our detection algorithm, we can also identify whether the changes in the abundance of dwarfs in the different regions of the parameter space are consequences of the detection incompleteness or real changes in the abundances. Between −17 mag < M_r′ < −12 mag, the abundance of dwarfs seems to drop both toward low and high SBs. In this region, the abundance drop takes place well before we start to be incomplete in the detections, so it appears to be caused by the span of the intrinsic parameter distribution of dwarfs. On the other hand, for galaxies with M_r′ > −12 mag we are limited by our completeness and selection limits, which makes it impossible to trace the trends in the dwarf galaxy magnitude-SB distribution in that region.

Fig. 9. Surface brightness of Fornax cluster dwarfs. Apparent r′-band magnitudes (mr′) and mean effective SBs (${\overline{\mu }}_{e,r^{\prime }}$) of the FDSDC galaxies (black symbols) compared with the LSB extension galaxies described in this work (red symbols). The histograms on the x and y axes show the distributions of the magnitudes and SBs with the same colors as in the scatter plot. The blue and green contours show the 50% and 75% detection limits found using the artificial galaxies (Sect. 3.2), and the black and dashed gray lines show the a > 2 arcsec and SB selection limits, respectively. There are also galaxies with Re < 2 arcsec as the selection was based on MTO detection a > 2 arcsec. For a comparison, we also show a sample of LG dwarfs and globular clusters (green squares clearly above these selection limits) from Brodie et al. (2011).

Sizes and luminosities of the Local Group (LG) dwarfs can also give insights whether we are likely to miss many dwarfs. Based on the comparison with LG dwarfs (Brodie et al. 2011), we do not seem to exclude many dwarf galaxies (apart from ultra-faint dwarfs) from our sample with our selection limits if the Fornax cluster and LG dwarf populations resemble each other (see Fig. 9). The green symbols clearly above the selection limits of the Fornax galaxies are LG globular clusters.

As this work extends the FDSDC especially in the low-luminosity end, it is interesting to see the effects of these newly identified galaxies on the low-luminosity end slope of the LF. Based on the dwarf galaxy distribution and completeness limits in Fig. 9, we can well trace the dwarf galaxy population down to M_r′ = −12 mag, without any significant completeness corrections for compensating biases arising from selection cuts or detection completeness. In the upper and middle panels of Fig. 10, we show the LF of the dwarf galaxies in the Fornax cluster in r′ and g′ bands, respectively. For the r′ band we used the total magnitudes from Sérsic fits, and for the g′ band we added the g′−r′ color measured within the r′-band R_e to those magnitudes. We also show separately the galaxies within and outside of the Fornax virial radius R_vir = 2.2 deg (0.7 Mpc). In the bottom panel of Fig. 10, we show the stellar mass function of the Fornax cluster dwarf galaxies. We transform the galaxy luminosities to mass using the transformations of Taylor et al. (2011), which adjusts the mass-to-light ratio by the galaxy colors (using the colors measured within R_e).

Fig. 10. Galaxy luminosity and mass functions. Upper and middle panels: r′- and g′-band LFs of the Fornax cluster dwarf galaxies, respectively. Bottom panel: stellar mass function. The black, red, blue, and green histograms show the LF for all the galaxies, for galaxies within the virial radius of the Fornax cluster, those outside of it, and those within the cluster core, respectively. The curves show the Schechter function fits to the data, and the fit parameters are shown with corresponding colors in the upper-left corner of the image. The dotted gray lines show the completeness limits for the different parameters.

We fit the LFs using the Schechter function (Schechter 1976):

$$\begin{array}{c}\hfill n(M)\mathrm{d}M=0.4ln(10){\varphi }^{\ast }{\left[{10}^{0.4(M^{\ast }-M)}\right]}^{\alpha +1}exp(-{10}^{0.4(M^{\ast }-M)})\mathrm{d}M,\end{array}$$(5)

where ϕ^* is a scaling factor, M^* is the characteristic magnitude where the function turns, and α is the parameter describing the steepness of the low-luminosity end of the function. As we are interested in the low-mass slope of the LF, we fit the galaxies in the range −18.5 mag < M_r′ < −12 mag, and use fixed characteristic magnitude of $M_{{r^{\prime }}^{}/g^{\prime }}^{*}$ = −20.48 mag. The M^* was selected based on the analysis of Smith et al. (2009), who analyzed a large sample of Sloan Digital Sky Survey (SDSS) galaxies. We fit the Schechter function to the bins in Fig. 10 and assume Poisson uncertainty in the bins while doing the fit. We obtain low-mass-end slopes of α = −1.38 ± 0.02, −1.42 ± 0.02, and −1.46 ± 0.05 for r′-band and g′-band LF and stellar mass function, respectively. These values do not vary significantly, when dividing the galaxies into subsamples including galaxies within or outside the virial radius (2.2 deg) of the Fornax cluster.

The low-mass-end slopes that have been obtained for the luminosity or stellar mass functions previously in the Fornax cluster have large variance. Using the FDSDC galaxies, Venhola et al. (2019) obtained LF slopes between −1.13 < α < −1.44 in different cluster centric bins if no completeness correction was applied. When the LFs were corrected for completeness, the values varied between −1.27 < α < −1.58 in the different bins. For the whole FDSDC the uncorrected slope is α = −1.18, which is significantly flatter than found with this more complete sample of the current work. Hilker et al. (2003) and Mieske et al. (2007) calculated a low-mass slope of α = −1.11 ± 0.1 from a sample of early-type galaxies in the center of the Fornax cluster. Since they studied smaller sample of galaxies their uncertainties are much larger than ours. In both abovementioned studies, M_* was fit as a free parameter. In order to test whether that could explain the differences in the low-mass and low-luminosity end of the distribution, we also performed fitting leaving M_* as a free parameter. We then obtained low-mass-end slopes of α = −1.35 ± 0.03, −1.40 ± 0.03, and −1.35 ± 0.02 for the r′-band LF, the g′-band LF, and the stellar mass function, respectively. Hence, the change in M_* does not significantly affect the slope, and the 2σ difference between our value for α and the value obtained by Hilker et al. (2003) and Mieske et al. (2007) is not caused by the different fitting technique.

6.2. Intrinsic shapes and surface brightness correlations

As seen in Fig. 9, at any given luminosity dwarf galaxies have a range of different mean effective SBs. Additionally, there is a trend between the SBs and absolute magnitudes of dwarfs. Here we investigate, how much of this variation in the SBs at given mass or luminosity is due to the projection effects caused by the inclination of galaxy and different stellar populations of dwarfs, and whether the remaining differences are connected to the galactic environment.

Assuming a dwarf galaxy is shaped like an oblate spheroidal and the dust does not obscure the stellar light significantly, its apparent SB is highest when it is viewed edge-on, and lowest when viewed face-on. For a prolate (cigar-like) shape, the apparent SB is the highest when viewed along the longest axis. When the shape is triaxial, the relation between the viewing angle and the SB is more complicated. In order to reduce the bias in the observed SBs of the galaxies, this projection effect can be corrected if the shapes of a galaxy population are known.

In order to test whether significant projection effects exist among the dwarfs, we show the correlation between b/a and ${\overline{\mu }}_{e,r^{\prime }}$ and mean effective stellar SD (${\overline{\mathrm{\Sigma }}}_{e,\ast }$), in the upper panels of Fig. 11 for dwarfs in different stellar mass bins¹¹. We find that for galaxies with M_* > 10⁷ M_⊙, the SBs and surface densities increase remarkably with decreasing axis ratio. This is consistent with the behavior of disks. For galaxies less massive than that, there is no correlation between the SD or brightness of the galaxy with its apparent axis ratio.

Fig. 11. Inclination corrections. Correlation between the axis ratio (b/a), mean effective SB (${\overline{\mu }}_{e,r^{\prime }}$), and stellar SD (${\overline{\mathrm{\Sigma }}}_{e,\ast }$) of the galaxies. Upper panels: values without any inclination corrections, in the middle panels galaxies with log(M*/M⊙) > 7 have been corrected for their inclination, and in the bottom panels inclination correction has been applied to all the galaxies. The different colors correspond to different mass bins, as shown in the legend. The solid and dashed lines indicate the running mean (filter size of 0.2) and the mean of the mass bin.

In order to correct for the projection effects, we calculate the de-projected mean effective SB for each galaxy. For a thick oblate disk with a Sérsic-like luminosity profile, such as the massive dwarf galaxies in the Fornax are, the de-projection can be simply made by multiplying the apparent SB (in flux) by the apparent axis ratio (Salo et al., in prep.). Thus, we define de-projected mean effective SB ${\overline{\mu }}_{e,d}$ = ${\overline{\mu }}_e-2.5{log}_{10}(b/a)$ for dwarfs with M_* > 10⁷ M_⊙, and ${\overline{\mu }}_{e,d}$ = ${\overline{\mu }}_e$ for dwarfs with M_* < 10⁷ M_⊙. We show the relation between the de-projected SBs and densities with the axis ratios of the galaxies in the lower panels of Fig. 11. The correlations with the axis ratios disappear when de-projected quantities are used. We use the de-projected SBs in this subsection. In the bottom panels of Fig. 11, we demonstrate that if the same inclination corrections that are used for the M_* > 10⁷ M_⊙ galaxies are applied for lower-mass galaxies, this introduces an artificial trend between ${\overline{\mu }}_{e,d}$ and b/a, which indicates that the assumption about disk shape for those galaxies is wrong.

In principle, this result could be caused by a selection bias: If we are biased toward selecting edge-on galaxies in the very low-luminosity end, this could diminish the correlation between the SB and axis ratio when analyzed in mass bins. However, according to our tests made with the mock galaxies, this seems not to be the case (see Fig. 1) as there is no selection bias related to the axis ratio in the low-mass end. In order to confirm that using the Fornax cluster dwarf galaxies, we show the b/a distribution in mass and magnitude bins in Fig. 12. It appears that the axis-ratio distribution is nearly constant in the low-mass bins, and that most of the low-mass galaxies have b/a ≈ 0.7.

Fig. 12. Axis ratio trends. Average axis ratios of the dwarf galaxies in the Fornax cluster in mass (upper panel) and absolute r′-band magnitude (lower panel) bins. The symbols indicate the average values with their uncertainties. The mass bins have a width of 0.6 dex, and the magnitude bins have a width of 1 mag.

In addition to the geometry of dwarfs, we analyze the concentration of their light profiles as a function of their mass. Typically disks of disk galaxies are well fit with an exponential profile, which corresponds to n = 1, and elliptical galaxies and the central bulges of disk galaxies have n ≥ 2. In Fig. 13, we show the Sérsic indices of galaxies as a function of their stellar mass. The running average shows that low-mass galaxies have on average n ≈ 1 or slightly below that and the concentration of the profiles increases toward more massive galaxies. In order to find out whether the scatter of n values around the average is due to the measurement uncertainties or intrinsic to the dwarf population, we compare the measured scatter of the population with the measurement uncertainties that we estimated using the mock galaxies in Sect. 4. We find that the measured scatter (red highlighted area in Fig. 13) in the Sérsic indices is similar to the measurement uncertainty green highlighted area in Fig. 13) for galaxies with log₁₀(M_*/M_⊙) = 7–8.5. For galaxies more massive than that, the dwarf population seems to have intrinsic variation in n much larger than the measured uncertainties. Galaxies with log₁₀(M_*/M_⊙) < 7 seem to have slightly larger scatter than expected from the measurement uncertainties. To summarize, the light profiles of most low mass dwarf galaxies are well approximated by an exponential profile and it is possible that all the measured deviations from that are due to measurement uncertainties. Massive dwarfs however have much higher scatter in n values than the measurement uncertainties, which indicates an intrinsic differences in the profiles of the dwarf galaxies.

Fig. 13. Sérsic indices as a function of galaxy stellar mass. The black points show the measurements for the dwarf galaxies in the Fornax cluster, and the red line shows the running average with a filter size of 0.5 dex. The green and red shaded areas correspond to the scatter expected from the measurement uncertainties and the measured scatter, respectively.

In order to investigate the connection between a galaxy’s SB and its other properties, we calculated the running mean along the log(M_*)-${\overline{\mu }}_{e,d,r^{\prime }}$ – relation of the galaxies, and calculated the running standard deviation of the galaxies with respect to that trend. Similarly as was done using the SB of the galaxies, we also calculated the means using surface mass densities. We transformed the de-projected mean effective SBs into the corresponding surface densities (${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$).

The relation between ${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$, ${\overline{\mu }}_{e,d,r^{\prime }}$ and M_* of the galaxies are shown in Fig. 14. Trends in the SBs and surface densities are very similar, and have similar spread when the factor of 2.5 between the two quantities is taken in account. The deviations from the means of these two quantities are strongly correlated as can be seen in the lower middle panel of Fig. 14. In other words this means that, on average, the galaxies that have brighter or fainter SB than the mean SB at given mass, have also SD that deviates similarly from the average. Thus, the difference in the SB is mostly due to an actual variation in stellar mass density, not due to projection or stellar population effects

Fig. 14. Cluster-centric trends in galaxies’ relative surface brightness and density. Upper-left and lower-left panels: de-projected mean effective SB (${\overline{\mu }}_{e,d,r^{\prime }}$) and stellar SD (${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$) as a function of galaxy stellar mass (M*) for the Fornax cluster dwarfs, respectively. The solid and dashed red lines show the running mean and standard deviations from that, respectively. The RHSB galaxies, RLSB galaxies, normal galaxies, and UDGs are indicated with the different colors, shown in the legend in the upper-left panel. The definitions of those classes are described in the text. Upper-middle panel: normalized mass distributions of the different SB subsamples in the mass bins indicated with the vertical lines in the upper-left panel. Lower-middle panel: correlation between $\mathrm{\Delta }{\overline{\mu }}_{e,d,r^{\prime }}$ and $\mathrm{\Delta }{\overline{\mathrm{\Sigma }}}_{e,d,\ast }$. Right panels: cumulative cluster-centric distance distribution for the different subsamples when defined using ${\overline{\mu }}_{e,d,r^{\prime }}$ (upper-right panel) and ${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$ (lower-right panel). K-S test p values for the assumption of two samples being drawn from the same distribution are also listed for all the sample pairs in the lower right of the right panels.

In order to neutralize the effects of mass and study how the environment affects the SB of galaxies, we then divided the galaxies into three parts: Relatively high surface brightness (RHSB) galaxies are those that have a SB > 1σ from the running mean with respect to stellar mass; relatively low surface brightness (RLSB) galaxies deviate by > 1σ toward fainter SBs; and the galaxies in between those groups are “normal” galaxies. In addition, we studied a subgroup of LSB galaxies, UDGs, defined as galaxies having ${\overline{\mu }}_{e,d,r^{\prime }}>24$ mag arcsec⁻² and R_e > 1.5 kpc. Similarly, we separated galaxies based on their SD into relatively low surface density (RLSD) and relatively high surface density (RHSD) groups. The different classes are shown in the upper-left panel of Fig. 14. The normalized M_* distributions of the different SB groups are also shown in the upper-middle panel of Fig. 14.

We calculated the cumulative distributions of the galaxies’ cluster-centric distances (projected distances to NGC 1399) for the different galaxy subgroups. These cumulative profiles are shown in the right panels of Fig. 14. It appears that the RHSB and RHSD galaxies are less centrally concentrated in the cluster than the diffuse ones. We calculated also the Kolmogorov-Smirnov (K-S) test p values for the assumption of any two of the subgroups being drawn from the same underlying distribution. The K-S test results between the subgroups are shown in the right panels of Fig. 14. We find that significant differences are found between the UDGs and RHSB galaxies and between the UDGs and RHSD galaxies (p values 2.5 × 10⁻³/5.0 × 10⁻⁵), RLSB galaxies with respect to RHSB galaxies (p = 3.7 × 10⁻⁵), between normal galaxies and RHSB, and between normal and RHSD galaxies (p = 1.6 × 10⁻³/2.6 × 10⁻⁶). When using surface densities, the difference between the RLSD and normal galaxies is also significant (p = 0.013).

As the galaxies with different surface densities are distributed differently in the cluster, the galactic environment is likely to be responsible for those differences. In order to further understand how RHSD and RLSD differ from each other, we calculated normalized deviations from mean for a range of parameters and compared those with the excess of galaxy’s SD with respect to the mean at given mass. In practice, the normalized deviations were calculated by deriving a running average of the parameter at the given stellar mass, and then normalizing the deviations by running standard deviations.

In Fig. 15 we compare galaxies’ normalized deviations from the mean g’-i’ color, the effective radius, and Sérsic index with their deviations from the mean SD. The deviations are normalized by the running standard deviation calculated within the same bin as the running average. We used g′−i′ color, as it is available for the whole sample and is independent from the r′ band, which is used to calculate the SB and R_e. We find that galaxies’ SD has no correlation with their color and only a slight correlation with their Sérsic indices, so dense galaxies are slightly more centrally peaked than the average. As expected, the clearest correlation between the galaxy density is with the galaxies’ effective radii, so the higher the galaxy’s SD is the smaller R_e it has. The correlation between ${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$ and R_e is not fully linear due to the differences in stellar populations.

Fig. 15. Correlations between the excess mean effective stellar SD and properties of the FDS dwarfs. Upper panel: comparison with the excess color (with respect to the mean at a given mass), the middle panel with the excess Sérsic index, and the lower panel with the excess effective radius. The red symbols highlight the UDGs, and the black symbols show the other dwarfs.

7. Discussion

The reanalysis of the FDS images has improved the completeness of the FDSDC in the LSB regime, which makes it interesting to compare our findings with other deep dwarf galaxy surveys and the theoretical framework.

7.1. Environmental dependence of the low-mass end of the luminosity function

There are some works on the dwarf galaxy LF that match our image depth and the detection completeness. In nearby groups, there is the advantage that galaxies can be resolved into stars, which makes it possible to be complete down to fainter magnitudes. Thus, we first make a comparison with nearby group environments.

Low-mass dwarf galaxies have been well studied in the LG. We used the compilation of LG galaxies made by Ferrarese et al. (2016) to compare the LFs in the LG and in the central parts of the Fornax cluster (D < R₂₀₀). We first transformed our magnitudes into V-band magnitudes using the transformation formula¹² of Jordi et al. (2006); we compare the cumulative LFs of these two environments in Fig. 16. We made the comparison for galaxies with M_V = −12.5 mag since both samples are more than 75% complete at that magnitude limit. We find that the low-mass end of the LF is steeper in the Fornax cluster than in the LG, but the number of galaxies in the LG is not large enough to exclude the possibility that these distributions follow the same underlying distribution (p = 0.28).

Fig. 16. Empirical cumulative luminosity functions. Comparison of the V-band LFs in the Fornax cluster, the LG, and other nearby environments. We use the compilation of Ferrarese et al. (2016) for the LG and Virgo, the Müller et al. (2019) compilation for the Centaurus A group, the compilation of Carlsten et al. (2021) for other nearby groups, and the galaxies in this work for the Fornax cluster and the Fornax A group. The vertical dashed lines show the cutoff limits of MV = −18 mag and MV = −12.5 mag.

In order to improve the statistics of the group environment LF and thus obtain more insight about possible differences, we add also other groups to our comparison. Carlsten et al. (2021, 2019) gathered a compilation of satellite galaxy LFs in nearby groups with deep observations. We add the satellite LFs of NGC 1023, NGC 2903, NGC 4258, NGC 4565, NGC 4631, M 51, M 104, and M 81 from that work into our comparison. Since many of the groups of that work contain few satellites, we combine the LFs of those groups as a single group LF. In addition, to those groups we add the LF of Centaurus A by Müller et al. (2019) into our comparison, and show also the LF of the Fornax A subgroup separately, in order to study whether there are differences between the LFs in the different environments. These LFs are shown in Fig. 16.

Another well-studied cluster environment is the center of the Virgo cluster. Ferrarese et al. (2016) studied the core of the cluster using the data of the Next Generation Virgo Cluster Survey (NGVS, Ferrarese et al. 2012, 2020), which has similar depth as the FDS. The NGVS is complete down to M_g = −12.5 mag.

All the abovementioned satellite compilations include a lot of galaxies even with M_V > −10 mag, but the completeness drops significantly in the low-luminosity end. Therefore, we adopted M_v = −12.5 mag for the lower-luminosity limit. In this region, all of the considered galaxy samples should be fully complete.

In Fig. 16 we show the cumulative LFs of the different environments. There seems to be differences between some of the cumulative LFs of the different environments. In order to quantify whether the LFs have statistically significant differences, we use K-S tests between the different samples. We tabulate the values in Table 3. Based on the K-S tests, we find no statistical evidence of differences between the different samples.

These comparisons show that the different deep surveys show similar behavior of the faint end of the LF down to M_V = −12.5 mag. These results seem robust in light of the available data. For the galaxies fainter than that magnitude limit, it is likely that some of the galaxies will be missed as their ${\overline{\mu }}_{e,r^{\prime }}$ drops below 27 mag arcsec⁻² and some of the galaxies will be unresolved with the typical ground-based seeing of 1 arcsec. Obtaining certainty about the universality of the LF and extending the results toward lower luminosities would require a homogeneous ultra-deep survey with good seeing that has a large sky coverage and thus a large range of different environments. Future surveys such as Euclid¹³ and the Legacy Survey of Space and Time (LSST; Ivezić et al. 2019) will likely shed light on this issue.

7.2. Comparison of the luminosity function with IllustrisTNG

One important test for cosmological simulations is that they need to produce the right amount of luminous low-mass galaxies. The LF obtained from FDS data can be used for making such a comparison. We perform the comparison with the recent IllustrisTNG (Nelson et al. 2019)¹⁴ cosmological simulation, which includes prescriptions for baryonic matter physics.

IllustrisTNG is a set of full-scale cosmological hydrodynamical simulations that include dark matter, gas, stars, stellar winds, magnetic fields, and black hole components. We analyzed its 110.7³ Mpc³ simulation box (TNG100), which has baryonic and DM mass resolutions of 1.4 × 10⁶ M_⊙ and 7.5 × 10⁶ M_⊙, respectively, and softening lengths of 185 pc for gas and 740 pc for DM and stellar particles. Since the softening lengths are large compared to the typical sizes of the Fornax dwarfs (R_e ≈ 700 pc), it is not safe to compare the structure of the observed and simulated dwarfs. Thus, we restricted our comparisons to the stellar mass function of galaxies.

The TNG100 simulation includes 22 galaxy clusters within the mass range 5–10 × 10¹³ M_⊙, similar to the Fornax cluster, 7 × 10¹³ M_⊙ (Drinkwater et al. 2001). We made comparisons with the LF of all these clusters but also compared the Fornax cluster specifically with group number 21 of the IllustrisTNG-100 since it is the most Fornax-like cluster from the group of 22. Cluster 21 has a mass (7.34 × 10¹³ M_⊙), a size (R₂₀₀ = 0.56 Mpc), and a 2D velocity dispersion (σ = 318 km s⁻²) that very closely resemble those of the Fornax cluster (M₂₀₀ = 7 × 10¹³ M_⊙, R₂₀₀ = 0.7 Mpc, and σ = 370 km s⁻¹).

We show the comparisons between the stellar mass functions of the 22 IllustrisTNG clusters and the Fornax cluster in Fig. 17. We performed these comparisons down to log(M_*/M_⊙) > 6.5, which corresponds roughly to the 75% completeness of our catalog. We compare the stellar mass functions between 6.5 < log(M_*/M_⊙) < 9, 7 < log(M_*/M_⊙) < 9, and in the range where IllustrisTNG resolves the galaxies well, log(M_*/M_⊙) > 8. We find that, generally, IllustrisTNG has a shallower low-mass-end slope than that found by us in the Fornax cluster within all the selected limits. These differences are statistically (> 2σ) significant for 18, 17, and 1 clusters when the low, mid, and high lower-mass limits are used, respectively. For the most Fornax-like cluster, we obtained K-S test p values of 0.001, 0.0009, and 0.4 for the low, mid, and high lower-mass limits. Clearly, the differences become smaller and stop being statistically significant when the comparison is made only for the high-mass range, where galaxies are well resolved by the simulation. On the other hand, there are only few galaxies in the highest-mass range, so the statistical strength of the test is not very good in that part.

Fig. 17. Cumulative stellar-mass functions for the Fornax cluster (red lines) and IllustrisTNG Fornax-like groups (gray lines). The IllustrisTNG group most similar to Fornax is shown with the green lines. The different panels show the comparisons using different stellar-mass ranges: The upper panel shows galaxies with 6.5 < log(M*/M⊙) < 9, the middle panel shows galaxies between 7 < log(M*/M⊙) < 9, and the bottom panel shows galaxies with 8 < log(M*/M⊙) < 9.

A similar conclusion to ours regarding the low-luminosity-end LF slopes between Cen A and IllustrisTNG-100 was also obtained by Müller et al. (2019). In their comparison, which was made for groups with lower masses than those in this work, they found that the observed LF low-luminosity end is steeper than found in the simulation.

7.3. Shape of the low-mass galaxies

We found that galaxies with log₁₀(M_*/M_⊙) > 7 require an inclination correction in order to remove the correlation between the SB and apparent axis ratio of the galaxies. On the other hand, the less massive galaxies do not have a correlation between these observables, which can indicate that they are not (thin) disks or that the shape distribution consists of galaxies with various shapes.

Sánchez-Janssen et al. (2019) found that the low-luminosity (−10 mag > M_g > −15 mag) galaxy disks become thicker with decreasing total luminosity. According to that work, this result seems to hold regardless of the galaxy morphology, although absolute thickness of the disks of different morphological classes differ. Similar results were obtained by Kado-Fong et al. (2021). These results are compatible with our findings that low-mass galaxies appear not to be thin disks, but we do not find an increasing trend in the mean b/a distributions (Fig. 12), which would indicate a systematic thickening of the galaxies.

Su et al. (2021) analyzed the FDSDC galaxies using multicomponent photometric decompositions and found that the fraction of galaxies with multiple components, such as a disk and a bar or a disk and a bulge, decreases toward lower masses. A similar finding for dwarf galaxies in the Virgo cluster was made by Janz et al. (2016). Additionally, Su et al. showed that the fraction of galaxies described well with a single Sérsic component also increases for the Spitzer Survey of Stellar Structure in Galaxies (S⁴G, Sheth et al. 2010; Salo et al. 2015) toward lower masses. This disappearance of disk or multicomponent features happens around −18 mag > M_r′ > −16 mag, which is the same range where the correlation between the observed b/a and ${\overline{\mu }}_{e,r}$ in our sample disappears. These findings may be an indication that the galaxies less luminous than that may have more dispersion-dominated thick “disks” that are not cool enough to be forming typical disk structures found in the larger disk galaxies. If we assume that this interpretation is true, it would have significant consequences for the kinematical analysis of the dwarf galaxies.

7.4. Connection between size and other properties

In Sect. 6.2 we showed that the SB of galaxies depends on projection effects and on its stellar populations. However, even after transforming the stellar light into stellar mass and correcting for the projection effects galaxies at a given mass have a large range of surface densities and thus different sizes. Taking into account the complexity of galaxy evolution, it is a rather unsurprising result that present day galaxies of a given mass may have a range of sizes. Anyways, at the same time, it is possible that some galaxies could be out of equilibrium and their sizes temporarily increased due to tidal interactions with other galaxies and the cluster potential.

Using only the currently available data, it is not possible to know the individual 3D positions and velocities of the galaxies and calculate the exact tidal forces acting on them. The tidal forces are strongest in the center of the cluster, and thus the projected distance from the cluster center can be statistically used as a proxy for the tidal forces.

Tidal forces affect the dwarf galaxy structure and morphology, so we can try to identify whether we can detect such signs in the Fornax cluster dwarfs. Possible effects of tidal interactions are distortions in the shapes of the galaxies (i.e., increased elongation and possibly tidal tails and arms). Since tidal forces act in the radial direction, during and immediately after the tidal encounters, elongation of interacting galaxies tend to be aligned toward the source of the tidal force. This may manifest into the alignment of the galaxies in the cluster, so the tidally perturbed galaxies may be preferably elongated toward the cluster center. If the effects of the tidal forces are significant at the scale of the whole dwarf population, we should also detect more elongated galaxies in the inner parts of the cluster and find more tidally disturbed morphologies within those galaxies than those in the cluster outskirts.

In Fig. 19 we show the cumulative distributions of cluster-centric distance, cluster-centric alignment angle, axis ratio, and color for dwarfs. We divided the sample into three parts based on the excess size (R_e) of the galaxies with respect to mean size at given mass. The three groups considered are: galaxies that deviated by less than 1σ from the distribution mean (blue lines), galaxies that deviate by more than 1σ (green lines), and galaxies that deviate by more than 2σ (red lines). Assuming that the environment or tidal interactions do not play a role in the sizes of the galaxies, we should not find any differences between the samples. However, we do find differences in the properties of the normal galaxies and the galaxies with large R_e: in addition to the concentrated distribution of the large R_e galaxies (K-S test for the assumption of similar distributions p = 0.002) in the cluster, they seem to also have a slight tendency to have their major axis preferably aligned toward the cluster center (p = 0.09 for more than 2σ deviators) and have larger fraction of galaxies with particularly small axis ratios (p = 0.01). There is no significant difference between the colors of the different samples.

Fig. 18. Distribution of tidal morphologies. Upper panels: cumulative distributions of cluster-centric distance (left panel), excess size (middle panel), and axis ratios (right panel) for the galaxies with different tidal morphologies. The colors of the curves in the panels correspond to the morphological classes named in the bottom panel pie charts and indicated with the same colors. Bottom panels: pie charts showing the fractions of the tidal morphological classes among galaxies that deviate by less than 1σ from the mass normalized Re distribution mean (left), galaxies that deviate by more than 1σ (middle), and galaxies that deviate by more than 2σ (right).

Fig. 19. Distribution of galaxies with large excess size. Cumulative distributions of cluster-centric distance (top panel), cluster-centric orientation (second panel), axis ratio (third panel), and color (bottom panel) for the Fornax cluster dwarfs. The different colors indicate the galaxies with different excess sizes: The blue lines show the galaxies with excess Re < ⟨Re(M*)⟩ + 1σ, the green lines galaxies with Re > ⟨Re(M*)⟩ + 1σ, and the red lines galaxies with Re > ⟨Re(M*)⟩ + 2σ.

The radial alignment of dwarf galaxies in the Fornax cluster was also identified by Rong et al. (2019). However, they did find the alignment signal for all the dwarf galaxies independent of their size. We do not find a significant signal for all dwarfs, but only a weak preferable alignment for the ones with large excess size.

In addition to the previous analysis, we can also use the morphological classifications (Sect. 5) of tidal features in dwarfs to identify effects of the cluster’s tidal forces. In Fig. 18, we show the distributions of cluster-centric distances, excess size, and axis ratios for the different morphological types and show the appearance of the different tidal morphologies among the different excess size classes discussed before. We find that the cluster-centric distances decrease and excess sizes of the dwarfs increase from the galaxies with regular morphology to possibly disturbed and disturbed galaxies. The K-S test shows significant difference between the distributions of all the studied parameters for morphologies one (regular) and two (slightly disturbed, p = 0.02, 0.02, 0.02 for distance, axis ratio, and excess size, respectively). The differences are also significant between classes one and three (clearly disturbed, p = 0.07, 0.03, 1e-7 for distance, axis ratio, and excess size, respectively). The two disturbed classes only differ significantly by their excess size (p = 2e−5).

The results of the previous analyses are compatible with the idea that tidal forces drive dwarfs out of dynamical equilibrium in the inner parts of the cluster. Based on our calculations (Venhola et al. 2019), galaxies need to either travel closer than a few hundred parsec from the NGC 1399 in order to have the tidal radius within 2 R_e from the center or spend a few gigayears in the cluster to experience similar stripping due to harassment. However, with the available information we cannot identify with certainty whether these tidal-forces whose effects we identify in galaxies are related to the cluster potential or interactions between individual galaxies.

7.5. Formation of UDGs and LSB dwarfs

In Sect. 6 we analyzed the properties of dwarf subpopulations defined using their mass, SB and surface density and size. Regardless of the used nomenclature (that we adopted here just for the sake of comparing the distributions), there seems to be no bimodal clustering in the galaxy properties that would give a physical justification to assign separate names “LSB-dwarf” or “UDG” (see, e.g., Figs. 14, 15, and 9). In general, the SD of dwarfs drops with decreasing total luminosity; in addition to that dependence, at any given mass there are deviations from that trend that approximately follow a normal distribution with a standard deviation of 0.5 dex in the surface densities. As seen in Fig. 15, some of the UDGs deviate by less than 1σ from the mean ${\overline{\mu }}_{e,r^{\prime }}$ of the dwarfs and are thus typical dwarfs. In general, the names LSB dwarf and UDG are misleading and should not be understood as separate types of dwarf galaxies, but rather as referring to a normal galaxy with low SB. In light of our data, any SB limit for those classes seems arbitrary.

The formation mechanism of UDGs has been investigated in many theoretical studies during the last years (Sales et al. 2020; Jiang et al. 2019; Tremmel et al. 2020; Amorisco & Loeb 2016; Di Cintio et al. 2017). Based on these works, UDGs are a subpopulation of dwarf galaxies that form naturally in isolation (e.g., Amorisco & Loeb 2016; Di Cintio et al. 2017; Sales et al. 2020), but they are also produced in galaxy clusters via tidal forces and fading after quenching of their star-formation (Sales et al. 2020; Jiang et al. 2019; Tremmel et al. 2020).

According to simulations, the UDGs formed via the last environmental effects are a ≈60%-majority, which leads to the theoretical prediction that UDGs in clusters are more centrally concentrated in the cluster than other dwarfs (Sales et al. 2020; Jiang et al. 2019). Our analysis of the distributions of UDGs and dwarfs in the Fornax cluster is consistent with this theoretical prediction. If we compare the fraction of UDGs of the total dwarf galaxy population at different cluster-centric radii, within D < 0.5R_vir we get 14 UDGs out of 251 dwarfs = 5.6 ± 1.4%, within D < R_vir: 24/557 = 4.3 ± 0.8%, and D > R_vir 8/221 = 3.6 ± 1.2%. However, the statistics are not strong enough to give support for the significance of the differences.

The tidal forces in the cluster center do not affect only UDGs but the whole dwarf galaxy population. As discussed in Sect. 7.4, this phenomenon may also explain our finding that the RLSB dwarfs are more frequently disturbed, are distributed closer to the cluster center than the normal dwarfs, tend to have smaller apparent axis ratios than others, and that their major axes are preferably aligned toward the cluster center.

According to Jiang et al. (2019) and Sales et al. (2020), the tidally enlarged UDGs fell into the cluster earlier than the majority of the dwarf population. This fact makes mean ages of their stellar populations older than the average. This is consistent with our observation that UDGs tend to have redder-than-average colors (see Fig. 15). However, according to, for example, Sales et al. (2020), cluster UDGs that are not tidally enlarged but are large LSB galaxies in equilibrium falling for the first time into the cluster should also comprise 40% of the population.

Tremmel et al. (2020) showed that it is not only the tidal enlargement of the effective radii of galaxies, but also their fading stellar populations after the quenching of the star formation happens that makes the SB of galaxies fade in galaxy clusters. As a result from this physical process, Tremmel et al. find a correlation between the central SB and stellar age in the inner parts of dwarfs. We discussed this scenario in the Fornax cluster in our previous paper Venhola et al. (2019), and showed that it is likely that all the galaxies with M_* < 10⁸M_* are quenched by RPS when they enter the inner parts of cluster for the first time. We also showed that the correlation between the ${\overline{\mu }}_{e,r^{\prime }}$ and colors of the dwarfs are consistent with the fading scenario when studied in stellar mass bins. This correlation has been afterward identified also in the Virgo and Centaurus cluster dwarfs by Janz et al. (2021).

To test whether the correlation found in the simulations of Tremmel et al. still holds in the Fornax cluster after including the dwarf galaxies identified in this work to the analysis, we plot the SB versus color relation in the upper panels of Fig. 20. We find that there is no significant correlation between the absolute SB and colors of the galaxies. However, there exists the luminosity-SB relation for dwarfs, which needs to be taken in account. In the lower panels of Fig. 20 we compare the SBs with the mean SB at given mass and find the significant correlation between the color and SB.

Fig. 20. Surface density ans surface brightness as a function of color. Upper panels: galaxy g′−r′ colors as a function of their central (left) and mean effective (right) SBs. Lower panels: same measures, but the SBs are compared with the mean value at a given mass. Spearman rank correlation coefficients with their uncertainty are listed above the panels.

The dwarfs with log₁₀(M_*/M_⊙) > 8 follow a different relation than the less massive ones (upper panel of Fig. 21), so we exclude them from the further comparisons. In the mid panel of Fig. 21, we compare the remaining dwarfs with a single stellar population evolution track using the models of Vazdekis et al. (2010). We use a Kroupa initial mass function (IMF), and show the model for stellar population with metallicity of log₁₀(Z/Z_⊙) = −1.0. As the x axis shows the mass-dependent excess SB, the location of the stellar evolution track on the x axis is arbitrary. As shown by Venhola et al. (2019) in the Fornax cluster and Janz et al. (2021) for the Virgo cluster dwarfs, the stellar population evolution can fit the SB and color distribution of most of the galaxies. This is also the case after including the LSB extension. The scatter in the color axis is consistent with the typical 0.1–0.2 mag uncertainties of the colors.

Fig. 21. Effects of stellar population ageing. Two upper panels: galaxy g′−r′ colors as a function of their excess mean effective SB. Top and middle panels: Dwarf galaxies with log10(M*/M⊙) > 8 and log10(M*/M⊙) < 8, respectively, shown separately. Spearman rank correlation coefficients with their uncertainty are listed above the panels. In the middle panel, we also present a stellar population evolution model that shows the color and SB evolution of an aging stellar population (see text for details). The stellar population ages corresponding to different colors are shown in red on the left. The blue cross in the middle panel shows the average uncertainties in the parameters. Bottom panel: Cluster-centric distributions of the dwarfs located in the different quadrants of the middle panel.

The degeneracy between age and metallicity in the optical colors causes some difficulty in straightforwardly interpreting the colors. However, the change in metallicity from log₁₀(Z/Z_⊙) = 0 to −1.5, which covers the expected range for dwarfs well, only has a 0.3 dex change in the colors, which is on the order of measurement uncertainties. The change in metallicity does not have a significant effect on the SB evolution, so even if there is a range of metallicities among the dwarfs, a similar trend in the color-SB plane is expected (Janz et al. 2021).

If the SB and color distribution of dwarfs manifests the rapid quenching of star formation in the cluster environment and subsequent aging of the stellar populations, we should find a trend in the cluster-centric distributions of the dwarfs with different quenching times. In order to check this, we divided the excess SB-color relation into four quadrants (mid panel of Fig. 21) and plotted the cluster-centric distributions in the bottom panel of Fig. 21. If the locations of galaxies in the excess SB-color relation are caused by their evolution along the evolutionary curve of the stellar population, we should observe galaxies in the quadrant one to be the outermost population from the cluster center, galaxies in the second in the middle, and galaxies in the quadrant three as the innermost population. This is the case as it is shown in the bottom panel of Fig. 21.

The galaxies in the fourth quadrant of Fig. 21 are hard to explain with the model relying solely on the evolution of stellar populations: they are blue galaxies with particularly low SB. However, as discussed in Sect. 7.4, tidal forces likely cause disturbances in some galaxies making their SB fainter than typical. If this assumption is true we should find those galaxies distributed in the inner parts of the cluster. As seen in the bottom panel of Fig. 21, quadrant four galaxies are the innermost population together with the quadrant three galaxies. The differences in the distributions are significant between the galaxies in quadrants one and three (K-S test p = 0.0003) and two and three (P = 0.02).

Since both the age fading of stellar populations and tidal interactions affect the SB of a galaxy, but only the latter alters the surface mass density of the galaxy, it makes sense to recreate the middle panel of Fig. 21 using surface mass density instead of brightness. In the upper panel of Fig. 22, we show the excess SD-color relation for the Fornax cluster dwarfs. Now, the galaxy’s location on the color axis depends on the quenching time and the excess SD can be increased by the tidal forces. We acknowledge that taking into account those two processes is not enough to uniquely define a galaxy’s location on the given axis, but if the tidal interactions have significance, we should observe the tidally disturbed galaxies to have lower excess SD compared to galaxies with similar colors. In the upper panel of Fig. 21, we show galaxies with different amounts of morphological disturbance with different colors and show their cumulative distributions in the lower panel. We find that galaxies with disturbed morphology indeed tend to have lower SD than the typical galaxies at a given color. The differences between all the morphological classes are significant (K-S test p < 0.002).

Fig. 22. Relation between tidal morphology and surface density. Upper panel: Galaxy g′−r′ colors as a function of excess SD. The different colors correspond to galaxies with different amounts of disturbed morphology. Lower panel: cumulative distributions of the excess SD of the different morphological classes of the upper panel.

These observational results are well in agreement with the theoretical frameworks that Sales et al. (2020), Jiang et al. (2019), and Tremmel et al. (2020) have described for the formation of the UDGs. However, here we show that the stellar population aging and tidal interactions can be used to understand the formation of RLSB/RLSD dwarfs in the Fornax cluster in the whole dwarf population with log10(M_*/M_⊙) < 8.

8. Summary and conclusions

In this work we have quantified the detection efficiency and selection effects of the MTO using simulated images and the FDS data. The detection efficiency of MTO by far exceeds that of SE and thus justifies its use for LSB galaxy detection. We found that the MTO is able to increase the detection completeness in the FDS data by reaching 0.5–1 mag arcsec⁻² fainter SBs; additionally, by detecting more accurate sizes for the galaxies, it decreases the amount of galaxies that get excluded from the sample by the size cuts. We then combined MTO with an automated galaxy fitting pipeline that applies GALFIT to fit galaxies with Sérsic profiles. As a result, we obtained an LSB galaxy sample that extends the previous FDS dwarf galaxy catalog of the Fornax cluster by V18. Using the new catalog, we studied the low-luminosity end of the LF in the Fornax cluster and analyzed the structural trends of the dwarfs in the cluster. Our main results are the following:

• We identified 265 new LSB dwarf candidates in the Fornax cluster that were not included in the FDSDC. Extending the FDSDC with the galaxies of this work increases its total number of galaxies to 821. This galaxy sample is more than 75% complete down to M_r′ = −12 mag and ${\overline{\mu }}_{e,r^{\prime }}$ = 26 mag arcsec⁻² and reaches its 50% completeness limits at M_r′ = −10 mag and ${\overline{\mu }}_{e,r^{\prime }}$ = 26.5 mag arcsec⁻² (Fig. 1). The limits are 0.5–1 mag deeper than for the FDSDC. The faintest galaxies of the sample have stellar masses of M_* = 5 × 10⁴ M_⊙.

• The faint-end LF in the Fornax cluster has a slope of α = −1.38 ± 0.02 down to the completeness limit of our sample, M_r′ = −12 mag (Fig. 10). Our sample is size limited and thus excludes extremely compact galaxies, such as Ultra-Compact Dwarfs (UCD) and dwarfs with a < 200 pc.

• When galaxies are classified based on their SB or density, the galaxies that are sparser than average are more centrally clustered in the Fornax cluster than the dense galaxies (Fig. 13). Ultra-diffuse galaxies, being the innermost population of the cluster dwarfs in their mass range (as long as UCDs are not considered), follow the same trend.

• Our observation that the excess size of the galaxies with respect to the average galaxy at a given mass increases toward the center of the cluster (Fig. 17) is consistent with the theoretical prediction that the tidal forces of the cluster potential input energy to the galaxies and thus increase their sizes. Galaxies with large excess sizes tend to also be more elongated than normal galaxies, and their major axes are preferably aligned toward the cluster center (Fig. 17). We also observe that the number of galaxies with disturbed morphologies increases toward the central parts of the cluster. These observations give further support to the importance of tidal forces in the cluster.

• We find no evidence for variations in the low-luminosity end of the LF between the Fornax cluster and the different galaxy environments. This result holds for galaxies with −18.5 mag < M_V < −12.5 mag (Table 2 and Fig. 17).

• The low-mass slope of the galaxy stellar mass function in the Fornax cluster is steeper than in the cosmological simulation IllustrisTNG (Fig. 16). As these discrepancies become more severe toward the resolution limits of IllustrisTNG, it is likely that the differences are at least partly numerical.

• We find that there is a correlation between the apparent SB and axis ratio of dwarf galaxies with log₁₀(M_*/M_⊙) > 7 (Fig. 11), which indicates that they are disk-like. This correlation disappears for the less massive galaxies. This indicates that their shapes are not well approximated by disks.

• Based on the analysis of the dependence between the location, SB, colors, and morphology of dwarfs in the Fornax cluster (Fig. 19), we find that their properties can be explained with the theoretical models of the evolution of dwarfs within a cluster environment (Sales et al. 2020; Jiang et al. 2019; Tremmel et al. 2020). The effects of the two dominant mechanisms, tidal interactions and the aging of the stellar populations, can both be identified in the photometric properties of the dwarf galaxies.

To conclude, in this work we have given new empirical support for the significance of the tidal forces in the formation of low surface brightness galaxies in the galaxy clusters. Due to the increased completeness of our galaxy catalog, we have also better constrained the low-mass end of the galaxy mass function in the Fornax cluster than before.

Acknowledgments

A.V. would like to thank the Emil Aaltonen Foundation for the financial support during the writing of this paper. R.F.P., E.L., H.S., E.I., M.H.F.W., C.H. and J.J. acknowledge financial support from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721463 to the SUNDIAL ITN network. H.S., E.L., and A.V. are also supported by the Academy of Finland grant n:o 297738.

References

Appendix A: Details of catalog production

A.1. Pixel value distribution moments

MTO and SE both measure semimajor and semiminor axis lengths for the objects. For calculating these parameters, second-order pixel value distribution moments are used. If we mark x and y coordinates and intensity of a pixel i as x_i, y_i and I_i, respectively, we can calculate the second-order moments as follows

$$\begin{array}{c}\hfill \begin{array}{c}\overline{x^2}=\frac{{\sum }_i{I}_i{x}_i^2}{{\sum }_i{I}_i}-{\overline{x}}^2,\\ \overline{y^2}=\frac{{\sum }_i{I}_i{y}_i^2}{{\sum }_i{I}_i}-{\overline{y}}^2,\\ \overline{\mathit{xy}}=\frac{{\sum }_i{I}_i{x}_i{y}_i}{{\sum }_i{I}_i}-\overline{x}\overline{y}.\end{array}\end{array}$$(A.1)

The second-order moments measure the spread along the x and y axes, and they are transformed to semiminor axis b and semimajor axis a as

$$\begin{array}{c}\hfill \begin{array}{c}{a}^2=\frac{\overline{x^2}+\overline{y^2}}{2}+\sqrt{{\left(\frac{\overline{x^2}-\overline{y^2}}{2}\right)}^2+{\overline{\mathit{xy}}}^2},\\ b^2=\frac{\overline{x^2}+\overline{y^2}}{2}-\sqrt{{\left(\frac{\overline{x^2}-\overline{y^2}}{2}\right)}^2+{\overline{\mathit{xy}}}^2},\end{array}\end{array}$$(A.2)

and the position angle θ,

$$\begin{array}{c}\hfill tan(2\theta )=2\frac{\overline{\mathit{xy}}}{\overline{x^2}-\overline{y^2}}.\end{array}$$(A.3)

In general, the major-axis a calculated from the pixel value distribution moments is not straightforwardly comparable with R_e. Their ratio is dependent on the concentration (i.e., the Sérsic index) of the profile. In order to give the reader insight how these two are related, we calculated the ratio between a and R_e for a pure Sérsic profile and show their ratio as a function of n in Fig. A.1. From the figure we see that the ratio of the two size measures is close to unity for the Sérsic n values typical for dwarfs (0.5 < n < 2). We also find that the relation can be well approximated by a fitting formula:

$$\begin{array}{c}\hfill \frac{a}{{\mathrm{R}}_e}=0.7\times exp\left(\frac{n}{2.38}\right).\end{array}$$(A.4)

Fig. A.1. Ratio of a and Re as a function of Sérsic index for a pure Sérsic profile. The black curve shows the values calculated from moments, and the dashed red line shows the analytical approximation.

A.2. SE configuration parameters

Configuration parameters of SE are described in detail in the SE guide¹⁵. The most important parameters for the object detection are the size of the background grid, detection threshold and the minimum area of the detections. One can also apply different convolution filters for images, but in order to keep the comparison simple, we used a simple 3 pix × 3 pix Gaussian filter with full width half maximum (FWHM) 2 pix, which is also used by MTO. In order to decide which parameter values we should adopt for the detection tests we ran a test for a set of parameters. We selected a few reasonable values for each parameter and ran SE using all combination of those parameters for one of the mock images described in Sect. 2.2. We used values of 500, 1000, and 10000 pixels for the background grid size, 0.5 and 1σ for the detection threshold, and 10, 50, and 100 pixels for the minimum detection area. We then selected the set of parameters that provided the best combination of completeness and purity of the detections. The optimal performance was obtained using a background size of 500 pixels, a detection threshold of 1σ, and a minimum detection area of 10 pixels. We then used the following set of configuration parameters when running SE:

DETECT_MINAREA   10,          # minimum number of pixels
                                above
                              # threshold
DETECT_THRESH    1,           # <sigmas>
ANALYSIS_THRESH  1,           # <sigmas>
FILTER           Y,           # apply filter for detection
DEBLEND_NTHRESH  64,          # Number of deblending
                              # sub-thresholds
DEBLEND_MINCONT  0.005,       # Minimum contrast for
                                deblending
CLEAN            Y,           # Clean spurious detections?
CLEAN_PARAM      1.0 ,        # Cleaning efficiency
MASK_TYPE        CORRECT,     # type of detection MASKing :
                              # NONE, BLANK or CORRECT
MAG_ZEROPOINT    0.0,         # magnitude zero-point
PIXEL_SCALE      0.2,         # size of pixel in arcsec
SEEING_FWHM      1.2,         # stellar FWHM in arcsec
STARNNW_NAME     default.nnw, # Neural-Network_Weight
                              # table filename 
BACK_SIZE        500,         # Background mesh <size>
BACK_FILTERSIZE  3,           # Background filter <size>
BACKPHOTO_TYPE   GLOBAL,      # can be GLOBAL or LOCAL
MEMORY_OBJSTACK  3000,        # number of objects in stack
MEMORY_PIXSTACK  3000000,     # number of pixels in stack
MEMORY_BUFSIZE   1024,        # number of lines in buffer

A.3. Detection completeness

In Fig. A.2 we show how the detections and the input galaxy properties are distributed in the different parameter spaces.

Fig. A.2. Detections (orange points) and non-detections (blue points) of the mock galaxies using MTO. The matrix plots show the distribution of galaxies in the different parameter subspaces. The upper histograms show the distributions of detected (orange) and all (blue) galaxies for the given parameters.

A.4. Examples of detection artifacts

MTO forms the max-tree by identifying sets of connected pixels with different thresholds and then labels the branches according to the rules given in Sect. 3. As MTO does not assume any profile for the detected objects, some parts of the diffuse light are often associated with a wrong object. This is especially the case with interacting galaxies and densely grouped galaxies. Their light profiles are connected at low levels and get associated with one of the galaxies, thus making the galaxy’s mean SB low.

Another type of low SB features that gets identified by MTO but is considered as an artefact in our LSB dwarf catalog are stellar reflection halos. As shown by V18, the inner reflections of the VST cause the PSF to have reflection rings around bright objects. The SB of those rings is ≈ 10 mag arcsec⁻² fainter than the peak of the PSF and have angular size on the order of 5 arcmin. The SB and size of these features are similar to LSB dwarfs, but they are easy to separate from each other based on their morphology and the fact that reflection rings appear near bright stars. Most of the false positives caused by the reflection rings can be identified using stellar catalogs of the Fornax area.

A.5. Identification of nuclei

In order to automatically decide whether a galaxy needs to be fitted with a Sérsic or Sérsic+nucleus, we compare the Gini coefficient of central part of the galaxy with the Gini coefficient of the PSF. The calculation is done within the innermost two arcsec from the center of the galaxy and the PSF. The Gini coefficient describes the distribution of flux within the selected pixels and is defined as follows:

$$\begin{array}{c}\hfill G=\frac{{\mathrm{\Sigma }}_{i=1}^n{\mathrm{\Sigma }}_{j=1}^n|x_i-x_j|}{2n^2\overline{x}},\end{array}$$(A.5)

where x_i and x_j correspond to the fluxes within the pixels i and j, n is the number of pixels, and $\overline{x}$ is the average flux within the pixels. Value of G = 1 means that all the flux is in the brightest pixel and G = 0 means that all the pixels have equal amount of flux.

As G describes how large fraction of the flux is in the brightest pixels, a point source has the maximum G that any object can have. In the case of astronomical images, the signal is convolved with the PSF, so the fraction G_obj/G_PSF tells how similar an object is to a point source. If we consider a typical dwarf galaxy in the Fornax cluster, its light profile is usually well described by a Sérsic profile with 0.5 < n < 1.5, and it may or may not have a point-like nucleus in the center. Clearly, if there is a point source that has high contrast with respect to the main body of the galaxy, the G_obj/G_PSF will be close to one in the core. However, the nucleus can also be faint with respect to the Sérsic component of the profile, or the profile may have high n, which makes it difficult to identify the nucleus. In order to test what would be a practical G_obj/G_PSF threshold value for indicating likely presence of nucleus, we generated a set of simulated galaxies with different SB, Sérsic n and nucleus fraction. We then calculated G_obj/G_PSF for all the galaxies.

First we studied how G_obj/G_PSF behaves as a function of Sérsic n and R_eof the galaxy. We simulated noiseless galaxies with different n and R_e and show the results in Fig. A.4. Our results show that G_obj/G_PSF is close to 0.5 – 0.6 for an exponential profile and then grows close to one when n grows. The size of the galaxy does not effect the value much as long as the galaxy is resolved.

Fig. A.3. False positives. Examples of sources that appear as LSB galaxies in the MTO detection but have different physical natures. The left panels show color images generated from the FDS data, and the right panels show the segmentation maps, where different object labels are shown with different colors. The two upper examples show groups of galaxies that are connected on low SB levels and where the common LSB envelope is associated with one of the galaxies, making its mean SB low. The two lower panels show reflection rings around bright stars. Those objects have similar SBs as LSB dwarfs.

Fig. A.4. Gobj/GPSF ratio as a function of Sérsic index for non-nucleated galaxies. The different panels correspond to galaxies with different effective radii.

In Fig. A.5 we show the G_obj/G_PSF ratio for nucleated and non-nucleated exponential galaxies with different SBs. In this simulation, we also add Gaussian noise to the images that corresponds to the FDS r’-band noise. In the case of the non-nucleated galaxy with the highest SB, the galaxy light dominates over the background noise and the G_obj/G_PSF is close to 0.5. With decreasing SB, the value goes down and eventually reaches G_obj/G_PSF = 0.3, which is identical to the value of noise. In the case of the nucleated galaxy, the G_obj/G_PSF value is close to 1 when the fraction of light in the nucleus is large and reaches the same value as the non-nucleated galaxy when the fraction of light in the nucleus decreases. The nucleus fraction at which the nucleated and non-nucleated galaxies become indistinguishable is dependent on the SB of the galaxy. In the case of the brightest dwarfs we can detect a nucleus that has 1/100th of the luminosity of the Sérsic profile, but in the case of the faintest dwarfs only a nucleus with luminosity larger than 1/10th of the Sérsic profile can be identified using the G_obj/G_PSF value.

Fig. A.5. Limits of nuclear cluster detection. Gobj/GPSF calculated for an exponential profile (red symbols), a nucleated exponential profile (blue symbols), and noise (gray symbols). The values on the x axis indicate the difference between the total magnitude of the simulated nucleus and that of the galaxy, and thus those values are only relevant for the red symbols. For other symbols, the different locations on the x axis correspond to different realizations of the same galaxy. The panels show the Gobj/GPSF values for galaxies with different SBs.

Based on these tests, we notice that G_obj/G_PSF ≈ 0.3 for noise and LSB exponential profiles. The G_obj/G_PSF ratio grows as Sérsic n grows, but since most of the dwarfs have 0.5 < n < 1.5, the typical maximum value of G_obj/G_PSF for a dwarf without a nucleus is around 0.5. However, for the non-nucleated dwarfs, the G_obj/G_PSF ratio quickly drops below 0.4 with decreasing SB but remains above that value for nucleated dwarfs (as long as the nucleus can be identified from the data). Thus, we selected G_obj/G_PSF = 0.4 as the threshold to include a nucleus into the GALFIT model.

Appendix B: Catalog of LSB dwarfs

Photometric parameters of the likely Fornax galaxies that were not identified by V18 are presented in an electronic catalog that is published with this paper. In Table B.1, we present a small sample of the catalog to demonstrate its contents.

All Tables

All Figures

Fig. 1. Results of detection completeness tests. Detection completeness of SE (left panels) and MTO (right panels) as a function of the input r′-band mean effective SB (${\overline{\mu }}_{e,r}$), effective radius (Re), and axis ratio (b/a). Upper and middle panels: results for the artificial images and the lower panels for the known core LSB catalog galaxies (Venhola et al. 2017). The gray scales in the upper and middle panels correspond to the completeness indicated by the color bar. The green and blue contours show the 75% and 50% completeness limits, respectively. Those same lines are also shown in the lower panels. In the lower panels, squares show the detected galaxies and red triangles show the non-detections.

In the text

Fig. 2. Accuracy of parameters for initial detections. Upper panels: ratio between the measured semimajor axis length and the input effective radii (aout/Re, in) for the mock galaxies as a function of their mean effective SB in the r′ band (${\overline{\mu }}_{e,r^{\prime }}$). Lower panels: difference between the input magnitudes (min) and the measured magnitudes (mout). Left and right panels: comparisons when using SE and MTO, respectively. The blue (MTO) and red (SE) lines correspond to the running means of the measurements with a filter size of $\mathrm{\Delta }{\overline{\mu }}_{e,r^{\prime }}$ = 1 mag arcsec−2. Shaded areas around the running means show the median absolute deviation from the means with the corresponding colors. The gray and black symbols in the right panels correspond to the objects that are only detected using MTO and to the objects detected using both methods, respectively.

In the text

Fig. 3. Comparison of the extracted parameters between SE and MTO. Left panel: ratio between the circularized radii of the objects obtained by SE (${\sqrt{\mathit{ab}}}_{\mathrm{SEx}}$) and by MTO (${\sqrt{\mathit{ab}}}_{\mathrm{MTO}}$) as a function of the input r′-band mean effective SB (${\overline{\mu }}_{e,r^{\prime }}$). Right panel: difference between the apparent magnitudes obtained with SE (mSEx) and MTO (mMTO). The red lines show the running medians of the measurements with a filter size of $\mathrm{\Delta }{\overline{\mu }}_{e,r^{\prime }}$ = 1 mag arcsec−2.

In the text

	Fig. 4. Objects missed due to offset in center coordinates. Examples of relatively bright objects that were not considered as detected with MTO by our test in Sect. 3.2.2 since the center of the detected object is too far from the dwarf center. Left panels: r′-band postage stamps, and the right panels: cutouts from the segmentation maps. The target objects are shown with the small crosses.
In the text

Fig. 5. GALFIT photometry. Example of the steps done in order to obtain the photometric parameters. First, the sigma-, science- and segmentation-image postage stamps centered on the object of interest are cut (left panels). The objects are selected for the mask and the GALFIT model (the green and red symbols in the second column picture from the left, respectively). A mask is generated (the upper panel of the third column from the left) using the segmentation map, and the initial model (the lower panel of the third column from the left) is built using the MTO object locations, magnitudes, and sizes. We note the large fuzzy object in the input model above and to the left of the target, which corresponds to the extended source outside the image field of view: The object is placed within the image in the input model, but it is allowed to move outside the image edges during the fitting. GALFIT is then used for model fitting. The right column panels from top to bottom show the fitted model with and without the noise with the mask overlaid and the residuals after subtracting the model from the data.

In the text

Fig. 6. GALFIT parameter accuracy. Comparison between the input and output values of the structural parameters of the mock galaxies. The upper panel shows the differences between the input and output magnitudes (mr′), effective radii (Re), Sérsic indices (n), and axis ratios (b/a) as a function of input mean effective r′-band SB (${\overline{\mu }}_{e,r^{\prime }}$). The black symbols show individual measurements, and the blue line shows the running mean with a 2 mag arcsec−2 filter size. The corresponding standard deviations are shown with the dashed blue lines. The lower panels show the magnitude of the standard deviations between the input and output values. The blue curves show the fits of Eq. (4) to the data.

In the text

Fig. 7. Parametric cuts used for the selection of LSB candidates. The panels show g′−r′ colors (top panel), SBs (middle panel), and the concentration parameter, C (bottom panel of the galaxies), as a function of their r′-band apparent magnitude, mr′. The selected galaxies (black symbols) and the galaxies rejected for being too red (red symbols), having too high of a SB (green symbols), and a too high C (blue symbols) are indicated with the different symbols. The black lines in the panels show the selection cuts (see the text and V18).

In the text

	Fig. 8. Tidal morphology. Examples of the different morphological labels given for the galaxies based on visual inspection. The galaxy in the middle panel shows slight irregularities in its outskirts and is thus classified as slightly disturbed. The galaxy in the bottom panel shows clear tidal arms, which justifies its classification as clearly disturbed.
In the text

Fig. 9. Surface brightness of Fornax cluster dwarfs. Apparent r′-band magnitudes (mr′) and mean effective SBs (${\overline{\mu }}_{e,r^{\prime }}$) of the FDSDC galaxies (black symbols) compared with the LSB extension galaxies described in this work (red symbols). The histograms on the x and y axes show the distributions of the magnitudes and SBs with the same colors as in the scatter plot. The blue and green contours show the 50% and 75% detection limits found using the artificial galaxies (Sect. 3.2), and the black and dashed gray lines show the a > 2 arcsec and SB selection limits, respectively. There are also galaxies with Re < 2 arcsec as the selection was based on MTO detection a > 2 arcsec. For a comparison, we also show a sample of LG dwarfs and globular clusters (green squares clearly above these selection limits) from Brodie et al. (2011).

In the text

Fig. 10. Galaxy luminosity and mass functions. Upper and middle panels: r′- and g′-band LFs of the Fornax cluster dwarf galaxies, respectively. Bottom panel: stellar mass function. The black, red, blue, and green histograms show the LF for all the galaxies, for galaxies within the virial radius of the Fornax cluster, those outside of it, and those within the cluster core, respectively. The curves show the Schechter function fits to the data, and the fit parameters are shown with corresponding colors in the upper-left corner of the image. The dotted gray lines show the completeness limits for the different parameters.

In the text

Fig. 11. Inclination corrections. Correlation between the axis ratio (b/a), mean effective SB (${\overline{\mu }}_{e,r^{\prime }}$), and stellar SD (${\overline{\mathrm{\Sigma }}}_{e,\ast }$) of the galaxies. Upper panels: values without any inclination corrections, in the middle panels galaxies with log(M*/M⊙) > 7 have been corrected for their inclination, and in the bottom panels inclination correction has been applied to all the galaxies. The different colors correspond to different mass bins, as shown in the legend. The solid and dashed lines indicate the running mean (filter size of 0.2) and the mean of the mass bin.

In the text

	Fig. 12. Axis ratio trends. Average axis ratios of the dwarf galaxies in the Fornax cluster in mass (upper panel) and absolute r′-band magnitude (lower panel) bins. The symbols indicate the average values with their uncertainties. The mass bins have a width of 0.6 dex, and the magnitude bins have a width of 1 mag.
In the text

	Fig. 13. Sérsic indices as a function of galaxy stellar mass. The black points show the measurements for the dwarf galaxies in the Fornax cluster, and the red line shows the running average with a filter size of 0.5 dex. The green and red shaded areas correspond to the scatter expected from the measurement uncertainties and the measured scatter, respectively.
In the text

Fig. 14. Cluster-centric trends in galaxies’ relative surface brightness and density. Upper-left and lower-left panels: de-projected mean effective SB (${\overline{\mu }}_{e,d,r^{\prime }}$) and stellar SD (${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$) as a function of galaxy stellar mass (M*) for the Fornax cluster dwarfs, respectively. The solid and dashed red lines show the running mean and standard deviations from that, respectively. The RHSB galaxies, RLSB galaxies, normal galaxies, and UDGs are indicated with the different colors, shown in the legend in the upper-left panel. The definitions of those classes are described in the text. Upper-middle panel: normalized mass distributions of the different SB subsamples in the mass bins indicated with the vertical lines in the upper-left panel. Lower-middle panel: correlation between $\mathrm{\Delta }{\overline{\mu }}_{e,d,r^{\prime }}$ and $\mathrm{\Delta }{\overline{\mathrm{\Sigma }}}_{e,d,\ast }$. Right panels: cumulative cluster-centric distance distribution for the different subsamples when defined using ${\overline{\mu }}_{e,d,r^{\prime }}$ (upper-right panel) and ${\overline{\mathrm{\Sigma }}}_{e,d,\ast }$ (lower-right panel). K-S test p values for the assumption of two samples being drawn from the same distribution are also listed for all the sample pairs in the lower right of the right panels.

In the text

	Fig. 15. Correlations between the excess mean effective stellar SD and properties of the FDS dwarfs. Upper panel: comparison with the excess color (with respect to the mean at a given mass), the middle panel with the excess Sérsic index, and the lower panel with the excess effective radius. The red symbols highlight the UDGs, and the black symbols show the other dwarfs.
In the text

Fig. 16. Empirical cumulative luminosity functions. Comparison of the V-band LFs in the Fornax cluster, the LG, and other nearby environments. We use the compilation of Ferrarese et al. (2016) for the LG and Virgo, the Müller et al. (2019) compilation for the Centaurus A group, the compilation of Carlsten et al. (2021) for other nearby groups, and the galaxies in this work for the Fornax cluster and the Fornax A group. The vertical dashed lines show the cutoff limits of MV = −18 mag and MV = −12.5 mag.

In the text

Fig. 17. Cumulative stellar-mass functions for the Fornax cluster (red lines) and IllustrisTNG Fornax-like groups (gray lines). The IllustrisTNG group most similar to Fornax is shown with the green lines. The different panels show the comparisons using different stellar-mass ranges: The upper panel shows galaxies with 6.5 < log(M*/M⊙) < 9, the middle panel shows galaxies between 7 < log(M*/M⊙) < 9, and the bottom panel shows galaxies with 8 < log(M*/M⊙) < 9.

In the text

Fig. 18. Distribution of tidal morphologies. Upper panels: cumulative distributions of cluster-centric distance (left panel), excess size (middle panel), and axis ratios (right panel) for the galaxies with different tidal morphologies. The colors of the curves in the panels correspond to the morphological classes named in the bottom panel pie charts and indicated with the same colors. Bottom panels: pie charts showing the fractions of the tidal morphological classes among galaxies that deviate by less than 1σ from the mass normalized Re distribution mean (left), galaxies that deviate by more than 1σ (middle), and galaxies that deviate by more than 2σ (right).

In the text

Fig. 19. Distribution of galaxies with large excess size. Cumulative distributions of cluster-centric distance (top panel), cluster-centric orientation (second panel), axis ratio (third panel), and color (bottom panel) for the Fornax cluster dwarfs. The different colors indicate the galaxies with different excess sizes: The blue lines show the galaxies with excess Re < ⟨Re(M*)⟩ + 1σ, the green lines galaxies with Re > ⟨Re(M*)⟩ + 1σ, and the red lines galaxies with Re > ⟨Re(M*)⟩ + 2σ.

In the text

	Fig. 20. Surface density ans surface brightness as a function of color. Upper panels: galaxy g′−r′ colors as a function of their central (left) and mean effective (right) SBs. Lower panels: same measures, but the SBs are compared with the mean value at a given mass. Spearman rank correlation coefficients with their uncertainty are listed above the panels.
In the text

Fig. 21. Effects of stellar population ageing. Two upper panels: galaxy g′−r′ colors as a function of their excess mean effective SB. Top and middle panels: Dwarf galaxies with log10(M*/M⊙) > 8 and log10(M*/M⊙) < 8, respectively, shown separately. Spearman rank correlation coefficients with their uncertainty are listed above the panels. In the middle panel, we also present a stellar population evolution model that shows the color and SB evolution of an aging stellar population (see text for details). The stellar population ages corresponding to different colors are shown in red on the left. The blue cross in the middle panel shows the average uncertainties in the parameters. Bottom panel: Cluster-centric distributions of the dwarfs located in the different quadrants of the middle panel.

In the text

	Fig. 22. Relation between tidal morphology and surface density. Upper panel: Galaxy g′−r′ colors as a function of excess SD. The different colors correspond to galaxies with different amounts of disturbed morphology. Lower panel: cumulative distributions of the excess SD of the different morphological classes of the upper panel.
In the text

	Fig. A.1. Ratio of a and Re as a function of Sérsic index for a pure Sérsic profile. The black curve shows the values calculated from moments, and the dashed red line shows the analytical approximation.
In the text

	Fig. A.2. Detections (orange points) and non-detections (blue points) of the mock galaxies using MTO. The matrix plots show the distribution of galaxies in the different parameter subspaces. The upper histograms show the distributions of detected (orange) and all (blue) galaxies for the given parameters.
In the text

Fig. A.3. False positives. Examples of sources that appear as LSB galaxies in the MTO detection but have different physical natures. The left panels show color images generated from the FDS data, and the right panels show the segmentation maps, where different object labels are shown with different colors. The two upper examples show groups of galaxies that are connected on low SB levels and where the common LSB envelope is associated with one of the galaxies, making its mean SB low. The two lower panels show reflection rings around bright stars. Those objects have similar SBs as LSB dwarfs.

In the text

	Fig. A.4. Gobj/GPSF ratio as a function of Sérsic index for non-nucleated galaxies. The different panels correspond to galaxies with different effective radii.
In the text

Fig. A.5. Limits of nuclear cluster detection. Gobj/GPSF calculated for an exponential profile (red symbols), a nucleated exponential profile (blue symbols), and noise (gray symbols). The values on the x axis indicate the difference between the total magnitude of the simulated nucleus and that of the galaxy, and thus those values are only relevant for the red symbols. For other symbols, the different locations on the x axis correspond to different realizations of the same galaxy. The panels show the Gobj/GPSF values for galaxies with different SBs.

In the text