© The Authors 2025

1. Introduction

Globular clusters (GCs) are found in galaxies of all morphologies and sizes, spanning a wide range of masses and magnitudes (e.g. Harris 1991; Georgiev et al. 2010; Harris et al. 2013). As these are primarily old stellar systems, studying the spatial distributions of GCs and their stellar populations through their colours and phase-space distribution (if radial velocities are available) can provide crucial insights into the interaction events of the host galaxy and, more broadly, the galaxy group or cluster environment (Brodie & Strader 2006; Georgiev et al. 2009). Observations of GCs are used to constrain the star formation and assembly history of galaxies, serving as valuable tools in theoretical and observational astronomy across a wide range of research topics in extragalactic astrophysics (Harris 2001; Peng et al. 2008; Durrell et al. 2014; Forbes 2020).

A globular cluster (GC) is a class of dense stellar agglomerates characterised as bright (mean absolute magnitude of M_V = −7.5 mag), compact (mean effective radii of r_e = 3 parsec)¹, and old (typical ages of ≥10 Gyr), with a stellar mass usually ranging between 10⁴ and 10⁶ M_⊙ (Harris 2001). As a first approximation, GCs host a simple stellar population (SSP), that is they have a single age and single metallicity. However, studies of GCs in our Galaxy (e.g. Piotto et al. 2007; Carretta et al. 2010) and the Magellanic Clouds (e.g. Milone et al. 2017; Martocchia et al. 2019) have revealed the presence of multiple stellar populations with varying light element abundances. Despite this, GCs host stellar populations that are much simpler than those of galaxies, in terms of iron abundance and age distributions (Gratton et al. 2019). Therefore, the intrinsic simplicity of GCs, together with their old age and high luminosity, makes them a powerful and robust tracer of a galaxy and its environment out to cosmological distances (Alamo-Martínez et al. 2013; Lee et al. 2022; Harris & Reina-Campos 2023). In the last few decades, and particularly after the Hubble Space Telescope (HST) began collecting data with superb image quality, the increased depth and quality of multi-passband data available for the detection of extragalactic GC systems have allowed astronomers to unveil numerous properties that represent a valuable tool to constrain the host galaxy properties. Because of the near-universal Gaussian shape of the GC luminosity function (GCLF) in the optical and near-IR passbands, which exhibits a peak called the turn-over magnitude (TOM) at $M_V^{\mathrm{TOM}}$ ∼ −7.5 mag (Rejkuba 2012), the use of old GCs has increased as a standard candle to act as a distance indicator (Hanes 1977; Lee et al. 2018; Ferrarese et al. 2020).

Most GC systems exhibit a bimodal colour distribution with a blue and a red peak (Brodie & Strader 2006). A combination of photometric and spectroscopic observations indicate that these GCs are quite homogeneous in terms of their old age, and so the colour differences mainly reflect their metallicity differences. Brodie & Strader (2006) attributed this to the hierarchical formation, which gives rise to two distinct GC sub-populations that have different peak metallicities. While the red GCs follow the radial trend of the stars in the bulge or the spheroid of the host galaxy and are more concentrated in their inner regions, the blue GCs have been observed to exhibit a wider radial extension (e.g. Harris 2009; D’Abrusco et al. 2016; D’Abrusco et al. 2022; Cantiello et al. 2020; Hazra et al. 2022). This seems to indicate that the red GC population is formed in situ, while the blue GCs could represent a population acquired through mergers, accretion and other tidal interactions (Brodie & Strader 2006; Forbes et al. 2011).

Harris & van den Bergh (1981) introduced the parameter specific frequency (S​_N) as a measure of the richness of the GCs normalised to their host galaxy’s luminosity. It is given by the following equation:

$$\begin{array}{c}\hfill S_{\mathrm{N}}=N_{\mathrm{GC}}\times {10}^{0.4(M_V+15)},\end{array}$$(1)

where N_GC is the total number of GCs and M_V is the V-band absolute magnitude of the host galaxy. This quantity has since been widely used to assess galaxy formation mechanisms. Recent discussions on this topic can be found in Georgiev et al. (2010), Harris et al. (2013), Müller et al. (2021), and Marleau et al. (2025). In particular, Harris et al. (2013) studied the GCs of a large sample of galaxies of different morphologies situated in different environments and showed the behaviour of S​_N with M_V. They found that the mid-range luminosity galaxies (10⁹ − 10¹⁰ L_⊙) form a relatively uniform group with S​_N ≈ 1, while the dwarfs and giant galaxies at the opposite ends show a higher mean S​_N (although with a large scatter).

Interactions between galaxies can lead to the stripping of field stars and GCs, forming tidal debris and contributing to the diffuse intra-group light (IGL) observed in galaxy systems (e.g. Arnaboldi & Gerhard 2022; Spavone et al. 2018). The IGL serves as an effective tracer of stripped stellar components, providing valuable insights into the past interactions and history of galaxy systems (Arnaboldi & Gerhard 2022; Ahvazi et al. 2024; Kluge et al. 2025). Given that groups host at least 50% of galaxies in the universe today (Muñoz et al. 2013), studying GCs in these environments is key to understanding galaxy evolution.

Much remains to be understood about GCs. The origin of the observed colour bimodality has been highly debated, for instance, in the studies of Yoon et al. (2006) and Cantiello & Blakeslee (2007). It has been broadly suggested that it could result from a continuous metallicity distribution combined with non-linear colour-metallicity relations. We also lack clarity on the mechanisms responsible for stripping and dispersing GCs from their host galaxies, as well as their relative efficiencies (Forbes et al. 2018). Additionally, it is unclear what fraction of the original GC population has survived to the present day compared to the early formation stages of their host galaxies (Brodie & Strader 2006). Another important parameter is the radial extent of the GC systems. In bright, massive galaxies, half of the total GC population typically extends out to three to five times the R_e of the host galaxy (Forbes 2017). However, GCs can also be found at much larger galactocentric distances, tracing galaxy formation and interactions (Lamers et al. 2017). A comprehensive discussion on the open questions regarding GC formation and evolution within a cosmological framework can be found in Forbes et al. (2018).

In this work, we use the deep, multi-passband, wide-field imaging of the NGC 5018 galaxy group collected with the VLT Survey Telescope (VST) as part of the VST Elliptical GAlaxy Survey (VEGAS; PI: E. Iodice, Capaccioli et al. 2015; Iodice et al. 2021) to conduct a systematic study of its GC system. The paper is organised as follows. Section 2 provides a literature review on the NGC 5018 group. In Sect. 3, we provide the details of the data used. Section 4 describes the data analysis procedure. The process adopted for GC selection is explained in Sect. 5. We present the results of our analysis in Sect. 6. A discussion of our results is provided in Sect. 7 and is summarised in Sect. 8.

2. NGC 5018 group from the literature

The NGC 5018 galaxy group is named after its brightest member, NGC 5018, which is a massive elliptical galaxy. The other bright galaxies in the group include the edge-on spiral NGC 5022, the face-on spiral NGC 5006, and two lenticulars MCG-03-34-013 and PGC 140148 (Kourkchi & Tully 2017). Table 1 lists the physical properties of these galaxies. For NGC 5018, we adopted a group distance modulus of 32.8 ± 0.1 mag (Tully et al. 2023), derived from the weighted average of all the available measures from the literature, based on Type Ia supernova, as well as the fundamental plane and Tully-Fisher relation. Using this value, we estimate the TOM of the GCLF in this galaxy at ∼26.7 mag, ∼25.3 mag and ∼24.7 mag in u-, g- and r-passbands, respectively (see more details on the TOM estimation in Sect. 5.1).

NGC 5018 displays several signs of tidal interactions such as shells, ripples, a complex system of dust lanes in the inner regions, a tail on the north-west side, and a bridge of HI gas towards its neighbouring galaxy NGC 5022 (Buson et al. 2004; Kim et al. 2012). The evidence of a perturbed inner structure indicates the occurrence of a minor merging event in the history of this galaxy (Spavone et al. 2018). The galaxy also shows some peculiar features such as a much weaker Mg₂ index compared to ellipticals of similar absolute magnitude (Schweizer et al. 1990), a low UV flux level (Rampazzo et al. 2007), and a metallicity lower than giant ellipticals (Bertola et al. 1993).

Furthermore, HI observations of NGC 5018 have revealed important evidence about the evolution of this galaxy. Kim et al. (1988) estimated the HI mass associated with NGC 5018 and NGC 5022 to be ∼4 × 10⁸ M_⊙ and ∼2 × 10⁹ M_⊙, respectively. Guhathakurta et al. (1990) made follow-up observations and found that the HI bridge actually connects NGC 5022 and MCG-03-34-013 passing NGC 5018 along the way. These authors estimated that the interaction that led to the observed plume in the north-west of NGC 5018 is as recently as ∼600 Myr ago. They suggested this stands as the first direct observational evidence for the formation of a shell system through the merger of an elliptical galaxy with a cold disk system.

X-ray observations have shed further light on the evolution of NGC 5018. Ghosh et al. (2005) studied this galaxy using the Chandra X-ray Observatory’s Advanced Imaging Charge-Coupled Device (CCD) Spectrometer. They detect diffuse hot gas that could be the remnant of interactions of NGC 5018 with its neighbouring galaxies. These authors suggest that although there is little current star formation in the galaxy, there is a large enough reservoir of gas to maintain a low, but steady level of star formation, which would explain the observed diffuse X-ray emission. Indeed, there seems to be evidence for the presence of a relatively young (∼3 Gyr) stellar population in the central regions of NGC 5018 (Bertola et al. 1993; Hilker & Kissler-Patig 1996; Leonardi & Worthey 2000; Buson et al. 2004).

The GC system of NGC 5018 has been studied previously. Hilker & Kissler-Patig (1996), using 6.4′×6.4′ area observations in the Bessel V- and Gunn i-passbands (each with an exposure time of ∼1 h) from the 1.54m Danish telescope at the European Southern Observatory (ESO) at La Silla in Chile, detected a poor GC system with S​_N = 1.10 ± 0.60 (a higher limit). These authors found that the GCs they detected in NGC 5018 can be divided into two sub-populations: a small population of young GCs (with age between several hundred Myr to 6 Gyr) and a second, larger population of older GCs. They suggest that the younger GCs must have formed during the last tidal interaction with one of the neighbouring galaxies and the older GCs could have formed in-situ representing the original population of the galaxy. Hilker & Kissler-Patig (1996) also reported the presence of a dust lane across NGC 5018, extending along south-east and north-west direction, which leads to a lack of blue GCs (V − I < 0.8). Humphrey (2009) studied GCs in NGC 5018 as a part of their sample of 19 early-type galaxies using HST Wide Field Planetary Camera 2 (WFPC2; exposure time of 13 and 30 minutes in the HST equivalent of V- and I-passbands, respectively). Similar to Hilker & Kissler-Patig (1996), they found a poor GC system and estimate S​_N = 0.46 ± 0.22 by inspecting an area of 2.5′×2.5′ around the core of the galaxy. No known works on GCs in the other four bright galaxies of the group exist in the literature.

Using the same dataset as the present work, Spavone et al. (2018) investigated the IGL in the NGC 5018 galaxy group, focusing on the region containing NGC 5018, NGC 5022 and MCG-03-34-013. They found that about 41% of the g-band luminosity of the group (L_g = 1.7 × 10¹¹ L_⊙) is attributed to the IGL. The estimated colour of the IGL component (g − r = 0.78 ± 0.35 mag) is consistent with the halo of NGC 5018, aligning with findings in similar groups (White et al. 2003; Da Rocha et al. 2008). Their analysis suggests that stellar stripping within the group is ongoing, with the total accreted mass fraction ranging from 78% to 92%. Additionally, they provided stellar mass estimates for the three galaxies involved (see Table 1). In addition to the three galaxies studied in detail by Spavone et al. (2018), in Table 1, we also report the stellar masses for NGC 5006 and PGC 140148. For this, we use the updated theoretical colour–M_*/L relations in the optical from Into & Portinari (2013). These authors provide the colour–M_*/L relations for exponential, disk, and dusty galaxy models for all the optical and NIR bands. We used the (B − K) colour and K-passband magnitude from the Hyperleda database (Makarov et al. 2014) to get the stellar mass estimate for NGC 5006 using the disk model. Due to a lack of a reliable K-passband magnitude for PGC 140148, we used the (B − I) colour and I-passband magnitude to get its stellar mass estimate using the disk model.

3. Observations and data reduction

The observational dataset used in this work is a part of the VEGAS survey (PI: E. Iodice; Capaccioli et al. 2015; Iodice et al. 2021)². It is a deep, multi-passband (u, g, r, i) imaging survey carried out using the VST telescope, an Istituto Nazionale di Astrofisica (INAF) facility³, located at the Paranal Observatory in Chile. This survey was approved as a Guaranteed Time Observation project for the period 2016-2021 with a total allocated observing time of ∼500 h. VST is an optical telescope with an aperture size of 2.6m and a field of view (FoV) of 1 sq. degree offered by the OmegaCAM camera that has a resolution of 0.21 arcsec/pixel (Kuijken 2011). A detailed description of the survey, the selected targets and the main scientific aims can be found in Capaccioli et al. (2015). Thanks to the wide FoV available and long exposure times, the VEGAS survey has acquired deep optical data, enabling us to:

1. Study galaxy outskirts out to 8−10 R_e and detect IGL and low-surface-brightness (LSB) features in the intra-cluster and intra-group space (Spavone et al. 2018; Iodice et al. 2021; Ragusa et al. 2022);

2. Estimate the mass assembly of galaxies by deriving the accreted mass fraction in the stellar halos (Spavone et al. 2017a,b, and references therein);

3. Trace the properties of GCs out to large galactocentric distances (up to 20 R_e or more) in different galaxy environments (Cantiello et al. 2015; D’Abrusco et al. 2016);

4. Build a census and provide a large catalogue of dwarf galaxies and ultra-diffuse galaxies (UDGs) in several clusters, such as Fornax (Venhola et al. 2018) and Hydra I (La Marca et al. 2022), as well as galaxy groups, such as NGC 3640 (Mirabile et al. 2024).

The u-, g-, and r-band data used in this work (noting that no i-band observations were taken for this field) were obtained in visitor mode and processed with the VST-Tube pipeline (Grado et al. 2012). The procedure is described in detail in Appendix A of Capaccioli et al. (2015). A normalisation of zero point to 30.0 mag was applied to the final co-added images. The calibration was done using a point spread function (PSF) magnitude within 4.0″ circular aperture using standard stars from the Sloan Digital Sky Survey (SDSS; Blanton et al. 2017). Figure 1 shows the ESO press release colour-composite image of the NGC 5018 group⁴. Table 2 lists the VST exposures used in this work. Our work is focused on the detection and characterisation of GCs in the 1.25 × 1.03 sq. degrees area centred on this galaxy group.

Fig. 1. Colour-composite image of the bright galaxies in the NGC 5018 group. Numerous foreground stars from our Milky Way and some background galaxies are visible in this image. The image size is 48′ × 60′; north is up and east is to the left. The inset (0.65′ × 0.65′) shows an LSB dwarf galaxy candidate (NGC 5018-LSB1) that we report on in this work (Credit: ESO/ Spavone et al. 2018).

For the NGC 5018 field, images were acquired using a step-dither observing strategy, involving a cycle of short exposures on both the target and an adjacent field taken close in time and space. This method allows for a larger field of view around galaxies with large angular sizes, although it requires more telescope time compared to standard dithering. The technique enhances sky background estimation around bright, extended galaxies. More details on the data reduction process are available in Spavone et al. (2017b).

4. Data analysis

4.1. Galaxy modelling and subtraction

Detecting faint sources in NGC 5018 is particularly challenging due to its peculiar morphology and the presence of dust in its inner regions. To enhance GC detection, a useful technique is to model and subtract the galaxy light profile from the image. For this work, we employed the strategy developed by Hazra et al. (2022), which is briefly outlined below.

To model the light distribution of the galaxy, we use the Python Elliptical Isophote Analysis package (Bradley et al. 2024)⁵. As a first step, we obtain a cutout frame of NGC 5018 from the original full frame. The size of the cutout is 8.4′ × 8.4′ extending out to ∼8 R_e on each side (R_e = 0.54′ for NGC 5018 in g-passband; see Table 1). Selecting a wide region allows us to make sure that we cover the entire light distribution of the galaxy to get a good model, while also avoiding getting too close to its neighbouring galaxies: NGC 5022 and MCG-03-34-013. To obtain a reliable galaxy model, we mask all the bright sources in and around NGC 5018 within the cutout frame. We then use the ellipse task based on the algorithm by Jedrzejewski (1987) to fit the isophotes taking as input our initial guesses for the position angle, ellipticity and the centre of the galaxy. In the fitting process, all geometric and photometric parameters are allowed to vary. Then, using the “build_ellipse_model” task from the Isophote package, we can create a model of the galaxy, which is subtracted from the cutout frame to obtain the residual image. This procedure (masking, modelling, and subtraction) is repeated until the residuals are satisfactory, that is, they are not dominated by modelling artefacts, but by potential merger signatures instead. Finally, we pad the residual frame to the original full frame. This residual full frame now contains our data with the light distribution of NGC 5018 subtracted for more efficient detection of faint, point-like sources.

Figure 2 illustrates the results of our procedure where we show the cutout frames of NGC 5018 (top panels) and the residuals obtained after model subtraction (bottom panels) for all the available passbands. Many faint sources now become visible in the residual frames. Some diffuse LSB features are also visible. These are mostly shells, ripples, and filaments, as discussed in Sect. 2.

Fig. 2. Cutout frames centred on NGC 5018 (top panels) and residuals obtained after model subtraction (bottom panels) in u- (left), g- (middle) and r- (right) bands. Each frame is of the size 8.4′ × 8.4′; north is up and east is left.

4.2. Source detection and photometry

To obtain the photometry of sources in our field, which is tailored to detect compact sources (stars, GCs, and unresolved galaxies), we used the photometry and source detection tool SExtractor (Bertin & Arnouts 1996). To optimise the detection of faint sources close to the bright galaxies, we adopted the following strategy:

1. Run SExtractor on the residual full frame with initial guesses for the input parameters, such as the full width at half maximum (FWHM), detection threshold, analysis threshold, using the weight maps from the VST-Tube pipeline;

2. Visually inspect the image generated by SExtractor where all the detected sources are subtracted from the residual full frame (‘– OBJECT’ frame). This is done to check if all the sources in the frame are detected, especially in the central regions of NGC 5018;

3. Check if the FWHM parameter we set initially is correct by comparing it with FWHM measured on the bright, compact and isolated sources after the initial SExtractor run;

4. Run SExtractor again on the residual full frame with the updated input parameters to improve the detection, adopting a more consistent FWHM.

Individual catalogues of sources in the observed field were obtained in the u-, g-, and r-passbands using the above strategy and were matched using a 1″ matching radius. The catalogue matched in all the three passbands (ugr-matched catalogue hereafter) contains ∼12 000 sources, whereas the catalogue matched in the g and r passbands (gr-matched catalogue, hereafter) contains ∼100 000 sources. The significant difference between the two matched catalogues is due to the highly incomplete u-band photometry, which will be discussed in later sections.

The following colour corrections, provided by the VST-Tube pipeline (Grado et al. 2012), were applied to the magnitudes obtained from the SExtractor run:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill m_{u_{\mathrm{corr}}}& =m_u+0.0255\times (m_u-m_g),\hfill \\ \hfill m_{g_{\mathrm{corr}}}& =m_g+0.0288\times [1.3336\times (m_g-m_r)+0.0444],\hfill \\ \hfill m_{r_{\mathrm{corr}}}& =m_r+0.0406\times [0.3853\times (m_g-m_r)+0.0137].\hfill \\ \hfill \end{array}\end{array}$$(2)

As in other works on GCs using the VEGAS survey data (e.g. D’Abrusco et al. 2016; Cantiello et al. 2018a, 2020), we used the magnitude within an 8 pixel circular aperture as our reference magnitude. This means that the flux of a source is measured within this finite aperture and, thus, we then need to correct for the flux outside the 8 pixel radius. To derive this aperture correction (a.c.), we measured the difference in magnitude at our selected aperture size and a larger aperture size of 19.05 pixels. This larger aperture size was selected based on the fact that the data is reduced by the VST-Tube pipeline, which uses a calibration radius of 19.05 pixels (i.e. 4″, Grado et al. 2012). The a.c. values for the three passbands are listed in Table 2. To obtain these values, we select only the bright and isolated stars from the individual photometric catalogues from SExtractor.

To verify the quality of our photometry, we compared it with the AAVSO Photometric All-Sky Survey (APASS). First, we obtained a catalogue of sources in the observed field from APASS using the VizieR catalogue query tool⁶. As APASS lacks u-passband, we adopted the strategy described in Cantiello et al. (2020). We transformed the B-passband magnitude from APASS to u-passband using the transformation equations available at the SDSS web pages⁷. In particular, we used the following equation:

$$\begin{array}{c}\hfill u=B_{\mathrm{APASS}}+0.8116\times {(u-g)}_{\mathrm{fit}}-0.1313,\end{array}$$(3)

where (u − g)_fit colour index is derived from the APASS (g − i) and (g − r) colour indices using a second-degree polynomial fit derived from the SDSS data over the M87 region. We selected M87 in particular because it is situated at a high galactic latitude and, thus, it has very small extinction. Therefore, by using the u-passband magnitude of sources in APASS derived as a function of the B, g, r and i photometry, we were able to proceed with the photometric comparison. For this, we selected only the bright and isolated stars from the ugr and APASS matched catalogue (matched using a 1″ radius). For this work, we adopted the AB mag photometric system.

Figure 3 shows the comparison between VST and APASS photometry in all three passbands. In Table 2, we report the median (green horizontal line in Fig. 3) and the RMS (green shaded region; derived from the median absolute deviation RMS_MAD) values of Δ mag (=m_VST − m_APASS). The small Δ mag values in all the passbands indicate that the VST photometry is in good agreement with that of APASS. The larger scatter in the r-band (right panel in Fig. 3) is due to its slightly worse image quality compared to the g-band which is visible in Table 2 (FWHM and a.c.) and in Figs. 4 and 6 (e.g. from the concentration index, CI, panel). The outliers in the right panel in Fig. 3 are mostly due to poor matching related to this issue of lower image quality. Finally, to correct for galactic extinction, we obtain the reddening (E_B − V) values from the IRSA dust query module of Astropy⁸ at the position of sources in the observed field. These E_B − V values are then multiplied by extinction factors of 4.239, 3.303 and 2.285 for u, g, and r passbands, respectively, from Schlafly & Finkbeiner (2011) to obtain the extinction correction. The E_B − V variation across the field is of the order of ∼0.03 mag.

Fig. 3. Comparison between VST and APASS magnitudes (Δ mag = mVST − mAPASS) in u- (left), g- (middle), and r- (right) passbands. Blue dots represent all the sources in ugr-matched catalogue which is matched with APASS catalogue. Red dots represent the bright, unsaturated and point-like sources that we select for photometric comparison. Median and RMSMAD values of Δ mag in the three passbands are shown by green horizontal line and shaded regions, respectively and the values are reported in Table 2.

4.3. Completeness

A further ingredient that will be useful in our forthcoming analysis is the completeness function in the observed field. This function describes how complete our photometric catalogue is in a given passband for point sources as a function of magnitude. The function is derived from simulations by injecting artificial sources into a given frame and then detecting them using the same strategy described in the previous section. The fraction of the number of detected sources over the number of injected sources (f = N_d/N_i) as a function of magnitude provides the completeness function. This fraction is close to one for bright sources and as we move to fainter magnitudes, an increasing number of sources are undetected, so the fraction approaches zero. The procedure we adopt to estimate the completeness function is described in more details in Mirabile et al. (2024). Here, we briefly summarise the main steps.

In a case such as this, where we have a large area format observations, it is ideal to characterise the homogeneity in terms of depth and image quality over the observed area. To study the stability of the completeness function with our data, we selected nine regions (shown in the top left panel in Fig. 4) in the field, including one on NGC 5018. The region on NGC 5018 is of the same size as the cutout used in Sect. 4.1 (8.4′ × 8.4′). For consistency, all the eight off-galaxy regions are the same size.

Fig. 4. Estimated completeness over the field. Left panel: r-passband image highlighting the regions selected for completeness analysis in the observed field. The area of the entire frame inspected for GCs is 1.25 × 1.03 sq. degrees; north is up, and east is left. The region highlighted in red is centred on NGC 5018. The eight off-galaxy regions surrounding the galaxy are highlighted with green, blue, magenta, cyan, olive, yellow, brown, and purple boxes. Each region is 8.4′ × 8.4′ in size. Middle and right panels: Completeness functions simulated and fitted using the modified Fermi function (Eq. 4) for the u- (middle top), g- (right), and r- passbands (middle bottom). The solid curves are colour-coded according to the boxed regions in the left panel. The vertical solid black line marks the expected TOM of the GCLF in the respective passbands. The grey-shaded region indicates the adopted magnitude range for GC selection in the three passbands (see Sect. 5.1). The dashed black curves represent the expected GCLF in the respective passbands (see Sect. 4.3 for details).

To run the completeness tests, we selected the bright, compact and isolated sources in the selected regions to generate models of PSF using the EPSFBuilder⁹ routine of the Photutils package. Then we simulated the magnitude of sources that are to be artificially injected in the image by generating a random sample of magnitudes using numpy.random.choice¹⁰ in the shape of the luminosity function of all the real sources detected in those regions. We injected ∼1700 artificial sources in the regions, using the PSF model obtained from the corresponding region, along an equispaced grid whose position is varied over 75 iterations. This means that during each iteration, the magnitude and the position of all the injected artificial sources in the regions are randomly changed, and they do not increase local density of sources.

As a test of the reliability of the PSF models, we compared the a.c. values obtained from the artificially injected sources with those derived from the real sources and we found a good agreement between the two, within 0.01 mag. Adopting the same input parameters of SExtractor that were used for the detection of real sources in the residual full frame, we now obtain a catalogue of sources detected on each of the simulated frames containing artificially injected sources. We matched this catalogue (using a 1″ matching radius) to the catalogue of injected sources and cleaned out any spurious sources. For this step, we inspected the broadening of difference between injected and detected magnitude versus the detected magnitude for each source. We used an iterative sigma clipping approach that is magnitude dependent. Finally, the ratio of the number of sources retrieved versus injected (f) in each magnitude bin gives us the completeness fraction of the data in all the three passbands.

We fit the data points (not shown in Fig. 4 to avoid their over-plotting on the curves) using the following modified Fermi function (Alamo-Martínez et al. 2013):

$$\begin{array}{c}\hfill f(m)=\frac{1+C·exp[b(m-m_{50})]}{1+exp[a(m-m_{50})]},\end{array}$$(4)

where m₅₀ is the 50% magnitude completeness limit, ‘a’ is a parameter that regulates the steepness of the cut-off, parameter ‘b’ (which must be < a) influences the point at which the deviation above unity starts, and parameter ‘C’ determines the amplitude of the deviation. The 80% completeness limits in the three passbands for the frame centred on NGC 5018 are reported in Table 2.

Figure 4 shows the completeness functions (coloured curves) for the selected regions in all three passbands, along with the expected GCLFs (dashed black curves; see Sect. 5.1). Each curve assumes a reference S​_N = 1 (Hilker & Kissler-Patig 1996) and our estimate of M_V = −22.28 ± 0.15 mag for NGC 5018 (see Sect. 6.4), used in Eq. (1) to estimate the total number of expected GCs. The completeness functions in the u-passband were derived only for the five regions nearest the galaxy. This is because given the completeness tests and the bright magnitude limit of the u-data (Sect. 4.2), our results did not rely on catalogues based on u-photometry, so no u-band completeness correction was required.

By inspecting the completeness functions shown in Fig. 4, we made the following observations:

1. In the case of u-passband (middle top panel), we did not reach the estimated TOM of the GCLF (26.7 mag; Sect. 5.1 explains the estimation procedure of TOM in all the three passbands);

2. We achieved a result of ∼50% completeness at the TOM in the g- (25.3 mag) and r- (24.7 mag) passbands in the region centred on NGC 5018 (red curves in the right and middle bottom panels);

3. Relative to the respective TOM, the r- and the g-passband have very similar completeness, while in absolute terms, the completeness drops to zero at ∼25.8 mag in r-, compared to ∼26.2 mag in the g-band;

4. The completeness varies by a larger fraction in the r-passband compared to the g-passband. As an example, around the TOM, the value of f from the different completeness experiments has an rms of 2.5% in g-passband and 4.7% in r-passband. This is also expected, as the g image has slightly better image quality parameters compared to the r image, as evidenced by the larger scatter in the comparison with APASS photometry (Fig. 3) and the quality parameters reported in Table 2. This behaviour will motivate our choice of selection of g-passband to characterise the GCLF (see Sect. 6.4).

Finally, we note that because of the peculiar observing strategy adopted in VEGAS (Sect. 3), the completeness drops towards the image edges (see also Mirabile et al. 2024).

5. GC selection

The identification of GCs in distant galaxies is challenging, as it suffers from contamination from foreground Milky Way stars and background galaxies. Following the same approach successfully adopted in the other works on GCs using VEGAS data (e.g. Cantiello et al. 2015, 2018b, 2020; D’Abrusco et al. 2022; Ragusa et al. 2022), we used a set of photometric (magnitude and colour) and morphometric (shape and compactness) properties to identify GCs. These are described below and summarised in Tables 3 and 4.

5.1. Photometric properties

To clean our catalogue from non-GC sources, we first applied a magnitude cut taking advantage of the expected magnitude range of GCs in NGC 5018. The GCLF exhibits a universal profile in the form of a Gaussian (Harris 2001) and has a peak at an absolute magnitude of M_g^TOM = −7.5 ± 0.2 mag (Villegas et al. 2010). It has a width σ^GCLF which scales with the luminosity of its host galaxy (Harris 2001; Villegas et al. 2010).

Using Eq. (4) from Villegas et al. (2010) and adopting a total galaxy magnitude M_z = −23.2 ± 0.1 mag for NGC 5018, we estimated σ_g^GCLF = 1.2 ± 0.2 mag for the GCLF of NGC 5018. At our adopted distance modulus for this galaxy (Table 1), we estimated the TOM of its GCLF at m_g^TOM = 25.3 ± 0.3 mag. Given the proximity of the g-passband to the r-passband, we assumed σ_r^GCLF = σ_g^GCLF. Then we took advantage of the GC median colour (g − r) = 0.6 mag obtained from spectroscopically confirmed GCs in the Fornax cluster using the Fornax Deep Survey data from VST (Cantiello et al. 2020, FDS data, hereafter) to obtain the TOM in the r-passband (m_r^TOM = 24.7 ± 0.3 mag). Finally, for the selection of GCs in the g-passband, we used the range from −3σ_g^GCLF for the brighter part to +1σ_g^GCLF for the fainter part around the TOM¹¹. We used a similar magnitude range from −3σ_r^GCLF to +1σ_r^GCLF in the r-passband. Figure 4 (middle and right panels) visually motivates our choice for the faint magnitude limit of +1σ^GCLF in both g- and r-passbands as the completeness fractions reach zero at these magnitude levels.

For the u-passband, adopting the GC median colour (u − r) = 2.0 mag from the FDS data, we estimated m_u^TOM = 26.7 ± 0.3 mag and assume σ_u^GCLF = 1.2 ± 0.2 mag, due to its proximity to the g passband. As discussed in Sect. 4.2, from Fig. 4 (middle top panel) it is evident that the u passband lacks the depth to reach the estimated TOM; hence, we chose to use the gr-matched catalogue as our reference and present those results in the next section. The ugr catalogue was still analysed, but primarily as a sanity check for the results obtained with the two-band catalogue. Assuming a common distance modulus for the galaxies in the group, we also expected to observe the GCs of four other bright galaxies within these magnitude ranges across the three passbands.

Using colour for GC selection helps reduce contamination from fore- and background compact sources. To define a colour range in the gr-matched catalogue, we used the range 0.3 ≤ (g − r)≤1.1 mag from the FDS data. In the case of the ugr-matched catalogue, we performed a more refined selection in the (u − r) versus (g − r) plane (see Fig. 5). Adopting the FDS data, we derived the isodensity contours and overlay them on the (u − r) versus (g − r) colour-colour plot of GC candidates, pre-selected based on the morphometric parameters from our ugr-matched catalogue. The isodensity contours represent fractions of the total GC population in the Fornax cluster, increasing outwards. In the figure, the red side of the plot (u − r > 2.5 mag) is nearly empty due to the shallow u-passband data. The over-dense regions visible in Fig. 5 are sequences of foreground stars and background galaxies, that is, contaminants that passed our preliminary selection criteria in the ugr-matched catalogue. Selecting a narrow contour implies lower contamination, but also lower completeness, while a wide contour increases both contamination and completeness. We chose the 80% contour level as optimal for GC selection (bold in the figure) and selected all sources within this contour as our final GC candidates from the ugr-matched catalogue. Although this choice allows some contaminants to enter (as seen in Fig. 5), these are effectively addressed by the background decontamination technique described in Sect. 6.2.

Fig. 5. 2D histogram of pre-selected GC candidates (based on morphometric and magnitude parameters only) in the (u − r) vs (g − r) plane, showing sources from the ugr-matched catalogue. The over-dense regions (red and pink) represent sequences of contaminants, mostly foreground stars and background galaxies. The isodensity contours, computed using spectroscopically confirmed GCs in the Fornax cluster from FDS data (Cantiello et al. 2020), are over-plotted. The contours range from 10% (innermost) to 90% (outermost) in steps of 10%. Only the sources within the 80% contour (highlighted with a thick black contour) are selected as GC candidates from the ugr-matched catalogue.

5.2. Morphometric properties

Assuming a mean R_e of 3 pc for GCs (Sect. 1) and using our adopted distance of 36 Mpc for NGC 5018 group (Table 1), we estimated the angular size of typical GCs in this group to be ∼0.02 arcsec. Therefore, the GCs at the distance of the NGC 5018 group are unresolved with our data. However, as has been shown in previous GC works using VEGAS (e.g. D’Abrusco et al. 2016; Cantiello et al. 2018a, 2020), we can still take advantage of morphometric properties to identify GCs through their shape.

A first selection was made using the concentration index (CI) parameter which is defined as the difference between magnitudes at two aperture sizes. The CI is a good indicator of compactness of the source (Peng et al. 2011). For this measurement, we adopted apertures with 4 and 8 pixel radii. These values were chosen as a good compromise between CI flatness and small rms after testing different radii. We select as GC candidates the sources with CI in the range 0.5 to 1.6 mag in both g- and r-passbands from the gr-matched catalogue and, additionally, between 0.5 and 2.0 mag in the u-passband from the ugr-matched catalogue. A slightly wider range was adopted in the u passband because of its worse image quality compared to g and r passbands (see Table 2).

Further cleaning of the sample was done using the ‘ELONGATION’ output parameter from SExtractor (elongation hereafter), which describes the shape of the detected source in terms of the ratio between its major and minor axes. We selected all the sources with elongation ≤3 in both the gr- and ugr-matched catalogues. The limit we applied is larger than what is typically adopted for GCs in the previous papers focussing on the analysis of small stellar systems (SSS) using VEGAS data (VEGAS-SSS; e.g. D’Abrusco et al. 2016; Cantiello et al. 2015, 2018b, 2020). We adopted such a wide cut mainly because, when matching our GC candidates in NGC 5018 with the catalogue of Humphrey (2009, mainly composed of candidates close to the galaxy centre), we found that some of their GCs had large elongation. This is due to the highly variable background in the innermost galaxy regions, caused by the combined effects of dust and the steep surface brightness gradient and the possibility of blending of multiple sources in the ground-based images.

However, we verified that using an elongation cut of ≤2.0 in both passbands does not change the results at large galactocentric distances (see right panel in Fig. A.3), and the results remain consistent with the general trends we observe, albeit with some offset due to the impact on background count levels when using different GC selection criteria. Applying such a lower cut on elongation came at the expense of losing many sources in the core region of NGC 5018. Since we wanted to include information from the internal galaxy regions, we retained the larger cut on elongation. Moreover, we highlight that, with all the other cuts made, the number of GCs with elongation between 2 and 3 is ∼10% of the selected sample, or less (see Fig. 6, middle lower panel). Finally, assuming, as we always do, that the background contamination is constant, even with such a large cut, the interlopers are properly accounted for with the background characterisation and decontamination procedure we adopt (see Sect. 6.2).

Fig. 6. Morphometric parameters used for the selection of GC candidates: CI, elongation, and (g − r) colour are shown in the top left/middle panels, bottom left and middle panels, and top right panel, respectively. Grey dots represent all sources in the gr-matched catalogue, while red dots indicate GC candidates within Re, GC of NGC 5018 (see Sect. 5.2). The dashed vertical and horizontal black lines indicate the adopted range for GC selection (see Table 3). The black error bars in all the panels denote the median error on magnitude (multiplied by a factor of 7 for visibility purpose) with a bin size of 2.0 mag, and on (g − r) colour (top right panel) within the given magnitude bin size.

Figure 6 shows the plots for the parameters that we adopt for GC selection in the case of gr-matched catalogue. The GC candidates within 5 R_e of NGC 5018 are shown (red dots) along with all the sources (grey dots) in this catalogue. The choice of 5 R_e comes from the observation that half the total GC population is observed to lie within ∼5 R_e of the host galaxy in the case of bright, massive ellipticals (Forbes et al. 2018). In the following sections we will identify this 5 R_e radius of NGC 5018 as its GC half-number radius: R_e, GC. Tables 3 and 4 list the criteria we adopted for the GC selection in the gr- and ugr-matched catalogues, respectively.

SExtractor has other output parameters such as ‘FWHM_IMAGE’, ‘FLUX_RADIUS’, and ‘CLASS_STAR’ that describe the morphometry of detected sources. These parameters can be exploited for selection of GCs (e.g. Cantiello et al. 2018a, 2020; Hazra et al. 2022). However, after testing the GC candidates selected using these parameters, we opted not to use them as no improvement over our existing technique was found.

Using the selection criteria described in Table 3, we obtained a catalogue of GC candidates from the gr-matched catalogue. In the next section, we describe the analysis of the various properties of the GC system based on its inspection. Table B.1 contains an extract of the final list of GC candidates with the position of the sources and their measured photometric and morphometric properties, including the u-passband data (when available).

In conclusion, since the u passband is shallower compared to the other passbands, by adopting the ugr sample, we expect to be potentially dominated by the fraction of blue GCs and, therefore, possibly contaminated by young massive clusters, as well as blue Milky Way stars and background compact galaxies. With the gr-matched catalogue, on the other hand, combined with our adopted criteria for GC selection, we expect to observe a dominant population of older GCs, although there may still be significant residual contamination (which depends on the distance from the bright galaxies). Consequently, we might miss intermediate- and young-age GCs that could exist in the NGC 5018 group (Sect. 2).

6. Results

As described in previous works based on VEGAS survey data (e.g. Iodice et al. 2016; Cantiello et al. 2018a; La Marca et al. 2022), one of the substantial advantages of the VEGAS images is the availability of a large area, which is useful for constraining the background properties in the field around the bright galaxies. This provides valuable information for studying the over-density of sources on the galaxies in our system which, under the selection criteria we adopt, are primarily the GCs hosted in the galaxy group. In the following sections, we examine the observational properties of the GC system in the NGC 5018 group, using the GC candidates from Table B.1, derived from the gr-matched catalogue as our reference.

6.1. 2D distribution of GCs

To generate the 2D distribution maps, we employed a kernel density estimator (KDE) from the Seaborn package (Waskom 2021)¹². The plots created with the KDE tool visualise the 2D distribution of the GC candidates using a continuous probability density curve. The bandwidth of this curve is determined by two parameters: ‘bw_method’, which controls the smoothing using a Gaussian kernel, and ‘bw_adjust’, a factor that multiplicatively scales the value of ‘bw_method’. Increasing the ‘bw_adjust’ parameter will make the density curve smoother.

Figure 7 displays the 2D distribution map for the GC candidates. We observe an over-density of sources centred on NGC 5018, while the other four bright galaxies in the group, particularly the two lenticulars, do not seem to exhibit significant over-densities of GC candidates. The plot reveals a radial trend, with higher density near the centre of the observed field and the group, decreasing towards the outskirts. The over-densities are spread in an elliptical shape extending diagonally along the direction of the five galaxies, including NGC 5018, and around the group itself. Its centre (RA = 198.290 deg; Dec = −19.385 deg) is located ∼0.13 deg north of NGC 5018.

Fig. 7. 2D distribution map of GC candidates in the observed field. Left panel: 2D map for the gr-matched catalogue obtained using bw_adjust = 0.14 and bw_method parameter set to Scott (see Sect. 6.1 for the definitions). The magenta and blue coloured areas represent an over-density of sources whereas the red and yellow-green coloured areas show the under-dense regions. The red coloured areas, mostly situated around the edges of the frame, are under-dense due to the low S/N because of the observing strategy. The position of the galaxies are indicated with crosses of colour yellow (NGC 5018), magenta (NGC 5022), cyan (MCG-03-34-013), orange (NGC 5006), light pink (PGC 140148). The white cross is the LSB dwarf galaxy candidate (NGC 5018-LSB1) reported in Sect. 6.3. Right panel: As in left panel, but using bw_adjust = 0.28. The black contours reported in both the panels indicate the regions higher than 5σ level from the background rms, obtained using bw_adjust = 0.28.

To determine whether this structure is an artefact of our GC selection procedure or a genuine feature of the GCs in the field, we perform several tests, detailed in Appendix A. These tests include examining the 2D distributions of: (i) the GC catalogue obtained with narrower constraints compared to the reference selection criteria in Table 3, (ii) two catalogues, one of very bright, saturated point sources representing foreground Milky Way stars and another of extended sources with non-GC colours resembling background galaxies, and (iii) two GC catalogues, one with a magnitude cut up to the TOM and another with a narrower cut on elongation (as discussed in Sect. 5.2). The results of such tests confirm that the elliptical-shaped over-density is preserved in all cases; hence, it is likely to be a real feature in the observed field, indicating the presence of an intra-group GC population (analysed in the sections below). To further support the reality of this feature, we anticipate that it shows a significant overlap with the IGL region studied by Spavone et al. (2018), but extends to larger group-centric radii. The black contours in Fig. 7 represent the 5σ level higher than the background, where the σ is assumed to be the mean of the two rms value of background densities described in the following Sect. 6.2 (σ = 0.175 GC/arcmin²).

A notable feature observed in our tests is a non-negligible over-density of GC candidates north-west of NGC 5018 around RA and Dec = [198.1 deg; −19.0 deg]. This region does not appear to contain any bright galaxies. However, it might represent an extension of the intra-group GC population resulting from interactions among group members or a region with higher foreground or background contamination. In the forthcoming discussion, we will focus on the central GC over-density within the described elliptical region. Further studies on the detailed sub-structures in the GC population, including the north-west extension, will require deeper multi-passband observations, as expected from surveys like Legacy Survey of Space and Time (LSST) at the Vera C. Rubin Observatory.

One additional test we present involves studying the 2D map by further smoothing the GC candidate density, shown in the right panel of Fig. 7. As expected, the small-scale structure of over- and under-density changes due to the higher smoothing factor (bw_adjust = 0.28), causing the edges of some substructures to merge. Despite this, the overall appearance of the map, including the excess of candidates on NGC 5018, the wider elliptical over-dense area, the north-west extension, and the asymmetry towards a local dwarf galaxy (see below), is confirmed. Hence, for the remainder of our analysis, we focussed on group-scale trends rather than small-scale over-densities. The only small-scale exception we highlight is a plume-like over-dense region north-east of NGC 5018. This feature is also preserved in the tests on 2D distribution maps (Figs. A.1 and A.3). Although it does not appear to be connected to any distribution of diffuse light in the area, this plume is oriented towards what appears to be a local nucleated LSB dwarf galaxy of the group whose brightness profile stretches along the direction of NGC 5018. Small cutouts centred on this LSB dwarf galaxy (NGC 5018 – LSB1 hereafter) are shown in top left corner in Fig. 1 (a zoomed-in colour image) and Fig. 8 (r-passband image). NGC 5018 – LSB1 is further discussed in details in Sect. 6.3.

Fig. 8. Cutout from the r-passband data of the LSB dwarf galaxy candidate (NGC 5018-LSB1) that we report in Sect. 6.1. The image size is 1.2′ × 1.2′; north is up and east is left. A zoomed-in colour version of this image is shown in the top left corner of Fig. 1.

We also analysed the 2D distribution map of GC candidates from the ugr-matched catalogue. Due to the shallow depth of the u-passband data (see middle top panel in Fig. 4), only the brightest and bluest GCs in the population were detected. The 50% completeness level in the u-passband is 23.4 mag, ∼3 mag brighter than the expected TOM in this band. Consequently, no significant over-density of GC candidates around NGC 5018 and the four other bright galaxies is observed in the ugr-matched catalogue. Furthermore, the few detected GCs are likely to be metal-poor, which in most galaxies are distributed at larger galactocentric distances than the red, metal-rich ones. This broader distribution dilutes their density over a larger area, making them more likely to be lost in statistical fluctuations of the background interlopers.

6.2. Radial and colour distribution of GCs

To study the radial density and (g − r) colour distribution of the GC candidates, we took advantage of the large format of the VST imaging data to apply a background decontamination technique. This approach allows us to inspect the properties of the GC candidates in the observed field. We adopted a method previously used in similar works on GCs using VEGAS survey data (e.g. Cantiello et al. 2015, 2020; D’Abrusco et al. 2016; Mirabile et al. 2024), which relies on the assumption that the fore- and background contaminants in the GC candidates catalogue are evenly distributed across the field. While there is cosmic variance among background galaxies, our approach of sampling different background areas helps smooth this variance. By studying sources in the background fields, far from the target galaxies, we can constrain the properties of the contaminants. These constraints were then used to correct the GC properties through on-galaxy to off-galaxy density subtraction. We did not rely on GC counts derived from areas at larger galactocentric distances due to the incompleteness of the catalogue at those radii (see Sects. 3 and 4.3, and Mirabile et al. 2024).

A key ingredient of this analysis is the identification of suitable background regions to be used for decontamination. Considering the small variations in depth and image quality over the field in g- and r-passbands (see discussion in Sect. 4.3), the observed geometry of the GC over-density region (highlighted in Fig. 10) and taking advantage of the large observed area, we decided to use two different regions as background:

1. A circular annulus centred on the observed field with radii between 26′ and 29′ (annular background hereafter; large black annulus in Fig. 10) with a density of 11.34 ± 0.15 GC/arcmin².

2. Ten random regions each with 2.7′ radius, considering their mean (10.60 ± 0.20 GC/arcmin²) as a background (random background hereafter; small black circles in Fig. 10).

The normalised 2D histograms of the colour-magnitude diagram for GC candidates in NGC 5018 and the two background regions are shown in Fig. 9. The plot includes all pre-selected GC candidates within R_e, GC = 2.7′ of NGC 5018 (left panel), in the random background (middle panel), and in the annular background (right panel), normalised to a peak density of one. The GC candidates in the figure were selected based on the criteria reported in Table 3, except for the (g − r) colour selection. The GC candidate over-density in NGC 5018 is visible in the left panel as a darker sequence (i.e. higher number counts) with (g − r) between ∼0.4 and ∼1.0 mag and m_g between ∼22.5 and ∼24.5 mag. The middle and right panels show the selected GC candidates from the control background regions, revealing a less prominent density at the expected GC colours and magnitudes.

Fig. 9. Normalised 2D histogram of colour-magnitude diagram of GC candidates selected using criteria reported in Table 3, except for (g − r) colour selection, around NGC 5018 (within Re, GC; left panel), random background (middle panel) and annular background (right panel). The (g − r) colour selection interval, and the bright magnitude cut are also shown in the panels with dashed black lines.

To estimate the uncertainty in the GC density for the background, we used Poissonian statistics for the number of counts in the annular background, and the rms between the GC counts in the selected circular areas for the random background, in both cases normalising to the area to obtain a density. Unlike in previous VEGAS-SSS works, we retain both background estimates in our analysis, as they are only broadly consistent with each other. Furthermore, we do not select any region close to the edges of the frame because the S/N is low there due to the VEGAS survey observing strategy (see Sects. 3–4.3) and this affected our source detection. For selecting GC candidates within the elliptical over-density region, we consider an ellipse with major and minor axes of length 50.4′ and 25.2′, respectively (approximately 0.50 Mpc and 0.25 Mpc; see Fig. 10). This size was selected to ensure that the general outline of the feature observed in Fig. 7 is encompassed, including the galaxies in the group. For the radial density and (g − r) profiles of the GC candidates in this region, we performed multiple tests with varying size and the position angle of the ellipse and found consistent results with negligible differences.

Fig. 10. Same as Fig. 7, but with the different regions selected for background decontamination highlighted. The circles centred on the five bright galaxies represent their 5 Re radius (Re, GC), which corresponds to 2.7′ in the case of NGC 5018. The 10 small black circles, each with a radius of 2.7′, are the regions selected to collectively represent the random background. The large black annulus (with inner and outer radii of 26′ and 29′, respectively) represents the annular background. The red dashed ellipse indicates the elliptical over-density region, with major and minor axes of 50.4′ and 25.2′, respectively. See text for details.

6.2.1. Radial density profile

To inspect the radial density profile of GC candidates on NGC 5018, we derived the azimuthally averaged density profile obtained using concentric circular annuli centred on the galaxy. From the profile, we subtracted the background level estimated in the two background regions.

Figure 11 shows our result where the profiles displays the expected larger GC density around NGC 5018. The observed low density in the core (galactocentric radius, R_gal ≤ 1.2′) of the galaxy is expected because of the poor detection efficiency in this region and also due to the presence of dust (see Figs. 1 and 2) which affects the observability of sources, especially at the faint magnitude levels.

Fig. 11. Radial density profile of GC candidates in NGC 5018 after subtracting random background (green points) and annular background (red points). The blue points represent the light profile of the galaxy in the g-passband from Spavone et al. (2018) after an arbitrary vertical shift. The vertical black dashed line represents Re, GC radius.

From the observed peak, a general decreasing trend is observed out to R_gal ∼ 3.6′ which also marks the transition radius between the two accreted components observed from the fit of the surface brightness profile of NGC 5018 (Spavone et al. 2018, Fig. 13). Additionally, a bump is observed at R_gal > 3.6′ which corresponds approximately to the plume over-density region of GC candidates that we observe to the north-east of the galaxy extending towards NGC 5018 – LSB1. The g-passband surface brightness profile of NGC 5018 is plotted in Fig. 11 (with an arbitrary shift) from Spavone et al. (2018). We observe that both the background corrected radial density profiles follow the galaxy light profile in the interval between 1.2′≤R_gal ≤ 3.6′.

Figure 12 shows the background subtracted density profiles within the elliptical over-density region. The radial distributions are obtained by selecting GC candidates within concentric ellipses (with fixed ellipticity) of increasing semi-major axis reaching 28′. The profiles appear to have a relatively complex shape. We observe the radial density as relatively flat out to the ellipse radius, R_ellipse ∼ 20′, with a small shift of 0.74 GC/arcmin² between the two different background profiles.

Fig. 12. Density profile of GC candidates in the elliptical over-density region obtained by subtracting annular background (left panel) and random background (right panel). The grey shaded region in both the panels beyond 26′ represents the annular background region. The red and green dotted horizontal lines and shaded regions in the left and right panels, respectively, represent the mean and RMSMAD density values within elliptical radius Rellipse < 25′. The yellow shaded regions in both the panels represent the region encompassing NGC 5018 and other GC over-densities surrounding the group (see Sect. 6.1).

The grey shaded region in Fig. 12 represents the annular background region, where the radial profiles show a sharp decrease. The observed distribution peaks between R_ellipse ∼ 10′ and R_ellipse ∼ 22′ (highlighted in yellow in Fig. 12), corresponding to the region where NGC 5018 and other observed GC over-densities are located. Moreover, the innermost regions of the distribution (R_ellipse < 10′) have a density lower than the average between R_ellipse = 10′ and 22′. The median and the RMS_MAD for R_ellipse < 25′ is 2.38 ± 0.57 GC/arcmin² for annular background subtracted profile and 3.12 ± 0.57 GC/arcmin² for random background subtracted profile. These values are shown in Fig. 12 by red and green dotted horizontal lines and shaded regions in the left and right panels, respectively. Although this inner decrease is consistent with the estimated uncertainties, if real, it could potentially be explained by the fact that the elliptical shape we have assumed is a simplification of the actual geometry of the GC candidates in the observed intra-group population.

We observe only a small GC over-density around the other four bright galaxies in the group, which is expected since these galaxies are less massive compared to NGC 5018 and host a smaller GC population, particularly the lenticulars (MCG-03-34-013 and PGC 140148). Moreover, the two brighter members of the group, NGC 5022 and NGC 5006, are spiral galaxies, making their GC populations harder to identify due to confusion from dust, star-forming regions, and disk structures. However, for both spirals, we observe regions of GC over-densities at large galactocentric distances, possibly part of their extended GC systems transitioning into the intra-group GC population. For comparison, the Milky Way GCs AM 1 and PAL 4, located ∼125 kpc from the Milky Way centre, at the distance of the NGC 5018 group would correspond to a projected distance of ∼12′, aligning with the over-dense regions around these spirals.

Figure 13 shows the background-subtracted radial density profile for NGC 5006. Within 5 R_e (vertical dashed black line), we observe a small GC candidates over-density. Beyond this, the GC population decreases, possibly transitioning into the intra-group component. For NGC 5022 (magenta cross in Fig. 7), we do not observe a local over-density, but it is worth noting that it shows the strongest evidence of interaction with NGC 5018 (see Fig. 2 and Spavone et al. 2018), and lies between two GC over-dense regions.

Fig. 13. Radial density profile of GC candidates on NGC 5006 after subtracting random background (green points) and annular background (red points). The vertical dashed black line represents 5 Re radius.

6.2.2. Colour distribution

As discussed in Sect. 1, the colour distribution of GC populations can provide insights into key properties of the host, in particular, by allowing us to trace their metallicity distribution. Differences in integrated colours are a good proxy for differences in metallicities of old GCs, and this can help us study the evolutionary history of the host galaxies.

We again take advantage of the large format of the VST images to apply the background decontamination technique for inspecting the (g − r) colour distribution of GC candidates. Figure 14 (left panel) shows the colour distribution of GC candidates density within R_e, GC of NGC 5018. The red and green dashed histograms represent annular and random background subtracted colour density distributions, respectively, derived after normalizing to their respective areas. They reveal the presence of a relatively blue GC population with a peak at (g − r)∼0.75 mag, along with a tail of red GC population with (g − r) > 0.95 mag. The colour distribution of GCs in NGC 1399 (within an 8′ radius of the galaxy centre) from the FDS data is also shown as a grey hashed histogram. We observe that the GCs in NGC 1399 have a peak at a very similar (g − r) colour. We further examined the spatial distribution of the GC population with (g − r) > 0.95 mag around NGC 5018 and found that a majority of these very red GC candidates lie on dust patches in its core region (R_gal ≤ 1.2′). Their presence in dusty regions makes them appear redder than their actual colour. Hence, this red peak should be attributed to the presence of GCs reddened by dust.

Fig. 14. Colour (g − r) distribution of GC candidates within Re, GC of NGC 5018 (left) and in the elliptical over-density region (right) after annular and random background subtraction (shown in red and green histograms, respectively). The grey dashed histogram (with arbitrary normalisation, to get a similar scale) represents the spectroscopically confirmed GCs in NGC 1399 (left) and in the entire Fornax cluster (right) from the FDS data (Cantiello et al. 2020).

The colour distribution of the GC candidates in NGC 5018 shows no evidence of the well-known colour bimodality feature. This is likely due to the small wavelength separation between the g- and r-passbands combined with the relatively low S/N of most GC candidates at faint magnitudes, which increases photometric errors and smooths out any residual bimodality. Similarly, the GCs in NGC 1399 do not show clear bimodality in (g − r) colour, though their distribution is bimodal in (u − r) and (g − i) colours, which involve wider wavelength separations (Cantiello et al. 2020). However, the absence of bimodality in NGC 5018 might also result from some GC candidates being relatively young and blue (Hilker & Kissler-Patig 1996), but still passing our parametric selections, for example due to internal reddening from the dust in the galaxy core which makes blue GCs appear redder.

We also inspect the (g − r) colour distribution of the GC candidates over the elliptical over-density region (right panel in Fig. 14). The colour profiles reveal two peaks: a blue component with (g − r) ∼ 0.45 mag and a relatively redder component, peaking at (g − r) ∼ 0.80 mag. For comparison, the entire GC sample from the FDS data is plotted as a grey hashed histogram. We observe a bimodal colour distribution for the GC candidates in the intra-group space of NGC 5018. Unlike the Fornax cluster GCs, it appears that the bulk of the GC population in the elliptical over-density region has two colour peaks. By inspecting the mean distance of blue (g − r ≤ 0.65 mag) and red (g − r > 0.65 mag) GC populations from the centre of the elliptical over-density, the two sub-populations appear indistinguishable, as the blue GCs have a mean group-centric radius of 13.2′, while the red GC population has a mean of 12.9′. When measured with respect to the centre of NGC 5018, the mean radius is 16.3′ for the blue and 13.0′ for the red GCs. Based on this, we suggest that a certain fraction of the extended blue GC population in the intra-group space could represent the original blue old GCs system of NGC 5018 that was possibly dispersed due to the tidal interactions with the neighbouring galaxies. The tidal interactions might also have resulted in some younger GCs forming in NGC 5018, but it is unlikely that these young GCs are dispersed in the IGL, which would instead be more dominated by old and metal-poor ones.

To further inspect the colour distribution, we tested the properties of the brightest GCs in the sample, which reduces scatter caused by sources with low S/N ratios. The results are shown in Fig. 15, where we plot (g − r) colour histograms for GCs with: (i) 21.7 mag ≤ m_g ≤ 24.1 mag (brighter than 1σ from the m_g^TOM; bright sample, top panels), and; (ii) 24.1 mag ≤ m_g ≤ 25.3 mag (between 1σ brighter than the m_g^TOM and up to m_g^TOM; fainter sample, bottom panels). These histograms are presented for two regions: one within 5 R_e of NGC 5018 (left panels), and the other for elliptical over-density region (right panels). Magnitude selection was applied only in the g-band, while all other selection criteria remain the same reported in Table 3.

Fig. 15. Colour (g − r) distribution of GC candidates. Top panel: Same as Fig. 14 but for GC candidates with 21.7 ≤ mg ≤ 24.1 mag. The change in magnitude is applied only in g-passband and the rest of the selection criteria is same as in Table 3. Bottom panel: Similar to top panel, but for GC candidates with 24.1 ≤ mg ≤ 25.3 mag.

Similar to Fig. 14, red and green histograms represent the annular and random background subtracted distributions in Fig. 15. This test allows us to come up with the following considerations: (i) bright GCs on NGC 5018 (top left panel): the background subtracted colour distribution shows a dip at (g − r) ∼ 0.6 mag, suggesting a bimodality that was not evident in the global sample (Fig. 14 left panel); (ii) fainter GCs on NGC 5018 (bottom left panel): the peak at (g − r) ∼ 0.7 mag observed in Fig. 14 is preserved; (iii) bright intra-group GC candidates (top right panel): the (g − r) colour distribution shows two peaks, similar to the full sample in Fig. 14. However, the sample size is insufficient for a robust identification, particularly of the blue peak; and (iv) fainter intra-group GC candidates (bottom right panel): the distribution resembles that of the full sample in Fig. 14, although with some differences in the shape. In conclusion, this test supports the presence of a broad colour distribution for intra-group GCs, with two colour peaks. For NGC 5018, there are hints of potential bimodality among the brightest GC candidates. However, further insights would require deeper, multi-band data, for instance, from the Legacy Survey of Space Time (LSST) survey.

6.3. NGC 5018-LSB1 dwarf galaxy

In this section, we briefly discuss the properties of the dwarf galaxy identified in Sect. 6.1. As previously noted, inspection of the g- and r-band images, along with the 2D GC candidates map, reveals a diffuse nucleated galaxy with a plume of GC over-density extending towards it from NGC 5018. The brightness profile of the galaxy also stretches in the direction of NGC 5018, suggesting a possible tidal origin due to interactions between the two galaxies.

Using standard techniques for the morphological and photometric analysis of faint diffuse objects (see Mirabile et al. 2024), we estimated the main properties for the dwarf, summarised in Table 5. Based on these characteristics and assuming the group distance (Table 1), we classify NGC 5018-LSB1 as an Ultra-Diffuse Galaxy (UDG) candidate adopting the van Dokkum et al. (2015) definition. However, due to its faint magnitude and the presence of relatively bright point-like contaminants, further data are needed to confirm this result.

6.4. GC luminosity function

In this section we focus on the GCLF analysis within R_e, GC of NGC 5018. Using the gr-matched catalogue would require deriving combined g- and r-passband completeness functions. This approach would set our reference to the worst passband and require spatial and colour-dependent corrections, which, as anticipated, would be excessive for the present dataset. Therefore, we decided to proceed by inspecting the luminosity function using single-band data. Although single-passband catalogues exhibit higher contamination, they offer greater completeness. Due to poorer image quality and depth in the r-passband compared to the g passband (see Figs. 3, 4, 6, and Table 2), we decided to analyse the GCLF using only the g-passband data.

To obtain an accurate GCLF, we corrected it for the fraction of undetected sources. For the sources on NGC 5018, we used the completeness function derived from the region containing this galaxy (red curve in Fig. 16). For the background correction, we used the mean completeness derived from the four outermost off-galaxy regions (yellow, brown, purple, and olive-coloured boxes in the top left panel of Fig. 10) selected for the completeness analysis. These regions correspond to the locations of the two background areas used in this study. The derived completeness function for these backgrounds is shown in Fig. 16 (blue curve). As expected, Fig. 16, similar to Fig. 4, shows that the completeness on-galaxy is lower than that of the background (off-galaxy) regions.

Fig. 16. Completeness functions derived for the GCLF analysis in the g-passband. The red curve represents the completeness function obtained from the region on NGC 5018 (same as red curve in right panel in Fig. 4). The blue curve represents the mean completeness function derived for the two background regions (explained in Sect. 6.4). The vertical dashed black line is the estimated TOM in this passband (mgTOM). The grey shaded region is the adopted GCLF range in this band (see Table 3).

Figure 17 (left) presents the luminosity function of the GC candidates after applying the completeness correction within R_e, GC of NGC 5018 (black dashed histogram) and the annular and random background. Figure 17 (right) shows the residual luminosity functions obtained by subtracting the background luminosity functions from that of NGC 5018. In both cases, we fit the residual with a Gaussian function for the annular and random background regions independently (red and green dashed curves, respectively). The fitting was performed using the curve_fit¹³ task from the SciPy python library. This was a two-step process: first, rough guesses for the Gaussian parameters were input, and the resulting fits were then used as new inputs to obtain optimised fits.

Fig. 17. Luminosity function of GC candidates. Left: g-band completeness corrected luminosity function of GC candidates within Re, GC of NGC 5018 (black dashed histogram) and for the annular background (red dashed histogram) and random background (green dashed histogram). Right: GCLF obtained after annular background subtraction (solid red histogram) and random background subtraction (solid green histogram), both fitted with Gaussian function (dashed red and green curves, respectively). The vertical dashed black line represents the expected TOM of the GCLF in this passband.

Table 6 lists the various parameters obtained from the GCLF fitting along with the expected values (estimated in Sect. 5.1). The fitted peak of the GCLF is in good agreement with the expected values. By averaging the two m^TOM values and using our adopted M_g^TOM = −7.5 ± 0.2 mag, we obtain a distance modulus of 32.90 ± 0.45 mag, corresponding to a distance of 38.02 ± 7.89 Mpc for NGC 5018. This result agrees with the expected value (36.0 ± 1.7 Mpc) reported in Table 1. The observed dispersion of the GCLF in both cases appears slightly broader than the estimated value for both backgrounds.

As a test, we analysed the luminosity functions of the GC candidates in the elliptical over-density region using the gr-matched catalogue. We also divided the full population of this region into blue (g − r ≤ 0.65 mag) and red GC candidates (g − r > 0.65 mag). Contrary to the single passband case, we did not make a more detailed analysis of the GCLF peak and width, as that would require a completeness correction on the combined gr-matched catalogue. For the full population, after background subtraction, we observed that the luminosity functions behaved as Gaussian peaking at ∼25.2 mag, as expected for GCs in the NGC 5018 group. The peak of blue GCs residual was observed at a slightly brighter magnitude. The red GCs residual peak was instead observed at a slightly faint magnitude level (∼25.5 mag). This is consistent with the findings in literature (e.g. Brodie & Strader 2006; Peng et al. 2009).

Although we do not observe any (g − r) colour bimodality for NGC 5018, we do observe it in the case of GC candidates in the elliptical over-density region (Fig. 14). It is wort noting that the peak colours of the typical metal-poor and metal-rich GCs are (g − r) ∼ 0.60 and 0.75 mag, respectively, as shown in the FDS GC catalogue (Cantiello et al. 2020, Figs. 8 and 15). So as a further test, we divide the GC sample of the elliptical over-density region into three groups: blue GCs (0.30 < g − r ≤ 0.45 mag), intermediate GCs (0.45 < g − r ≤ 0.85 mag) and red GCs (0.85 < g − r ≤ 1.10 mag); and inspect their luminosity functions shown in Fig. 18. The figure shows the luminosity functions of blue, intermediate and red GCs in top, middle and bottom rows, respectively, for the elliptical region (left panels), the two backgrounds (middle panels), and residual obtained after background subtractions (right panels). As in the previous case, these luminosity functions are obtained using the gr-matched catalogue and, thus, they are not completeness corrected. We observe a larger density of intermediate colour GCs in these luminosity functions. Also, we observe that the residuals for all three GC sub-populations appear roughly Gaussian. Although the sample size of blue and red GCs is relatively small, we observe that the peak of luminosity functions of blue GCs is at a brighter magnitude (∼24.9 mag) than the intermediate GCs (∼25.2 mag). This is consistent with a possible brighter TOM for blue GCs or a residual contamination from young-ish bright star clusters. The peak of luminosity function of red GCs (∼25.5 mag) appears to be fainter than the intermediate GCs consistent with the results in the literature (e.g. Di Criscienzo et al. 2006).

Fig. 18. Luminosity function of GC candidates. Top row: For blue GC candidates (0.30 < g − r ≤ 0.45 mag) from the gr-matched catalogue in the elliptical over-density region (black dashed histogram in the left panel), annular and random backgrounds (red and green dashed histograms, respectively, in the middle panel) and residuals obtained after background subtractions (red and green solid histograms in the right panel). Middle and Bottom rows: Same as top row, but for intermediate GC candidates (0.45 < g − r ≤ 0.85 mag) and red GC candidates (0.85 < g − r ≤ 1.10 mag), respectively (see Sect. 6.4).

As a further test, we also inspected the radial density profiles (background subtracted) and 2D distribution maps of blue, intermediate and red GCs. Unsurprisingly, the intermediate GCs have a larger density of sources, compared to blue and red GCs, in the radial density profiles with the peaks lying between R_ellipse = 10′ and 22′, corresponding to regions of NGC 5018 and other GC over-densities surrounding the group (similar to Fig. 12). On the other hand, the radial density profiles of blue and, especially of the red GCs appeared relatively flat. In the 2D distribution maps, we observed a clear over-density of intermediate GCs on top of NGC 5018 and in the red dashed ellipse shown in Fig. 10. Blue and red GCs appeared instead to have a more patchy surface density. In particular, we observed blue and red GC over-densities aligning along the five bright galaxies of the group or with the tidal features.

6.5. Total number of GCs (N_GC) and specific frequency (S​_N)

To estimate the specific frequency for NGC 5018, we use the mean of the two background corrected N_GC values (N_GC = 485 ± 150) obtained from the GCLF fitting described in the previous section. It is useful to note that the numbers reported in Table 6, obtained by integrating the GCLF up to the observed TOM from the fitting, are derived by doubling the results from integration twice: once for the GCLF part fainter than the TOM and once more to account for the spatial coverage due to the fact that we only consider the GC population within R_e, GC (see Sect. 5.2 for the definition).

For the S​_N estimate, we use Eq. (1) and derive an estimate for NGC 5018 total V-passband magnitude using the transformation equation provided on the SDSS webpage (Lupton 2005) to convert g- and r-passband magnitudes to the V-passband magnitude:

$$\begin{array}{c}\hfill V=g-0.5784\times (g-r)-0.0038.\end{array}$$(5)

Using m_g = 10.9 mag for NGC 5018 from Spavone et al. (2018) and the (g − r) = 0.7 ± 0.2 mag from the same paper (as reported in their Table 6), we obtain m_V = 10.52 ± 0.12 mag. At our adopted distance modulus this leads to M_V = −22.28 ± 0.15 mag. Our derived value for M_V is consistent with the value reported in Rampazzo et al. (2007). The resulting S​_N = 0.59 ± 0.27 is consistent within the errors with the existing estimates from the literature reported in Table 7.

Considering the GC distribution over the elliptical over-density region (red dashed ellipse in Fig. 10) and the total area of the assumed geometry, we derive an order-of-magnitude estimate of the total intra-group GC population. Unlike the approach used to estimate the total population of NGC 5018, we use the gr-matched catalogue, which, as previously discussed, has different completeness properties compared to the single-band catalogues. For this, we assume the GCLF in the g- and r-bands of the intra-group GC candidates is roughly complete down to the TOM (see Fig. 4). After background subtraction, the number obtained needs to be doubled to account for the fainter half of the GCLF and then multiplied by the total area of the ellipse. This gives a total GC population of 4800 ± 470 and 3100 ± 370 after annular and random background subtraction, respectively. However, since the gr-matched catalogue is less complete than the single-band catalogues, both total populations and their uncertainty should be considered lower limits.

We can use these numbers to estimate the specific frequency for the group: S​_N, gr. The area over which Spavone et al. (2018, see their Fig. 5) estimates the total luminosity of the NGC 5018 group (L_g = 1.7 × 10¹¹ L_⊙; Sect. 2) has a similar geometry, though it is less extended, covering roughly half of our elliptical region. This total luminosity includes the contributions from the IGL component, NGC 5018, NGC 5022 and MCG-3-34-013. By doubling the total luminosity of the group¹⁴ and adding the V-band luminosities of NGC 5006 (using M_B reported in Table 1 and assuming B − V ∼ 0.6 mag for spiral galaxies; Tomita et al. 1996) and PGC 140148 (using M_B reported in Table 1 and assuming B − V ∼ 0.9 mag for lenticular galaxies; Barway et al. 2005), we estimate the total V-band magnitude of the NGC 5018 group to be M_V, gr = −24.5 mag. Then using a mean intra-group GC population of N_GC, gr ∼ 4000, we obtain a specific frequency of S​_N, gr ∼ 0.6. This should be considered a lower limit (Peng et al. 2011; Durrell et al. 2014), as the total IGL luminosity is likely overestimated (we assumed twice the estimate from Spavone et al. 2018, neglecting the fact that the IGL fades at larger radii), while the number of GCs is underestimated for reasons explained above.

7. Discussion

In this study, we used VEGAS survey data to investigate the GC system in the NGC 5018 group. The quality and characteristics of the VEGAS imaging data allowed us to identify GC candidates across a field of 1.25 × 1.03 sq. degrees, further enabling us to reveal the presence of an intra-group GC system in this galaxy group.

Our analysis revealed several key features of the GC system in the NGC 5018 group. The 2D distribution maps show a notable elliptical over-density of GC candidates, offset from the centre of NGC 5018. This over-density extends diagonally in the direction of the brightest galaxies in the group and includes a plume-like structure to the north-east of NGC 5018 (Fig. 7). The consistency of this feature across various tests (Figs. A.1 and A.3), and its alignment with the geometry of the IGL, suggests that it represents a genuine intra-group GC population rather than an artefact of our selection criteria or background contamination. It is important to recognise that using an elliptical geometry serves as a simplification to make our analysis and presentation clearer, rather than accurately depicting the true geometry of the intra-group GCs component.

The radial density profiles support this finding, showing a relatively flat distribution of GC candidates accompanied by complex structures with several local peaks (Fig. 12). Despite the relatively low efficiency of GC detection around NGC 5018, we observed a significant peak in GC density on the galaxy, with a general decrease towards the outskirts (Fig. 11). A plume-like feature in the 2D GCs density map points towards a local nucleated dwarf galaxy which is aligned with NGC 5018. By inspecting the morphological and photometric parameters of this dwarf galaxy, NGC 5018-LSB1, we find it appears to be a UDG candidate. The radial profiles of GC candidates around other bright galaxies in the group indicate less prominent GC systems, but the radial density profiles around the two spiral galaxies in the group, NGC 5022 and NGC 5006 (Fig. 13), suggest they blend into the intra-group GC population.

In terms of colour distribution, the GC candidates in NGC 5018 display a broad range of colours without a clear bimodal distribution (left panel in Fig. 14). This lack of bimodality may be influenced by photometric limitations, such as the relatively small separation between the g- and r-passbands. Nevertheless, it might also be due to the presence of a fraction of relatively young GCs (Hilker & Kissler-Patig 1996).

On the contrary, the (g − r) colour profile of GC candidates in the elliptical over-density region (right panel in Fig. 14) reveals two peaks: a bluer peak at g − r ∼ 0.45 and a red peak at ∼0.80 mag. When inspecting the mean distance of blue (g − r ≤ 0.65 mag) and red (g − r > 0.65 mag) GC populations from the centre of the elliptical region, we find that they have similar spatial distributions. However, with respect to the centre of NGC 5018, the mean distance of the blue GC population is 16.3 arcmin, while the red population is at 13.0 arcmin. Thus, relative to NGC 5018, the blue population of the intra-group GCs is more extended than the red population. This may suggest that a portion of this extended intra-group blue GC population originally belonged to NGC 5018 and was later dispersed due to tidal interactions with neighbouring galaxies.

Our results from the GCLF analysis, based solely on the g-band data, indicate that the GCLF observed in NGC 5018 has characteristics (peak and width) consistent with expectations (Fig. 17). Using the results from the GCLF, we estimate the total population of GCs to be N_GC = 485 ± 150. This corresponds to a specific frequency of S​_N = 0.59 ± 0.27, which agrees with previous estimates from the literature (based on smaller areas and/or shallower photometry). It also classifies NGC 5018 as a relatively GC-poor galaxy, as elliptical galaxies similar to NGC 5018 typically have S​_N > 1.

The GCLF analysis of intra-group GC candidates within the elliptical over-density region (Fig. 18) show that peak of the luminosity function of blue GCs (0.30 < g − r ≤ 0.45 mag) is observed at a slightly brighter magnitude compared to the intermediate GCs (0.45 < g − r ≤ 0.85 mag). Also, the peak of luminosity function of red GCs (0.85 < g − r ≤ 1.10 mag) is observed at a slightly faint magnitude level than the intermediate GCs, consistent with the results in the literature.

The low S​_N value, combined with the blue GC population observed in the intra-group space, could be explained by the loss of GCs from NGC 5018, which enriched the intra-group environment due to ongoing interactions between the galaxies in the group, together with the brighter magnitude due to the post-merger status of the group. For the intra-group GC population, we estimate a lower limit of N_GC, gr ∼ 4000, which translates to S​_N, gr > 0.6 when combined with the intra-group light and galaxy luminosity estimates from Spavone et al. (2018).

We also examined the observed field for potential ultra-compact dwarf (UCD) candidates (Hilker 2006; Evstigneeva et al. 2008). In the gr-matched catalogue, which are complete at the magnitude levels expected for UCDs, we applied a cut two magnitudes brighter than the brightest limit for GCs in the g- and r-bands and used a stricter elongation cut (≤1.5 in both passbands). The (g − r) colour and CI selection criteria remained unchanged. The resulting UCD candidate counts and 2D distribution map showed a flat, homogeneous spatial distribution. After background subtraction, the number of UCD candidates was consistent with zero. However, this does not necessarily imply the absence of UCDs, as they could be masked by statistical fluctuations, particularly from Milky Way stars in the field.

8. Summary

We present an analysis of GCs in the NGC 5018 group using the deep, multi-passband and wide-field imaging from the VEGAS survey obtained using the VST telescope. A summary of the main results we obtained is provided below.

1. The 2D distribution map shows an over-density of GC candidates on the brightest member of the group, NGC 5018. No significant GC over-densities are observed on the other galaxies in the group.

2. We observe the presence of an intra-group GC population aligning along the five bright galaxies and surrounding the group itself. This observed geometry aligns with IGL detected in the group, but extended to larger group-centric distances.

3. We report the discovery of a local nucleated LSB dwarf galaxy candidate, NGC 5018-LSB1, which is possibly interacting with the nearby NGC 5018 and is a candidate UDG galaxy in the group.

4. The radial density profile of GC candidates in NGC 5018 follows the surface brightness profile of the galaxy, although the central deviation of the profile (R_gal ≤ 1.2′) is due to dust and associated incompleteness in the GC detection and the bump observed in the profile at R_gal > 3.6′ is due to observed plume of GC over-density towards NGC 5018 – LSB1.

5. The radial density profile of the spirals NGC 5022 and NGC 5006 reveals no central GC over-density, but it does highlight the presence of an extended GC population that fades into the intra-group GC population.

6. The colour profile of the GC candidates in NGC 5018 shows a major component with a peak at (g − r) ∼ 0.75 mag. The profile shows no hints of the well-known colour bimodality typically observed in such massive, bright ellipticals. This could be due to a combination of a part of GC population being young, along with the short wavelength separation between the g- and r-passbands and low S/N values characterising GCs at faint magnitude levels.

7. The colour profile of the intra-group GC candidates shows the presence of blue and red GC components with peaks at (g − r) ∼ 0.45 and ∼0.80 mag, respectively. The blue GC component is found to be more extended compared to the red GC component with respect to NGC 5018. This, combined with previous results on the IGL of the system, might suggest that a part of blue GC population in the intra-group space originally belonged to NGC 5018, which was dispersed due to tidal interactions with the neighbouring galaxies.

8. By inspecting the GCLF, we obtained parameters consistent with the known distance to the galaxy and with the expected Gaussian parameters (width and peak).

9. Using the fitted GCLF, we estimated N_GC = 485 ± 150, resulting in S​_N = 0.59 ± 0.27 for NGC 5018, in agreement with the previous works confirming that this galaxy harbours a relatively poor GCs system. For the intra-group GCs, we estimate N_GC, gr ∼ 4000 and a lower limit for specific frequency of the group S​_N, gr > 0.6.

10. The GCLF analysis of intra-group GC candidates show that the peak of luminosity function of blue GCs is at a slightly brighter magnitude level than the intermediate GCs. For red GCs, this is at a slightly faint magnitude level than the intermediate GCs, which is consistent with the results reported in the literature.

The results presented here relied mostly on deep g- and r- passband wide field images from the VEGAS survey. The work, we demonstrate that the dynamically interacting system of galaxies in the NGC 5018 group supports the relevance of wide-area GC studies for constraining the properties of the host environment. In the future, thanks to the advent of large sky surveys, such the LSST from the Vera C. Rubin Observatory and Euclid, we will have wider areas covered with multiple filters and at deep magnitude limits for the NGC 5018 and other galaxy groups. This will greatly contribute to enhancing our understanding of GCs in galaxy groups and their roles in the formation and evolution of host galaxies and their environment.

Data availability

Full Table B.1 is available at the CDS via anonymous ftp to cdsarc.cds.unistra.fr (130.79.128.5) or via https://cdsarc.cds.unistra.fr/viz-bin/cat/J/A+A/694/A231

Acknowledgments

This work is based on visitor mode observations taken at the European Southern Observatory (ESO) La Silla Paranal Observatory within the VST Guaranteed Time Observations; Programme IDs 096.B-0582(B), 097.B-0806(A), and 099.B-0560(A). M.S. wishes to thank the ESO staff of the Paranal Observatory for their support during the observations at VST. PL acknowledges financial support from the Astronomical Observatory of Abruzzo of the Italian National Institute for Astrophysics (INAF-OAAb) and University of Rome Tor Vergata. EI, MS and MC acknowledge the support by the Italian Ministry for Education University and Research (MIUR) grant PRIN 2022 2022383WFT “SUNRISE”, CUP C53D23000850006 and by VST funds. EI, MS, MC and RH acknowledge funding from the INAF through the large grant PRIN 12-2022 “INAF-EDGE” (PI: L. Hunt). This research has made use of the NASA/IPAC Extragalactic Database (NED), which is funded by the National Aeronautics and Space Administration (NASA) and operated by the California Institute of Technology. We also acknowledge the usage of the HyperLeda database (http://leda.univ-lyon1.fr; Makarov et al. 2014) and the Extragalactic Distance Database (EDD, https://edd.ifa.hawaii.edu/). We made extensive use of the softwares SExtractor (Bertin & Arnouts 1996) and Topcat (https://www.star.bris.ac.uk/~mbt/topcat/; Taylor 2005). This research has made use of the AAVSO Photometric All Sky Survey (APASS) database, located at the AAVSO website (https://www.aavso.org). Funding for APASS has been provided by the Robert Martin Ayers Sciences Fund. This research has made use of the VizieR catalogue access tool, Strasbourg Astronomical Data Center (CDS), Strasbourg, France (Ochsenbein 1996). The original description of the VizieR service was published in Ochsenbein et al. (2000). This research has made use of the Sloan Digital Sky Survey (SDSS) database www.sdss.org. Funding for the SDSS V has been provided by the Alfred P. Sloan Foundation, the Heising-Simons Foundation, the National Science Foundation, and the Participating Institutions. SDSS acknowledges support and resources from the Center for High-Performance Computing at the University of Utah. SDSS telescopes are located at Apache Point Observatory, funded by the Astrophysical Research Consortium and operated by New Mexico State University, and at Las Campanas Observatory, operated by the Carnegie Institution for Science. This work made use of Astropy (http://www.astropy.org) a community-developed core Python package and an ecosystem of tools and resources for astronomy (Astropy Collaboration 2013, 2018, 2022). This research made use of Photutils, an Astropy package for detection and photometry of astronomical sources (Bradley et al. 2024).

References

Appendix A: Testing the stability of the elliptical over-density feature

As discussed in Sect. 6.1, we detected the presence of an intra-group GC population roughly centred on the observed field using the 2D distribution map of GC candidates from the gr-matched catalogue (Fig. 10). This feature is aligned along the five bright galaxies in the group and shows a significant overlap with the IGL region studied by Spavone et al. (2018), but extends to larger group-centric radii. To verify the persistence of this feature against our GC selection criteria, the quality of the observational dataset and the possibility that the feature is the result of non GC-contaminants remaining in our catalogue, we conducted several tests described below.

1. We narrowed down the GC selection criteria to study two cases: i) ‘case-1’, where we apply more stringent selections in the g-passband, using the reference magnitude selection for the r-band data; and ii) ‘case-2’, where more stringent selections are applied in both the g- and r-passbands. Additionally, we apply a narrower (g − r) colour range in both cases to obtain a less contaminated, although more incomplete, catalogue of GC candidates compared to our reference catalogue. The new criteria for the two cases are detailed in Tables A.1 and A.2. Figure A.1 shows the resulting 2D distribution maps for the new tests. Although the number of candidates decreased, both new maps confirm the overall appearance of our reference maps. In these new tests though, the GC over-density appears more concentrated in the regions covered by the five bright group members.

2. As a second check, we consider two additional cases: i) ‘case-3’, where we select bright point sources that should be foreground Milky Way stars; and ii) ‘case-4’, where we select candidate extended sources with GC-like magnitudes but non-GC colours, i.e, sources that are likely background galaxies in the observed field. The criteria adopted are shown in Tables A.3 and A.4. If the ellipse were a contamination artefact, we would expect to see some residual evidence of the elliptical over-density. The results shown in the two panels of Fig. A.2 indicate that this is not the case, as we observe a relatively homogeneous distribution of sources in both cases.

3. For a final check, we consider two additional cases: i) ‘case-5’, where we inspect the GC candidates brighter than TOM (as they are the most reliable candidates) while keeping the rest of the selection criteria unchanged; and ii) ‘case-6’, where we lower the elongation threshold compared to our reference. The criteria adopted are shown in Tables A.5 and A.6. The results of these tests are shown in Fig. A.3. In both cases, we observe that the elliptical over-density feature persists; in these tests also the plume we associated with the LSB galaxy NGC 5018-LSB1 appears.

Fig. A.1. 2D distribution maps obtained for ‘case-1’ (left) and ‘case-2’ (right). The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).

Fig. A.2. 2D distribution maps obtained for ‘case-3’ and ‘case-4’. The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).

Fig. A.3. 2D distribution maps obtained for ‘case-5’ and ‘case-6’. The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).

To conclude, we emphasise that despite the changes possible in the selection criteria, the over-density of GC candidates along the region of the five brightest group members persists. This result, combined with previous findings on the IGL distribution, supports the existence of an intra-group GC component aligned but more extended than the observed IGL. However, it is important to note that adopting an elliptical geometry is a simplification meant to facilitate a clearer analysis and presentation of our results, rather than representing the true geometry of the intra-group component.

Appendix B: GC candidate catalogue

All Tables

All Figures

	Fig. 1. Colour-composite image of the bright galaxies in the NGC 5018 group. Numerous foreground stars from our Milky Way and some background galaxies are visible in this image. The image size is 48′ × 60′; north is up and east is to the left. The inset (0.65′ × 0.65′) shows an LSB dwarf galaxy candidate (NGC 5018-LSB1) that we report on in this work (Credit: ESO/ Spavone et al. 2018).
In the text

	Fig. 2. Cutout frames centred on NGC 5018 (top panels) and residuals obtained after model subtraction (bottom panels) in u- (left), g- (middle) and r- (right) bands. Each frame is of the size 8.4′ × 8.4′; north is up and east is left.
In the text

Fig. 3. Comparison between VST and APASS magnitudes (Δ mag = mVST − mAPASS) in u- (left), g- (middle), and r- (right) passbands. Blue dots represent all the sources in ugr-matched catalogue which is matched with APASS catalogue. Red dots represent the bright, unsaturated and point-like sources that we select for photometric comparison. Median and RMSMAD values of Δ mag in the three passbands are shown by green horizontal line and shaded regions, respectively and the values are reported in Table 2.

In the text

Fig. 4. Estimated completeness over the field. Left panel: r-passband image highlighting the regions selected for completeness analysis in the observed field. The area of the entire frame inspected for GCs is 1.25 × 1.03 sq. degrees; north is up, and east is left. The region highlighted in red is centred on NGC 5018. The eight off-galaxy regions surrounding the galaxy are highlighted with green, blue, magenta, cyan, olive, yellow, brown, and purple boxes. Each region is 8.4′ × 8.4′ in size. Middle and right panels: Completeness functions simulated and fitted using the modified Fermi function (Eq. 4) for the u- (middle top), g- (right), and r- passbands (middle bottom). The solid curves are colour-coded according to the boxed regions in the left panel. The vertical solid black line marks the expected TOM of the GCLF in the respective passbands. The grey-shaded region indicates the adopted magnitude range for GC selection in the three passbands (see Sect. 5.1). The dashed black curves represent the expected GCLF in the respective passbands (see Sect. 4.3 for details).

In the text

Fig. 5. 2D histogram of pre-selected GC candidates (based on morphometric and magnitude parameters only) in the (u − r) vs (g − r) plane, showing sources from the ugr-matched catalogue. The over-dense regions (red and pink) represent sequences of contaminants, mostly foreground stars and background galaxies. The isodensity contours, computed using spectroscopically confirmed GCs in the Fornax cluster from FDS data (Cantiello et al. 2020), are over-plotted. The contours range from 10% (innermost) to 90% (outermost) in steps of 10%. Only the sources within the 80% contour (highlighted with a thick black contour) are selected as GC candidates from the ugr-matched catalogue.

In the text

Fig. 6. Morphometric parameters used for the selection of GC candidates: CI, elongation, and (g − r) colour are shown in the top left/middle panels, bottom left and middle panels, and top right panel, respectively. Grey dots represent all sources in the gr-matched catalogue, while red dots indicate GC candidates within Re, GC of NGC 5018 (see Sect. 5.2). The dashed vertical and horizontal black lines indicate the adopted range for GC selection (see Table 3). The black error bars in all the panels denote the median error on magnitude (multiplied by a factor of 7 for visibility purpose) with a bin size of 2.0 mag, and on (g − r) colour (top right panel) within the given magnitude bin size.

In the text

Fig. 7. 2D distribution map of GC candidates in the observed field. Left panel: 2D map for the gr-matched catalogue obtained using bw_adjust = 0.14 and bw_method parameter set to Scott (see Sect. 6.1 for the definitions). The magenta and blue coloured areas represent an over-density of sources whereas the red and yellow-green coloured areas show the under-dense regions. The red coloured areas, mostly situated around the edges of the frame, are under-dense due to the low S/N because of the observing strategy. The position of the galaxies are indicated with crosses of colour yellow (NGC 5018), magenta (NGC 5022), cyan (MCG-03-34-013), orange (NGC 5006), light pink (PGC 140148). The white cross is the LSB dwarf galaxy candidate (NGC 5018-LSB1) reported in Sect. 6.3. Right panel: As in left panel, but using bw_adjust = 0.28. The black contours reported in both the panels indicate the regions higher than 5σ level from the background rms, obtained using bw_adjust = 0.28.

In the text

	Fig. 8. Cutout from the r-passband data of the LSB dwarf galaxy candidate (NGC 5018-LSB1) that we report in Sect. 6.1. The image size is 1.2′ × 1.2′; north is up and east is left. A zoomed-in colour version of this image is shown in the top left corner of Fig. 1.
In the text

	Fig. 9. Normalised 2D histogram of colour-magnitude diagram of GC candidates selected using criteria reported in Table 3, except for (g − r) colour selection, around NGC 5018 (within Re, GC; left panel), random background (middle panel) and annular background (right panel). The (g − r) colour selection interval, and the bright magnitude cut are also shown in the panels with dashed black lines.
In the text

Fig. 10. Same as Fig. 7, but with the different regions selected for background decontamination highlighted. The circles centred on the five bright galaxies represent their 5 Re radius (Re, GC), which corresponds to 2.7′ in the case of NGC 5018. The 10 small black circles, each with a radius of 2.7′, are the regions selected to collectively represent the random background. The large black annulus (with inner and outer radii of 26′ and 29′, respectively) represents the annular background. The red dashed ellipse indicates the elliptical over-density region, with major and minor axes of 50.4′ and 25.2′, respectively. See text for details.

In the text

	Fig. 11. Radial density profile of GC candidates in NGC 5018 after subtracting random background (green points) and annular background (red points). The blue points represent the light profile of the galaxy in the g-passband from Spavone et al. (2018) after an arbitrary vertical shift. The vertical black dashed line represents Re, GC radius.
In the text

Fig. 12. Density profile of GC candidates in the elliptical over-density region obtained by subtracting annular background (left panel) and random background (right panel). The grey shaded region in both the panels beyond 26′ represents the annular background region. The red and green dotted horizontal lines and shaded regions in the left and right panels, respectively, represent the mean and RMSMAD density values within elliptical radius Rellipse < 25′. The yellow shaded regions in both the panels represent the region encompassing NGC 5018 and other GC over-densities surrounding the group (see Sect. 6.1).

In the text

	Fig. 13. Radial density profile of GC candidates on NGC 5006 after subtracting random background (green points) and annular background (red points). The vertical dashed black line represents 5 Re radius.
In the text

Fig. 14. Colour (g − r) distribution of GC candidates within Re, GC of NGC 5018 (left) and in the elliptical over-density region (right) after annular and random background subtraction (shown in red and green histograms, respectively). The grey dashed histogram (with arbitrary normalisation, to get a similar scale) represents the spectroscopically confirmed GCs in NGC 1399 (left) and in the entire Fornax cluster (right) from the FDS data (Cantiello et al. 2020).

In the text

	Fig. 15. Colour (g − r) distribution of GC candidates. Top panel: Same as Fig. 14 but for GC candidates with 21.7 ≤ mg ≤ 24.1 mag. The change in magnitude is applied only in g-passband and the rest of the selection criteria is same as in Table 3. Bottom panel: Similar to top panel, but for GC candidates with 24.1 ≤ mg ≤ 25.3 mag.
In the text

Fig. 16. Completeness functions derived for the GCLF analysis in the g-passband. The red curve represents the completeness function obtained from the region on NGC 5018 (same as red curve in right panel in Fig. 4). The blue curve represents the mean completeness function derived for the two background regions (explained in Sect. 6.4). The vertical dashed black line is the estimated TOM in this passband (mgTOM). The grey shaded region is the adopted GCLF range in this band (see Table 3).

In the text

Fig. 17. Luminosity function of GC candidates. Left: g-band completeness corrected luminosity function of GC candidates within Re, GC of NGC 5018 (black dashed histogram) and for the annular background (red dashed histogram) and random background (green dashed histogram). Right: GCLF obtained after annular background subtraction (solid red histogram) and random background subtraction (solid green histogram), both fitted with Gaussian function (dashed red and green curves, respectively). The vertical dashed black line represents the expected TOM of the GCLF in this passband.

In the text

Fig. 18. Luminosity function of GC candidates. Top row: For blue GC candidates (0.30 < g − r ≤ 0.45 mag) from the gr-matched catalogue in the elliptical over-density region (black dashed histogram in the left panel), annular and random backgrounds (red and green dashed histograms, respectively, in the middle panel) and residuals obtained after background subtractions (red and green solid histograms in the right panel). Middle and Bottom rows: Same as top row, but for intermediate GC candidates (0.45 < g − r ≤ 0.85 mag) and red GC candidates (0.85 < g − r ≤ 1.10 mag), respectively (see Sect. 6.4).

In the text

	Fig. A.1. 2D distribution maps obtained for ‘case-1’ (left) and ‘case-2’ (right). The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).
In the text

	Fig. A.2. 2D distribution maps obtained for ‘case-3’ and ‘case-4’. The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).
In the text

	Fig. A.3. 2D distribution maps obtained for ‘case-5’ and ‘case-6’. The red dashed ellipse from Fig. 10 and the coloured crosses representing the five bright galaxies in the group are reproduced here for reference, including the dwarf galaxy (NGC 5018 - LSB1; white cross).
In the text