© ESO, 2016

1. Introduction

XMM-Newton (Jansen et al. 2001) is the second cornerstone mission from the European Space Agency Horizon 2000 programme. It was launched in December 1999, and thanks to the ~1500 cm² of geometric effective area (Turner et al. 2001) for each of the three X-ray telescopes aboard, it has the largest effective area of any X-ray satellite (Longinotti 2014). This fact, coupled with the large field of view (FOV) of 30′, means that a single pointing on average detects 50 to 100 serendipitous X-ray sources (Watson et al. 2009).

For the past 19 yr, the XMM-Newton Survey Science Centre¹ (SSC), a consortium of ten European Institutes (Watson et al. 2001) has developed much of the XMM-Newton Science Analysis Software (SAS; Gabriel et al. 2004) for reducing and analysing XMM-Newton data and created pipelines to perform standardised routine processing of the XMM-Newton science data. The XMM SSC has also been responsible for producing catalogues of all of the sources detected with XMM-Newton. The catalogues of X-ray sources detected with the three EPIC (Strüder et al. 2001; Turner et al. 2001) cameras that are placed at the focal point of the three X-ray telescopes have been designated 1XMM and 2XMM successively (Watson et al. 2009), with incremental versions of these catalogues indicated by successive data releases, denoted -DR in association with the catalogue number. This paper presents the latest version of the XMM catalogue, 3XMM. The original 3XMM catalogue was data release 4 (DR4). The publication of this paper coincides with the release of 3XMM-DR5. This version includes one extra year of data and increases the number of detections by 7%, with respect to 3XMM-DR4. The number of X-ray detections in 3XMM-DR5 is 565 962, which translate to 396 910 unique X-ray sources. The median flux of these X-ray sources is ~2.4 × 10^-14 erg cm^-2 s^-1 (0.2–12.0 keV), and the data taken span 13 yr. The catalogue covers 877 sq. deg of sky (~2.1% of the sky) if the overlaps in the catalogue are taken into account. 3XMM-DR5 also includes a number of enhancements with respect to the 3XMM-DR4 version, which are described in appendix A. The 3XMM-DR5 catalogue is approximately 60% larger than the 2XMMi-DR3 release and five times the current size of the Chandra source catalogue (Evans et al. 2010). 3XMM uses significant improvements to the SAS and incorporates developments with the calibration. Enhancements include better source characterisation, a lower number of spurious source detections, better astrometric precision, greater net sensitivity and spectra, and time series for fainter sources, both with better signal-to-noise. These improvements are detailed throughout this paper.

A complimentary catalogue of ultra-violet and optical sources detected with the XMM-Newton Optical Monitor (OM, Mason et al. 2001) in similar fields to the XMM catalogue is also produced in the framework of the XMM-Newton SSC and is called the XMM-Newton Serendipitous Ultraviolet Source Survey (XMM-SUSS in its original form, with the more recent version named XMM-SUSS2, Page et al. 2012). 3XMM is also complementary to other recent X-ray catalogues, such as the Chandra source catalogue mentioned above, and the Swift-X-ray Telescope (XRT) Point Source (1SXPS) catalogue (Evans et al. 2014b) of 151524 X-ray point sources detected with the Swift-XRT over eight years of operation. 1SXPS has a sky coverage nearly 2.5 times that of 3XMM, but the effective area of the XRT is less than a tenth of each of the telescopes on board XMM-Newton (Longinotti 2014). Other earlier catalogues include all-sky coverage, such as the ROSAT All-Sky Survey (RASS Voges et al. 1999), but the reduced sensitivity of ROSAT compared to XMM-Newton means that the RASS catalogue contains just 20% of the number of sources in 3XMM-DR4. However, the different X-ray source catalogues in conjunction with 3XMM allow searches for long-term variability.

Whilst this paper covers the 3XMM catalogue in general, some of the data validation presented was carried out on the 3XMM-DR4 version that was made public on 23 July 2013. 3XMM-DR4 contains 531 261 X-ray detections that relate to 372 728 unique X-ray sources taken from 7427 XMM-Newton observations.

The paper is structured as follows. Section 2 contains information concerning the observations used in the 3XMM-DR5 catalogue. Section 3 covers the 3XMM data processing and details changes made with respect to previous catalogues (see Watson et al. 2009), such as the exposure selection, the time-dependent boresight implemented, the suppression of minimum ionising particle (MIP) events, the optimised flare filtering, the improved point spread function (PSF) used for the source detection, new astrometric corrections and the newly derived energy conversion factors (ECFs). We also outline the new source flagging procedure. Section 4 covers the source specific products associated with the catalogue, such as the enhanced extraction methods for spectra and time series and the variability characterisation. Section 5 describes the various screening procedures employed to guarantee the quality of the catalogue, and Sect. 6 outlines the statistical methods used for identifying unique sources in the database. Then, Sect. 7 describes the procedures used to cross-correlate all of the X-ray detections with external catalogues, Sect. 8 discusses the limitations of the catalogue and Sect. 9 characterises the enhancement of this catalogue with respect to previous versions, with the potential of the catalogue highlighted by several examples of objects that can be found in 3XMM, in Sect. 10. Finally, information on how to access the catalogue is given in Sect. 11, and future catalogue updates are outlined in Sect. 12, before concluding with a summary.

2. Catalogue observations

3XMM-DR5 is comprised of data drawn from 7781 XMM-Newton EPIC observations that were publicly available as of 31 December 2013 and that processed normally. The Hammer-Aitoff equal area projection in Galactic coordinates of the 3XMM-DR5 fields can be seen in Fig. 1. The data in 3XMM-DR5 include 440 observations that were publicly available at the time of creating 2XMMi-DR3, but were not included in 2XMMi-DR3 due to the high background or processing problems. All of those observations containing >1 ks clean data (>1 ks of good time interval) were retained for the catalogue. Figure 2 shows the distribution of total good exposure time (after event filtering) for the observations included in the 3XMM-DR5 catalogue and using any of the thick, medium or thin filters, but not the open filter. There are just three observations with one or more cameras configured with the open filter. The number of the 7781 XMM-Newton observations included in the 3XMM-DR5 catalogue for each observing mode and each filter is given in Table 1. Open filter data were processed but not used in the source detection stage of pipeline processing. The same XMM-Newton data modes were used as in 2XMM Watson et al. (2009) and are included in Appendix B of this paper, for convenience.

The only significant difference was the inclusion of mosaic mode data. Whilst most XMM-Newton observations are performed in pointing mode, where the spacecraft is locked on to a fixed position on the sky for the entire observation, since revolution 1812 (30 October 2009), a specific mosaic observing mode was introduced in which the satellite pointing direction is stepped across the sky, taking snapshots at points (sub-pointings) on a user-specified grid. Data from dedicated mosaic mode or tracking (mosaic-like) observations are recorded into a single observation data file (ODF) for the observation. In previous pipeline processing, the pipeline products from the small number of mosaic-like observations were generally generated, at best, for a single sub-pointing only. This is because the pipeline filters data such that only events taken during an interval where the attitude is stable and centred on the nominal observation pointing direction (within a 3′ tolerance), are accepted. Data from some, or all, of the other sub-pointings were thus typically excluded. During 2012, the XMM-Newton Science Operations Centre (SOC) devised a scheme whereby the parent ODF of a mosaic mode observation is split into separate ODFs, one for each mosaic sub-pointing. All relevant data are contained within each sub-pointing ODF and the nominal pointing direction is computed for the sub-pointing. This approach is applied to both formal mosaic mode observations and those mosaic-like/tracking observations executed before revolution 1812. For a mosaic mode observation, the first 8 digits of its 10-digit observation identifier (OBS_ID) are common for the parent observation and its sub-pointings. However, while the last two digits of the parent observation OBS_ID almost always end in 01, for the sub-pointings they form a monotonic sequence, starting at 31. Mosaic mode sub-pointings are thus immediately recognisable in having OBS_ID values whose last two digits are ≥31.

To the pipeline, mosaic mode (and mosaic-like) observation sub-pointings are transparent. No special processing is applied. Each sub-pointing is treated as a distinct observation. Source detection is performed on each sub-pointing separately and no attempt is made to simultaneously fit common sources detected in overlapping regions of multiple sub-pointings. While simultaneous fitting is possible, this aspect had not been sufficiently explored or tested during the preparations for the 3XMM catalogues.

There are 45 observations performed in the dedicated mosaic mode before the bulk processing cut-off date of 8 December 2012, of which 37 are included in 3XMM-DR5, see Appendix A, point 1. None of these was available for catalogues prior to 3XMM. In total, there are 356 processed mosaic sub-pointings in the 3XMM-DR5 catalogue.

3. Data processing

The data used for the 3XMM catalogues have been reprocessed with the latest version of the SAS and the most up to date calibration available at the time of the processing. The majority of the processing for 3XMM-DR5 was conducted during December 2012/January 2013, with the exception of 20 observations processed during 2013. The SAS used was similar to SAS 12.0.1 but included some upgraded tasks required for the pipeline. The SAS manifest for tasks used in the cat9.0 pipeline and the static set of current calibration files (CCFs) that were used for the bulk reprocessing are provided via a dedicated online webpage².

There are 31 observations in 2XMMi-DR3 that did not make it in to 3XMM-DR5, mainly due to software/pipeline errors during processing. Typical examples of the latter problems are due to revised ODFs (e.g. with no useful time-correlation information), more sophisticated SAS software that identified issues hitherto not trapped, or issues with exposure corrections of background flare light curves and pn time-jumps.

The main data processing steps used to produce the 3XMM data products were similar to those outlined in Watson et al. (2009) and described on the SOC webpages³. In brief, these steps were the production of calibrated detector events from the ODFs; identification of stable background time intervals; identification of “useful” exposures (taking account of exposure time, instrument mode, etc.); generation of multi-energy-band X-ray images and exposure maps from the calibrated events; source detection and parameterisation; cross-correlation of the source list with a variety of archival catalogues, image databases and other archival resources; creation of binned data products; application of automatic and visual screening procedures to check for any problems in the data products. The data from this processing have been made available through the XMM-Newton Science Archive⁴ (XSA).

3.1. Exposure selection

The only change applied for identifying exposures to be processed by the pipeline compared to that adopted in pre-cat9.0 processing (Watson et al. 2009, see their Sect. 4.1), was the exclusion of any exposure taken with the Open filter. This was done because use of the Open filter leads to increased contamination from optical light (optical loading). Eight exposures (from five observations) taken with the Open filter were excluded from the data publicly available for the 3XMM-DR5 catalogue.

3.2. Event list processing

Much of the pipeline processing that converts raw ODF event file data from the EPIC instruments into cleaned event lists has remained unchanged from the pre-cat9.0 pipeline and is described in Sect. 4.2 of Watson et al. (2009). However, we describe 3 alterations to the approach used for 2XMM.

3.2.1. Time-dependent boresight

Analysis by both the XMM-Newton SSC and the SOC established the presence of a systematic, cyclic (≈362 day) time-dependent variation in the offset of each EPIC (and OM and RGS) instrument boresight from their nominal pointing positions, for each observation. This seasonal dependence is superposed on a long term trend, the semi-amplitude of the seasonal oscillation being ≈1.2′′ in the case of the EPIC instruments (Talavera et al. 2012). These variations of the instrument boresights have been characterised by simple functions in calibration (Talavera et al. 2012; Talavera & Rodríguez-Pascual 2014). The origin of the variation is uncertain but might arise from heating effects in the support structures of the instruments and/or spacecraft star-trackers – no patterns have been identified in the available housekeeping temperature sensor data though these may not sample the relevant parts of the structure.

During pipeline processing of XMM-Newton observations for the 3XMM catalogues, corrections for this time-dependent boresight movement are applied to individual event positions in each instrument, via the SAS task attcalc, based on the observation epochs of the events.

3.2.2. Suppression of minimum ionizing particle events in EPIC-pn data

High energy particles can produce electron-hole pairs in the silicon substrate of the EPIC-pn detector. While onboard processing and standard pn event processing in the pipeline removes most of these so-called minimum ionizing particle events (Strüder et al. 2001), residual effects can arise when MIPs arrive during the pre-exposure offset-map analysis and can give rise to features that appear as low-energy noise in the pn detector. Typically, these features are spatially confined to a clump of a few pixels and appear only in band 1. However, in pre-cat9.0 pipeline processing, such features were sometimes detected as sources during source detection and these were not always recognised and flagged during the manual flagging process outlined in Sect. 7.4 of Watson et al. (2009). The SAS task, epreject was incorporated into the pipeline processing for 3XMM and in most cases corrects for these MIP events during processing of pn events.

3.2.3. Optimised flare filtering

In previous pipeline processing (pre-cat9.0 pipelines), the recognition of background flares and the creation of good time intervals (GTIs) between them was as described in Sect. 4.3 of Watson et al. (2009), where the background light curves were derived from high energy data and the count rate thresholds for defining the GTIs were based on (different) constant values for each instrument. In the processing for 3XMM, two key changes have been made.

Firstly, rather than adopting fixed count rate thresholds in each instrument, above which data are rejected, an optimisation algorithm has been applied that maximises the signal-to-noise (S/N) for the detection of point sources. Secondly, the light curves of the background data used to establish the count rate threshold for excluding background flares are extracted in an “in-band” (0.5–7.5 keV) energy range. This was done so that the process described below resulted in maximum sensitivity to the detection of objects in the energy range of scientific interest.

The overall process for creating the background flare GTIs for each exposure within each observation involved the following steps:

• 1.

  For each exposure, a high energy light curve (from 7 to 15 keV for pn, >14 keV for MOS) is created, as previously, and initial background flare GTIs are derived using the optimised approach employed in the SAS task, bkgoptrate (see below).

• 2.

  Following the identification of bad pixels, event cleaning and event merging from the different CCDs, an in-band image is then created, using the initial GTIs to excise background flares.

• 3.

  The SAS task, eboxdetect then runs on the in-band image to detect sources with a likelihood >15 – this is already very conservative as only very bright (likelihood ≫100), variable sources are able to introduce any significant source variability component into the total count rate of the detector (accumulated from most of the field).

• 4.

  An in-band light curve is subsequently generated, excluding events from circular regions of radius 60′′ for sources with count rates ≤0.35 counts/s or 100′′ for sources with count rates >0.35counts/s, centred on the detected sources.

• 5.

  The SAS task, bkgoptrate, is then applied to the light curve to find the optimum background rate cut threshold and this is subsequently used to define the final background flare GTIs.

The optimisation algorithm adopted, broadly follows that used for the processing of ROSAT Wide Field Camera data for the ROSAT 2RE catalogue (Pye et al. 1995). The process seeks to determine the background count rate threshold at which the remaining data below the threshold yields a S/N ratio, , for a (constant) source that is a maximum. Here C_s is the number of source counts and C_b is the number of background counts. Since we are interested, here, in finding the background rate cut that yields the maximum S/N and are not concerned about the absolute value of that S/N, then for background light curves with bins of constant width, as created by the pipeline processing, S can be expressed as (1)where N is the number of bins with background count rates below the threshold, r_T, and r_i is the count rate in time bin i: the summation is over the time bins with a count rate <r_T. Time bins are of 10 s width for pn and 26 s for MOS. The process sorts the time bins in order of decreasing count rate. Starting from the highest count rate bin, bins are sequentially removed, computing Eq. (1) at each step. With the count rate of the bin removed at each step representing a trial background count rate cut threshold, this process yields a curve of S/N vs. background count rate cut threshold. The background cut corresponding to the peak of the S/N curve is thus the optimum cut threshold.

In Fig. 3 we show four examples of in-band background time series in the top row, accompanied by the respective S/N vs. background-cut-threshold plots in the bottom row. The first panel in each row represents a typical observation (MOS1) with some significant background flaring activity. The optimum cut level of 1.83 cts/s leads to the creation of GTIs that exclude portions of the observation where the background exceeds the cut threshold. The second panels are for a pn observation with a stable, low background level. The optimum cut in the background includes all the data and thus generates a GTI spanning the entire observation. This is also true for the third panels which show a MOS1 case where the background is persistently high (above the level where the whole observation would have been rejected in pre-cat9.0 pipeline processing). The fourth panels are for an example of a variable background which gives rise to a double peaked S/N v background-rate-cut curve. Here, raising the threshold from ~18 cts/s to ~28 cts/s simply involves a steeply rising background rate early in the observation, causing a dip in the S/N verses background-rate-cut curve. However, as the rate cut threshold is increased above 30 cts/s, although the count rate is higher, a lot more exposure time is available, so the S/N curve rises again and the optimum cut includes almost all the data. It should be emphasised that the fixed cut thresholds used for MOS and pn in previous XMM processings can not be directly compared to the optimised ones used here because of the change in energy band being used to construct the background light curve. It is, however, worth noting that the fixed cuts used previously often result in very similar GTIs to those generated by the optimisation process described above. This is because the previous fixed instrument thresholds were based on analyses that sought to find a representative level for the majority of XMM-Newton observations.

We discuss some of the gains of using this optimisation approach in Sect. 9.3 and some known issues in Sect. 8.

3.3. Source detection using the empirical Point Spread Function (PSF) fitting

The bulk reprocessing for 3XMM took advantage of new developments related to the EPIC PSFs. The source detection stage in previous pipelines (Watson et al. 2009, see their Sect. 4.4.3) made use of the so-called “default” (or medium accuracy) PSF functions determined by ray tracing of the XMM-Newton mirror systems. However, these default PSF functions recognised no azimuthal dependence in the core of the source profile, did not adequately describe the prominent spoke structures seen in source images (arising from the mirror support structures) and were created identically for each EPIC camera.

To address the limitations of the default EPIC PSFs, a set of empirical PSFs were constructed, separately for each instrument, by careful stacking of observed XMM source images over a grid of energy and off-axis angles from the instrument boresights. The cores and spoke patterns of the PSFs were then modelled independently so that implementation within the XMM-Newton SAS calibration software then enables PSFs to be reconstructed that take the off-axis and azimuthal locations of a source into account, as well as the energy band. The details of the issues associated with the default PSF and the construction and validation of the empirical PSF are presented in Read et al. (2011).

The use of the empirical PSF has several ramifications in source detection. Firstly, the better representation of structures in the real PSF results in more accurate source parameterisation. Secondly, it helps reduce the number of spurious detections found in the wings of bright sources. This is because the previous medium accuracy PSFs did not adequately model the core and spoke features, leaving residuals during fitting that were prone to being detected as spurious sources. With the empirical PSFs, fewer such spurious detections are found, especially in the wings of bright objects positioned at larger (>6′) off-axis angles. Thirdly, as a result of the work on the PSFs, the astrometric accuracy of XMM-Newton source positions has been significantly improved (see Read et al. 2011).

3.3.1. Other corrections related to the PSF

During the late stages of testing of the pipeline used for the bulk reprocessing that fed into the 3XMM-DR4 catalogue, an analysis of XMM-Newton X-ray source positions relative to the high-accuracy (≤0.1′′) reference positions of Sloan Digital Sky Survey (SDSS, DR9) quasars identified a small but significant, off-axis-angle-dependent position shift, predominantly along the radial vector from the instrument boresight to the source. The effect, where the real source position is closer to the instrument boresight than that inferred from the fitted PSF centroid, has a negligible displacement on axis and grows to ~0.65′′ at off-axis angles of 15′. This radial shift is due to the displacement between the true position of a source and the defined centroid (as determined by a 3-dimensional, circular, Gaussian fit to the model PSF profile) of the empirical PSF, which grows as the PSF becomes increasingly distorted at high off-axis angles. It should be noted that identifying and measuring this effect has only been possible because of the corrections for other effects (see Sect. 3.3 and below) that masked it, and because of the large number of sources available that provide sufficient statistics. In due course a correction for this effect will be applied directly to event positions, on a per-instrument basis, via the XMM-Newton calibration system, but for the 3XMM-DR4 catalogue, to avoid delays in its production, a solution was implemented within the catcorr SAS task. A correction, computed via a third-order polynomial function, is applied to the initial PSF-fitted coordinates of each source output by emldetect, i.e. prior to the field rectification step, based on the off-axis angle of the source as measured from the spacecraft boresight. This correction is embedded in the RA and Dec columns, which also include any rectification corrections (Sect. 3.4). The correction is computed and applied in the same way for both the 3XMM-DR4 and 3XMM-DR5 catalogues.

A second PSF-related problem that affected 2XMMi-DR3 positions was uncovered during early testing of the empirical PSF (see Read et al. 2011). This arose from a 0.5 pixel error (in both the x and y directions) in the definition of the pixel coordinate system of the medium-accuracy PSF map – as pixels in the PSF map are defined to be 1′′× 1′′, the error is equivalent to 0.5′′ in each direction. When transferred to the image frame during PSF fitting in emldetect, this error in the PSF map coordinate system manifested itself as an offset of up to 0.7′′ in the RA/Dec of a source position, varying with azimuthal position within the field. The introduction of the empirical PSF removes this error.

3.4. Astrometric rectification

3.4.1. Frame correction

Celestial coordinates of sources emerging from the PSF fitting step of pipeline processing of a given observation include a generally small systematic error arising from offsets in the spacecraft boresight position from the nominal pointing direction for the observation. The uncertainty is due to imprecisions in the attitude solution derived from data from the spacecraft’s star-trackers and may result in frame shifts that are typically ~1′′ (but can be as much as 10′′ in a few cases) in the RA and Dec directions and a rotation of the field about the boresight of the order of 0.1 deg. To correct for (i.e. rectify) these shifts, an attempt is made to cross-correlate sources in the XMM-Newton field of view with objects from an astrometric reference catalogue. X-ray sources with counterparts in the reference catalogue are used to derive the frame shifts and rotation that minimise the displacements between them. In all previous pipeline processing (and catalogues derived from them) these frame corrections were estimated using the SAS task, eposcorr, which used a single reference catalogue, USNO-B1.0, and the SAS task, evalcorr, to determine the success and reliability of the outcome (Watson et al. 2009, see their Sect. 4.5).

The processing system used to create the data for the 3XMM catalogues makes use of some important improvements to this field rectification procedure, which are embedded in the new SAS tasks, catcorr that replaces eposcorr and evalcorr. Firstly, the new approach incorporates an iterative fitting function (Nelder & Mead 1965) to find the optimum frame-shift corrections: previously the optimum shift was obtained from a grid-search procedure. Secondly, the cross-match between XMM-Newton and reference catalogue source positions is carried out using three reference catalogues: (1) USNO-B1.0 (Monet et al. 2003); (2) 2MASS (Skrutskie et al. 2006) and, where sky coverage permits; (3) the Sloan Digital Sky Survey (DR9; Abazajian et al. 2009). The analysis is conducted using each catalogue separately. When there is an acceptable fit from at least one catalogue, the RA and Dec frame shifts and the rotation derived from the “best” case are used to correct the source positions. A fit is considered acceptable if there are at least 10 X-ray/counterpart pairs, the maximum offset between a pair (X-ray source, i and counterpart, j) is < 10′′ and the goodness of fit statistic (2)where and q_ij = n_o(r_ij/r_f)². Here, p_ij is the probability of finding the counterpart at a distance >r_ij from the X-ray source position given the combined (in quadrature) positional uncertainty, σ_ij, while q_ij is the probability that the counterpart is a random field object within r_ij. An estimate of the local surface density of field objects from the reference catalogue is made by counting the number, n_o, of such objects within a circular region of radius r_f (set to 1′) around each XMM source. n_x is the number of X-ray sources in the XMM field. The L statistic, which represents a heuristic approach to the problem of identifying likely matching counterparts, is computed over the set of matching pairs and is a measure of the dominance of the closeness of the counterparts over the probability of random matches. The shifts in RA and Dec and the rotation are adjusted within the fitting process to maximise L. Extensive trials found that if L ≥ 5, the result is generally reliable. Where more than one reference catalogue gives an acceptable solution, the one with the largest L value is adopted.

In the 3XMM catalogues, the corrected coordinates are placed in the RA and Dec columns; the original uncorrected coordinates are reported via the RA_UNC and Dec_UNC columns. A catalogue identifier for the catalogue yielding the “best” result is provided in the REFCAT column. If the best fit has parameter values (e.g. the number of matches used) that fall below the specific constraints mentioned above, the original, uncorrected positions are retained (written to both the RA and Dec and RA_UNC and Dec_UNC columns) and the REFCAT identifier takes a negative value. Further details may be found in the documentation for the catcorr task. This new rectification algorithm is successful for about 83% of observations, which contain 89% of detections, reflecting a significant improvement compared to the previous approach where ~65% of fields could be corrected. The main gain comes from the use of the 2MASS catalogue which is particularly beneficial in obtaining rectification solutions in the galactic plane – it should be pointed out that similar gains would be obtained with eposcorr if used with the expanded set of reference catalogues. It should be noted that the extracted lists of objects from each of the three reference catalogues that lie within the full EPIC field of view for a given observation, are provided to users of XMM-Newton data products via the file-type=REFCAT product file, which is used by the task, catcorr.

3.4.2. Systematic position errors

As discussed in Sect. 9.5 of Watson et al. (2009), for the 2XMM catalogue (and relevant to subsequent incremental catalogues in the 2XMM series), the angular deviations of SDSS (DR5) quasars (Schneider et al. 2007) from their XMM-Newton X-ray counterparts, normalised by the combined position errors, could not be modelled by the expected Rayleigh distribution unless an additional systematic uncertainty (SYSERR parameter in 2XMM) was added to the statistical position error (RADEC_ERR parameter in 2XMM and see Appendix C) derived during the PSF fitting process. Watson et al. (2009) showed that this systematic was not consistent with the uncertainty arising from the rectification procedure used for the 2XMM processing and ultimately adopted an empirically-determined systematic error value that produced the best match between the distribution of XMM-quasar offsets and the expected Rayleigh curve.

As part of the upgrade applied to the rectification process for the bulk reprocessing used for the 3XMM catalogues, the uncertainty arising from this step has been computed, in particular, taking into account the error component arising from the rotational offset. Errors (1σ) in each component, i.e., on the RA offset, Δα_c, on the Dec offset (Δδ_c) and on the rotational angle offset (Δφ_c), have been combined in quadrature to give an estimate of the total positional uncertainty, Δr, arising from the rectification process as (3)where θ_c is the radial off-axis angle, measured in the same units as Δα_c and Δδ_c and Δφ_c is in radians.

Inclusion of this rectification error (column SYSERRCC in the 3XMM catalogues, see Appendix C), in quadrature with the statistical error, leads to a generally good agreement between the XMM-quasar offset distribution and the expected Rayleigh distribution compared to the previous approach and indicates that the empirically-derived systematic used in pre-3XMM catalogues is no longer needed. This is discussed further in Sect. 9.2.

3.5. Energy Conversion Factors (ECFs)

A number of improvements in the calibration of the MOS and pn instruments have occurred since the previous, 2XMMi-DR3, catalogue was produced, which lead to slight changes in the energy conversion factors (ECFs) that are used for converting count rates in the EPIC energy bands to fluxes (see Watson et al. 2009, Sect. 4.6 and see Appendix C). Of note is the fact that MOS redistribution matrices were provided for 13 epochs at the time of processing for 3XMM and for three areas of the detector that reflect the so-called “patch”, “wings-of-patch” and “off-patch” locations (Sembay et al. 2011).

For the 3XMM catalogues a simple approach has been adopted. ECFs were computed following the prescription of Mateos et al. (2009), for energy bands 1 to 5 (0.2–0.5 keV, 0.5–1.0 keV, 1.0–2.0 keV, 2.0–4.5 keV and 4.5–12.0 keV respectively) and band 9 (0.5–4.5 keV), for full-frame mode, for each EPIC camera, for each of the Thin, Medium and Thick filters. A power-law spectral model with a photon index, Γ = 1.7 and a cold absorbing column density of N_H = 3 × 10²⁰ cm^-2 was assumed. As such, users are reminded that the ECFs, and hence the fluxes provided in the 3XMM catalogues, may not accurately reflect those for specific sources whose spectra differ appreciably from this power-law model – see Sect. 4.6 of Watson et al. (2009).

For pn, the ECFs are calculated at the on-axis position. The pn response is sufficiently stable that no temporal resolution is needed. For MOS, to retain a direct connection between the ECFs and publicly available response files, the ECFs used are taken at epoch 13 and are for the “off-patch” location. The latter choice was made because the large majority of detections in an XMM-Newton field lie outside the “patch” and “wings-of-patch” regions, which only relate to a region of radius ≤40′′, near the centre of the field. The use of a single epoch (epoch 13) was made to retain simplicity in the processing and because the response of the MOS cameras exhibits a step function change (due to a gain change) between epochs 5 and 6, with different but broadly constant values either side of the step. None of the 13 calibration epochs represent the average response and thus no response file exists to which average ECFs can be directly related. The step-function change in the responses for MOS is most marked in band 1 (0.2–0.5 keV) for the “patch” location, where the maximum range in ECFs either side of the step amounts to 20%. Outside the “patch” region, and for all other energy bands, the range of the ECF values with epoch is ≤5% and is ≤2.5% for the “off-patch” region. Epoch 13 was chosen, somewhat arbitrarily, as being typical of epochs in the longer post-step time interval.

The ECFs, in units of 10¹¹ cts cm² erg^-1, adopted for the bulk reprocessing of data used for 3XMM, are provided in Table 2, for each camera, energy band and filter. The camera rate, ca_RATE, and flux, ca_FLUX, are related via ca_FLUX = (ca_RATE/ECF) (where ca is PN, M1 or M2).

3.6. Updated flagging procedures

A significant issue in terms of spurious detections in XMM-Newton data arises from detections associated with out-of-time (OoT) events. For sources that do not suffer significantly from pile-up, the background map used by emldetect includes a component that models the OoT features. However, for sources where pile up is significant, the OoT modelling is inadequate. This can give rise to spurious sources being detected along OoT features. For the more piled up objects, the numbers of spurious detections along OoT features can become large (tens to hundreds).

Another feature arising from bright sources that affects the MOS instruments is scattered X-rays from the Reflection Grating Arrays (RGA). These manifest themselves as linear features in MOS images passing through the bright object, rather similar in appearance to OoT features. These features are not modelled at all in the background map.

In previous catalogues, spurious detections associated with OoT and RGA features have simply been masked during manual screening. In the cat9.0 pipeline, for the first time, an attempt has been made to identify the presence of OoT and RGA features from piled up sources and to flag detections that are associated with them.

The SAS task, eootepileupmask, is used for this purpose. This task uses simple instrument (and mode) -dependent pre-defined thresholds to test pixels in an image for pile-up. Where it detects pixels that exceed the threshold, the column containing that pixel is flagged in a mask map for the instrument. The task attempts to identify and mask columns and rows associated with such pixels in OoT and RGA features.

Once the pile up masks are generated, the SAS task, dpssflag is used to set flag 10 of the PN_FLAG, M1_FLAG, M2_FLAG, EP_FLAG columns in the catalogues for any detection whose centre lies on any masked column or row.

4. Source-specific product generation

4.1. Optimised spectral and time series extraction

The pipeline processing automatically extracts spectra and time series (source-specific products, SSPs), from suitable exposures, for detections that meet certain brightness criteria.

In pre-cat9.0 pipelines, extractions were attempted for any source which had at least 500 EPIC counts. In such cases, source data were extracted from a circular aperture of fixed radius (28′′), centred on the detection position, while background data were accumulated from a co-centred annular region with inner and outer radii of 60′′ and 180′′, respectively. Other sources that lay within or overlapped the background region were masked during the processing. In most cases this process worked well. However, in some cases, especially when extracting SSPs from sources within the small central window of MOS Small-Window mode observations, the background region could comprise very little usable background, with the bulk of the region lying in the gap between the central CCD and the peripheral ones. This resulted in very small (or even zero) areas for background rate scaling during background subtraction, often leading to incorrect background subtraction during the analysis of spectra in XSPEC (Arnaud 1996).

For the bulk reprocessing leading to the 3XMM catalogues, two new approaches have been adopted and implemented in the cat9.0 pipeline.

• 1.

  The extraction of data for the source takes place from an aperture whose radius is automatically adjusted to maximise the signal-to-noise (S/N) of the source data. This is achieved by a curve-of-growth analysis, performed by the SAS task, eregionanalyse. This is especially useful for fainter sources where the relative importance of the background level is higher.

• 2.

  To address the problem of locating an adequately filled background region for each source, the centre of a circular background aperture of radius, r_b = 168′′ (comparable area to the previously used annulus) is stepped around the source along a circle centred on the source position. Up to 40 uniformly spaced azimuthal trials are tested along each circle. A suitable background region is found if, after masking out other contaminating sources that overlap the background circle and allowing for empty regions, a filling factor of at least 70% usable area remains. If none of the background region trials along a given circle yields sufficient residual background area, the background region is moved out to a circle of larger radius from the object and the azimuthal trials are repeated. The smallest trial circle has a radius, r_c, of r_c = r_b + 60′′ so that the inner edge of the background region is at least 60′′ from the source centre - for the case of MOS Small-Window mode, the smallest test circle for a source in the central CCD is set to a radius that already lies on the peripheral CCDs. Other than for the MOS Small-Window cases, a further constraint is that, ideally, the background region should lie on the same instrument CCD as the source. If no solution is found with at least a 70% filling factor, the background trial with the largest filling factor is adopted. For the vast majority of detections where SSP extraction is attempted, this process obtains a solution in the first radial step and a strong bias to early azimuthal steps, i.e. in most cases an acceptable solution is found very rapidly. For detections in the MOS instruments, about 1.7% lie in the central window in Small-Window mode and have a background region located on the peripheral CCDs. Importantly, in contrast to earlier pipelines, this process always yields a usable background spectrum for objects in the central window of MOS Small-Window mode observations. This approach to locating the background region was adopted primarily to provide a single algorithm that works for all sources, including those located in the MOS small window, when used. However, a drawback relative to the use of the original annular background region arises where sources are positioned on a notably ramped or other spatially variable background (e.g. in the wings of cluster emission), where the background that is subtracted can vary, depending on which side of the source the background region is located.

In addition, the cat9.0 pipeline permits extraction of SSPs for fainter sources. Extraction is considered for any detection with at least 100 EPIC source counts (EP_8_CTS). Where this condition is met, a spectrum from the source aperture (i.e. source plus background) is extracted. If the number of counts from spectrum channels not flagged as “bad” (in the sense adopted by XSPEC) is >100, a spectrum and time series are extracted for the exposure. The initial filter on EPIC counts is used to limit the processing time as, for dense fields, the above background location process can be slow.

4.2. Attitude GTI filtering

Occasionally, the spacecraft can be settling on to, or begin moving away from, the intended pointing direction within the nominal observing window of a pointed XMM-Newton observation, resulting in notable attitude drift at the start or end of an exposure. Image data are extracted from events only within “Good Time Intervals” (GTIs) when the pointing direction is within 3′ of the nominal pointing position for the observation. However, in pre-cat9.0 pipelines, spectra and time series have been extracted without applying such attitude GTI filtering. Occasionally, this resulted in a source location being outside or at the edge of the field of view when some events were being collected, leading to incorrect transitions in the time series. In some cases, these transitions gave rise to the erroneous detection of variability in subsequent time series processing. In the cat9.0 pipeline, attitude GTI filtering is applied during the extraction of spectra and time series.

4.3. Variability characterisation

As with pre-cat9.0 pipeline processing, the pipeline processing for the 3XMM catalogues subjects each extracted exposure-level source time series to a test for variability. This test is a simple χ² analysis for the null hypothesis of a constant source count rate (Watson et al. 2009, see their Sect. 5.2). Sources with a probability ≤10^-5 of being constant have been flagged as being variable in previous XMM-Newton X-ray source catalogues and this same approach is adopted for 3XMM.

In addition, for 3XMM, we have attempted to characterise the scale of the variability through the fractional variability amplitude, F_var (provided via the PN_FVAR, M1_FVAR and M2_FVAR columns and associated FVARRERR columns), which is simply the square root of the excess variance, after normalisation by the mean count rate, ⟨ R ⟩, i.e. (4)(e.g. Edelson et al. 2002; Nandra et al. 1997 and references therein), where S² is the observed variance of the time series with N bins, i.e. in which R_i is the count rate in time bin i. For the calculation of the excess variance, (S² − ⟨ σ_err² ⟩), which measures the level of observed variance above that expected from pure data measurement noise, the noise component, ⟨ σ_err² ⟩, is computed as the mean of the squares of the individual statistical errors, , on the count rates of each bin, i, in the time series.

The uncertainty, Δ(F_var), on F_var, is calculated following Eq. (B2) in Appendix B of Vaughan et al. (2003), i.e. This takes account of the statistical errors on the time bins but not scatter intrinsic to the underlying variability process.

5. Screening

As for previous XMM-Newton X-ray source catalogues (Watson et al. 2009, see Sect. 7), every XMM-Newton observation in the 3XMM catalogues has been visually inspected with the purpose of identifying problematic areas where source detection or source characterisation are potentially suspect. The manual screening process generates mask files that define the problematic regions. These may be confined regions around individual suspect detections or larger areas enclosing multiple affected detections, up to the full area of the field where serious problems exist. Detections in such regions are subsequently assigned a manual flag (flag 11) in the flag columns (PN_FLAG, M1_FLAG, M2_FLAG, EP_FLAG). It should be noted that a detection with flag 11 set to (T)rue does not necessarily indicate that the detection is considered to be spurious.

One significant change to the screening approach adopted for 3XMM relates to the flagging of bright sources and detections within a halo of suspect detections around the bright source. Previously, all detections in the halo region, including the primary detection of the bright source itself (where discernible), had flag 11 set to True (manual flag) but the primary detection of the bright object itself, also had flag 12 set. The meaning of flag 12 there was to signify that the bright object detection was not considered suspect. The use of flag 12 in this “negative” context, compared to the other flags, was considered to be potentially confusing. For this reason, for the 3XMM catalogues, we have dropped the use of flag 12 and simply ensured that, where the bright object detection is clearly identified, it is un-flagged (i.e. neither flag 11 or 12 are set). Effectively, flag 12 is not used in 3XMM. It should be noted that bright sources that suffer significant pile-up are not flagged in any way in 3XMM (or in previous XMM-Newton X-ray source catalogues).

The masked area of each image is an indicator of the quality of the field as a whole. Large masked areas are typically associated with diffuse extended emission, very bright sources whose wings extend across much of the image, or problems such as arcs arising from single-reflected X-rays from bright sources just outside the field of view. The fraction of the field of view that is masked is characterised by the observation class (OBS_CLASS) parameter. The distribution of the six observation classes in the 3XMM catalogues has changed with respect to 2XMMi-DR3 (see Table 3). The dominant change is in the split of fields assigned observation classes 0 and 1, with more fields that were deemed completely clean in 2XMMi-DR3 having very small areas (generally single detections) being marked as suspect in the 3XMM catalogues. Often these are features that were considered, potentially, to be unrecognised bright pixels, e.g. detections dominated by a single bright pixel in one instrument with no similar feature in the other instruments. It should be emphasised, however, that the manual screening process is unavoidably subjective.

6. Catalogue construction: unique sources

The 3XMM detection catalogues collate all individual detections from the accepted observations. Nevertheless, since some fields have at least partial overlaps with others and some targets have been observed repeatedly with the target near the centre of the field of view, many X-ray sources on the sky were detected more than once (up to 48 times in the most extreme case). Individual detections have been assigned to unique sources on the sky (i.e. a common unique source identifier, SRCID, has been allocated to detections that are considered to be associated with the same unique source) using the procedure outlined here. The process used in constructing the 3XMM catalogues has changed from that used for the 2XMM series of catalogues (Watson et al. 2009, see their Sect. 8.1).

The matching process is divided into two stages. The first stage finds, for each detection, all other matching detections within 15′′ of it, from other fields (i.e. excluding detections from within the same observation, which, by definition, are regarded as arising from distinct sources) and computes a Bayesian match probability for each pair as Budavári & Szalay (2008)(5)Here, B, the Bayes factor, is given by (6)where σ₁ and σ₂ are the positional error radii of each detection in the pair (in radians) and ψ is the angular separation between them, in radians. p₀ = N_∗/N₁N₂ where N₁ and N₂ are the numbers of objects in the sky based on the surface densities in the two fields and N_∗ is the number of objects common between them. Each of these N values is derived from the numbers of detections in the two fields and are then scaled to the whole sky. The value of N_∗ is not known, a priori, and in general can be obtained iteratively by running the matching algorithm. However, here we are matching observations of the same field taken with the same telescope at two different epochs so that in most cases, objects will be common. Of course this assumption is affected by the fact that the two observations being considered may involve different exposure times, different instruments, filters and modes used and different boresight positions (with sources within their fields of view being subject to different vignetting factors). To gauge the impact of such effects in determining N_∗, trials using an iterative scheme were run, which indicated that taking N_∗ = 0.9min (N₁,N₂) provides a good estimate of N_∗ without the need for iteration. Finally, with all pairs identified and probabilities assigned, pairs with p_match< 0.5 were discarded.

In the second, clustering stage, a figure-of-merit is computed for each detection, referred to here as the goodness-of-clustering (GoC), which is the number of matches the detection has with other detections, normalised by the area of its error circle radius (given by POSERR, see Appendix C). This GoC measure prioritises detections that lie towards the centre of a group of detections, and are thus likely to be most reliably associated with a given unique source. The list of all detections is then sorted by this GoC value. The algorithm works down the GoC-sorted list and for each detection, the other detections it forms pairs with are sorted by p_match. Then, descending this list of pairs, for each one there are four possibilities for assigning the unique source identifiers: i) if both detections have previously been allocated to a unique source and already have the same SRCID, nothing is done; ii) if neither have a SRCID, both are allocated the same, new SRCID; iii) if only one of them has already been assigned a SRCID, the other is allocated the same SRCID; iv) where both detections in the pair have allocated but different SRCIDs, this represents an ambiguous case – for these, the existing SRCIDs are left unchanged but a confusion flag is set for both detections.

This approach is reliable in matching detections into unique sources in the large majority of cases. Nevertheless, there are situations where the process can fail or yield ambiguous results. Examples typically arise in complex regions, such as where spurious sources, associated with diffuse X-ray emission or bright sources, are detected and, by chance, are spatially close to the positions of other detections (real or spurious) in other observations of the same sky region. Often, in such cases, the detections involved will have manual quality flags set (Watson et al. 2009, see their Sect. 7.5 and also Sect. 5 above).

Other scenarios that can produce similar problems include i) poorly centroided sources, e.g. those suffering from pile-up or optical loading; ii) cases where frame rectification (see Sect. 3.4) fails and positional uncertainties are larger than the default frame-shift error of 1.5′′ that is adopted for un-rectified fields; iii) sources associated with artefacts such as out-of-time event features arising from bright objects elsewhere in the particular CCD, or residual bright pixels and iv) where multiple detections of sources that show notable proper motion (which is not accounted for in pipeline processing) can end up being grouped into more than one unique source along the proper motion vector. Overall, in 3XMM-DR5, this matching process has associated 239 505 detections with 70 453 unique sources that comprise more than one detection.

7. External catalogue cross-correlation

The XMM-Newton pipeline includes a specific module, the Astronomical Catalogue Data Subsystem (ACDS), running at the Observatoire de Strasbourg. This module lists possible multi-wavelength identifications and generates optical finding charts for all EPIC detections. Information on the astrophysical content of the EPIC field of view is also provided by the ACDS.

When possible, finding charts are built using g-, r- and i-band images extracted from the SDSS image server and assembled in false colours. Outside of the SDSS footprint, images are extracted from the Aladin image server. The list of archival astronomical catalogues used during the 3XMM processing includes updated versions of those used for the 2XMM and adds some of the most relevant catalogues published since 2007. A total of 228 catalogues were queried including Simbad and NED. Note that NED entries already included in ACDS catalogues (e.g. SDSS) were discarded.

Among the most important additions are:

• i)

  the Chandra source catalogue version 1.1 (Evans et al. 2010). This release contains point and compact source data extracted from HRC images as well as available ACIS data public at the end of 2009. ACDS accesses the Chandra source catalogue using the VO cone search protocol;

• ii)

  the Chandra ACIS survey in 383 nearby galaxies (Liu 2011);

• iii)

  the SDSS Photometric Catalog, Release 8 (Aihara et al. 2011);

• iv)

  the MaxBCG galaxy clusters catalogue from SDSS (Koester et al. 2007);

• v)

  the 2XMMi/SDSS DR7 cross-correlation (Pineau et al. 2011);

• vi)

  the 3rd release of the RAVE catalogue (Siebert et al. 2011);

• vii)

  the IPHAS Hα emission line source catalogue (Witham et al. 2008);

• viii)

  the WISE All-Sky data Release (Cutri & et al. 2012);

• ix)

  the AKARI mid-IR all-sky survey (Ishihara et al. 2010) and version 1.0 of the all-sky survey bright source catalogue (Yamamura et al. 2010);

• x)

  the Spitzer IRAC survey of the galactic center (Ramírez et al. 2008);

• xi)

  the GLIMPSE Source Catalogue (I + II + 3D Churchwell et al. 2009);

• xii)

  the IRAC-24 micron optical source catalogue (Surace et al. 2004); and xiii) the Planck Early Release Compact Source Catalogue (Planck Collaboration VII 2011).

Table 4 lists, for a selection of archival catalogues, the number of EPIC detections having at least one catalogue entry in the 99.73% (3 Gaussian σ) confidence region.

The cross-matching method used for 3XMM is identical to that applied in the former XMM catalogues. Briefly speaking, ACDS retains all archival catalogue entries located within the 99.73% confidence region around the position of the EPIC detection. The corresponding error ellipse takes into account systematic and statistical uncertainties on the positions of both EPIC and archival catalogue entries. The 3XMM implementation of the ACDS assumes that the error distribution of EPIC positions is represented by the 2D Gaussian distribution with thereby correcting for the overestimated error value used during the 2XMM processing.

ACDS identifications are not part of the 3XMM catalogue fits file but are made available to the community through the XSA and through the XCAT-DB⁵, a dedicated interface developed by the SSC in Strasbourg (Motch et al. 2009; Michel et al. 2015). The XCAT-DB also gives access to the entire 3XMM catalogue and to some of the associated pipeline products such as time series and spectra. Quick look facilities and advanced selection and extraction methods are complemented by simple X-ray spectral fitting tools. In the near future, the database will be enriched by the multi-wavelength statistical identifications and associated spectral energy distributions computed within the ARCHES project (Motch & Arches Consortium 2014). Spectral fitting results from the XMMFITCAT database (Corral et al. 2015) are also partially available.

8. Known problems in the catalogue

A number of small but significant issues have been identified that affect the data in the 3XMM catalogues. Two of these affect both the 3XMM-DR4 and 3XMM-DR5 catalogue. The other issues affect only 3XMM-DR4 and are described in Appendix A.

• 1.

  The optimised flare filtering process (see Sect. 3.2.3) returns a background rate threshold for screening out background flares for each exposure during processing. However, while this process generally works well, when the background level is persistently high throughout the observation, the optimised cut level, while formally valid, can still result in image data with a high background level. In principle, such cases could be identified by testing the cut threshold against a pre-determined benchmark for each instrument. However, this is complicated by the fact that, since the analysis is now measured in-band, apparently high background levels can also arise in fields containing large extended sources. To simplify the process of identifying affected fields, a visual check was performed during manual screening and fields where high background levels were suspected were noted and detections from those fields are flagged in the 3XMM catalogues via the HIGH_BACKGROUND column. This screening approach has been somewhat conservative and subjective. A total of 21 779 (20 625) detections from 568 (552) XMM-Newton observations are flagged as such in the 3XMM-DR5: numbers in parentheses are for 3XMM-DR4.

• 2.

  A further issue recognised in the 3XMM catalogue is that of detections in the previous 2XMMi-DR3 catalogue that are not present in the 3XMM catalogue. There are 4921 XMM-Newton observations that are common between 2XMMi-DR3 and the 3XMM-DR5 catalogues, resulting in 349 444 detections in 2XMMi-DR3 and 359 505 detections in 3XMM-DR5. Of these, there are 274 564 point-like detections with a sumflag ≤ 1 in 2XMMi-DR3 and 283 436 in 3XMM-DR5. However, amongst these observations, there are ~54 000 detections that appear in 2XMMi-DR3 that are not matched with a detection in the same observation in the 3XMM-DR5 catalogue within 10′′. About 25 700 of these were classified as the cleanest (SUM_FLAG≤ 1), point-like sources in 2XMMi-DR3 – these are referred to as missing 3XMM detections in what follows. It should be noted that in reverse, there are ~64 000 detections in the 3XMM catalogues that are in common observations but not matched with a detection in 2XMMi-DR3 within 10′′, approximately 33 600 of which are classed as being clean and point-like. The details explaining these “missing sources” are given in Appendix D, but the main reasons for the source discrepancies between the two catalogues are two of the major improvements to the 3XMM catalogue with respect to the 2XMM catalogue, namely the new empirical PSF, described in Sect. 3.3 and the optimised flare filtering (see Sect. 3.2.3). Other origins, such as MIP events which were present in 2XMMi-DR3 but not recognised as such, and mostly removed in 3XMM-DR5, also contribute, but to a lesser extent, to the missing sources. In general though, the improvements to the pipeline that was used to create the 3XMM-DR5 data introduce (generally small) changes in the likelihood values (of mostly real sources). As such, the imposed threshold cut at L = 6 for inclusion in the catalogue results in a fraction of sources that had L< 6 in 2XMMi-DR3 having a likelihood a little above it in the 3XMM-DR5 processing, and vice versa, leading to losses and gains between the catalogue. Overall, the processing for 3XMM-DR5 is shown to be an improvement over the 2XMMi-DR3 procedure (see Appendix D), resulting in more sources.

9. Catalogue characterisation

9.1. General properties

The 3XMM-DR5 catalogue contains 565 962 (531 261) detections, associated with 396 910 (372 728) unique sources on the sky, extracted from 7781 (7427) public XMM-Newton observations – numbers in parentheses are for 3XMM-DR4. Amongst these, 70 453 (66 728) unique sources have multiple detections, the maximum number of repeat detections being 48 (44 for 3XMM-DR4), see Fig. 4. 55 640 X-ray detections in 3XMM-DR5 are identified as extended objects, i.e. with a core radius parameter, r_core, as defined in Sect. 4.4.4 of Watson et al. (2009), >6′′, with 52 493 of these having r_core< 80′′. Overall properties in terms of completeness and false detection rates are not expected to differ significantly from those described in Watson et al. (2009).

9.2. Astrometric properties

As outlined in Sect. 3.4, several changes have been made to the processing that affect the astrometry of the 3XMM catalogues relative to previous XMM-Newton X-ray source catalogues. To assess the quality of the current astrometry, we have broadly followed the approach outlined in Watson et al. (2009). Detections in the 3XMM-DR5 catalogue were cross-correlated against the SDSS DR12Q quasar catalogue (Paris et al., in prep.), which contains ~297 300 objects spectroscopically classified as quasars – positions and errors were taken from the SDSS DR9 catalogue. X-ray detections with an SDSS quasar counterpart within 15′′ were extracted. Point-like 3XMM-DR5 detections were selected with summary flag 0 (see Appendix C), from successfully catcorr-corrected fields, with EPIC detection log-likelihood >8 and at off-axis angles <13′. The SDSS quasars were required to have warning flag 0, morphology 0 (point-like) and r′ and g′ magnitudes both <22.0. This yielded a total of 6614 3XMM-QSO pairs. In the 13 cases where more than one optical quasar match was found within 15′′, the nearest match was retained.

The cross-matching used the catcorr-corrected RA and Dec X-ray detection coordinates. The measured separation, Δr, and the overall 1-dimensional XMM position error, σ_1D (=), were recorded. Here σ_pos is the radial positional error, POSERR, in the catalogues, which is the quadrature sum of the XMM positional uncertainties resolved in the RA and Dec directions. As noted by Watson et al. (2009), if the offset between the X-ray source and its SDSS quasar counterpart, Δr, is normalised by the total position error, σ_tot, i.e. x = Δr/σ_tot, the distribution of these error-normalised offsets is expected to follow the Rayleigh distribution, (7)Errors on the SDSS quasar positions were included in σ_tot though they are generally ≤0.1′′, much smaller than the vast majority of σ_1D values in 3XMM-DR5. The SDSS position errors were circularised using where σ_maj and σ_min are the errors in the major and minor axis directions of the SDSS position error ellipse. These were then combined in quadrature with the XMM position error to obtain . No systematic error was included for the QSO position error.

In Fig. 5 we show the distribution of x values for the selected XMM-QSO pairs as the red histogram, with the expected Rayleigh distribution overlaid in black. While there is broad overall agreement between the data and model, it is clear that there is a deficit of sources for 0.8 <x< 2 and an excess for x> 2.5. A total of 739 XMM-QSO pairs lie at 2.5 <x< 6 while the model predicts 291, the excess of 448 representing 6.8% of the total in the histogram.

To investigate the small discrepancies between the distribution of x values for the selected XMM-QSO pairs and the Rayleigh distribution, we carried out a number of tests detailed in Appendix E. The main results of these tests indicate that the excess of 3XMM-DR5 detections with error-normalised offsets from their SDSS quasar counterparts >3.5 appears to have a modest dependence on the off-axis location of the detection in the XMM field of view. A small fraction of detections at higher off-axis angles have either incorrect positions or underestimated errors, while sources near the centre may have slightly overestimated errors. Further, given that the sources at higher off-axis angles in EPIC images have rather elongated PSF profiles, the assumption of a circularly symmetric positional uncertainty distribution is probably not adequately representative of such sources, which may be contributing to the observed discrepancies.

Subsequent analysis investigated whether the discrepancies could be reduced by making phenomenological adjustments to the XMM position errors. In this analysis, the filtering applied to XMM and SDSS sources was similar to that outlined above but only matches within 5′′ were used and no magnitude limits were imposed on the SDSS objects, resulting in 6858 pairs. A two parameter adjustment was considered in which the XMM position errors were scaled by a constant, a, and a systematic error, b, was added in quadrature (i.e. ). One parameter adjustments, where only the systematic was added (i.e. where a is set to 1) were also tested. The error normalised XMM-quasar separations were recomputed as , where now combines with σ_QSO in quadrature. Using this prescription, the data were fit to the Rayleigh function to obtain the best-fit values for a and/or b, using a maximum likelihood approach. The results are shown in Fig. 6. While these parameterisations of the XMM position errors did improve the fit, particularly bringing the data in the tail closer to the expected Rayleigh curve, the fit remains poor overall, driving the peak of the data to x ≈ 0.7 (it should peak at 1.0) and introducing a notable excess at x< 1. Despite the fact these two forms of adjustment to the XMM position errors yield statistically unacceptable fits to the Rayleigh function, as they do improve agreement in the tail (i.e. for a given XMM-counterpart pair, x′<x), they reduce the chance that real matches of 3XMM sources with counterparts from other catalogues (or observations) will be erroneously excluded as candidate counterparts. As such, although the position error column values in the 3XMM catalogues are not adjusted, we provide the values of a(=1.12) and b(=0.27′′) for the two parameter fit so that users can apply the above adjustments to the XMM position errors if they wish. The one parameter case best fit yields b = 0.37.

Other tests involved (i) imposing a lower bound on the XMM position error ( and (ii) including an off-axis-dependent systematic involving a scalar, c, where Θ is the off-axis angle. These latter modifications provide slightly better matches to the Rayleigh curve but still drive the peak of the data to x ≈ 0.7, again creating an excess at x< 1. A further test in which the XMM position error is defined as for x<x_t and for x ≥ x_t (where d is a simple scalar and x_t is a threshold value in x) does yield a marked enhancement in the likelihood for the fit but in this case, the data undershoot the Rayleigh curve at x> 2 and exceed it at 0.6 <x< 2.

We conclude that while the more complex adjustments to the XMM position errors can formally improve the match between the error-normalised XMM-quasar separations and the Rayleigh curve, none provides a statistically acceptable match. Moreover, the cases that yield the best improvements in the fit likelihood have no compelling technical rationale.

9.3. Background flare filtering

As noted in Sect. 3.2.3, an optimisation algorithm was adopted to determine the count rate threshold for defining the flare GTIs. This process was employed to maximise sensitivity to source detection and can come at the expense of reduced exposure time. Often, the new process results in GTIs that are similar to those derived from the fixed threshold cuts used in pre-cat9.0 pipeline processing. However, in some cases, significant improvements can be obtained in sensitivity.

Of particular interest are cases where the background rises or falls slowly. In such cases, allowing a modest increase in the background count rate can yield a marked increase in exposure time, resulting in a significant improvement in the sensitivity to the detection of faint sources. A good example of this is illustrated in Fig. 7. As is evident from the light curves, the optimised cut threshold includes significantly more exposure time for a very modest increase in background level, producing a factor 5.5 increase in the harvest of detected sources.

Another aspect of the optimised flare filtering approach is that the increase in exposure time can result in exposures being used that were previously rejected in processing with pre-cat9.0 pipelines.

The pre-cat9.0 and cat9.0 light curves in Fig. 7 also highlight the fact that the change of energy band used can yield some significant differences in the strengths and even shapes of flare features in the data.

The implementation of the optimised flare filtering approach was done in conjunction with some of the other upgrades, such as the use of the empirical PSF (see Sect. 3.3). As such, we have not directly isolated the impacts on source detection of the optimised flare filtering process alone. Nevertheless, comparison of the numbers of source detections between the set of 4922 observations that are common to the 2XMMi-DR3 and 3XMM-DR5 catalogues, indicates a net increase of 10 047 detections in 3XMM-DR5, i.e. a 2.9% increase.

9.4. Extraction of spectral and time series products

As described in Sect. 4.1, spectra and time series of detections are now extracted using optimised extraction apertures that are intended to maximise the overall S/N of the resulting product. To assess this, spectra were re-extracted for all detections and exposures for which spectra were produced during the bulk reprocessing, using a circular aperture of fixed radius (28′′) in each case, centred at the same location as the detection position used during the bulk reprocessing. Other than the change of aperture radius, processing was essentially identical to that used in the bulk reprocessing. The S/N, S, of each spectrum was then computed as . Here C_s = C_T − C_b, where C_T is the total number of counts measured in the spectrum from the source aperture, C_s is the number of counts from the source in the source aperture and C_b is the number of background counts in the source aperture, the latter being estimated from the total counts in the background region, scaled by the ratio of source and background region areas. Counts included in this analysis were drawn from PHA channels with quality <5 (in XSPEC terms). The S/N was computed in this way for the spectra from the optimised and fixed apertures – the spectral data used for background subtraction were taken from the same background spectrum (from the bulk reprocessing) in each case and the background counts used were drawn only from the same channels as used for the source counts.

In Fig. 8 the log of the ratio of the S/N values from the spectra extracted from the optimised (S_o) and fixed (S_f) apertures, i.e. log ₁₀(S_o/S_f), is plotted against log (C_T) from the optimised aperture, for MOS1 spectra. Only spectra from the cleanest (SUM_FLAG=0), point-like (EP_EXTENT=0) detections are included.

It is evident from the positive asymmetry about log ₁₀(S_o/S_f) = 0, that the optimisation procedure does improve the S/N of the spectra, especially for spectra with lower (C_T< 500) numbers of extracted counts, as expected. Overall, 67.5% of the MOS1 spectra with 100 <C_T< 50 000 cts (within −1 < log ₁₀(S_o/S_f) < 1 which excludes 21 positive outliers) have higher S/N in the optimised aperture than those extracted from the fixed apertures. The red line in Fig. 8 shows the average of log ₁₀(S_o/S_f) of all the data as a function of C_T and indicates that spectra extracted from the optimised apertures with C_T = 100 cts have, on average, S/N values 12% higher than those extracted in the fixed apertures. It is anticipated that sources detected in fields with high background levels would benefit from the optimisation procedure. Indeed the blue line in Fig. 8, which reflects the subset of detections whose background levels are above 10^-8 cts s^-1 (sub-pixel)^-2 (i.e. amongst the highest 15% of background levels), demonstrates this – spectra of such detections extracted from optimised apertures with C_T = 100 cts, have average S/N values 39% higher than the spectra from the corresponding fixed apertures.

9.5. Extended sources

The detection and characterisation of extended sources for 3XMM was performed as in 2XMM (Watson et al. 2009). The caveats listed in Sect. 9.9 of that paper still apply to 3XMM. However the better representation of the PSF has helped to improve extended source detection and characterisation. Many extended sources with SC_SUM_FLAG = 4 in 2XMM now have SC_SUM_FLAG = 3 in 3XMM, indicating that the region is still complex but the detection itself is unlikely to be spurious. We have also looked at the distribution of extension likelihood vs. flux as in Fig. 15 of Watson et al. (2009). Figure 9 shows that 3XMM considers many bright extended sources to be reliable (SC_SUM_FLAG < 2) whereas in 2XMM most of them had higher flag values indicating more significant issues with the data quality.

We have complemented this study by inter-comparing the 3XMM (DR4) results when a source was observed more than once, and with an independent serendipitous search for clusters of galaxies. We restricted the comparison to the best-quality sources with SC_SUM_FLAG = 0. In 3XMM-DR4, 667 sources have been observed several times as extended, each observation being processed independently. We define as the “reference value” the extension (EP_EXTENT) associated to the detection with the highest likelihood value (EP_8_DET_ML column). We investigated the agreement of the extension parameter between the “reference” and the other observations of the same source. We ignored observations when a given source was not detected as extended (mostly because of insufficient exposure) or when the extension was set to 80″ (maximum value allowed in the fit).

In Fig. 10 we show the distribution (in log space) of the ratio between the “reference” extension Ext_ref and the current one Ext_cur, normalised by the corresponding error equal to , where σ_ref and σ_cur are the extension errors for the “reference” and current observation respectively. We fit the histogram result by a Gaussian function, obtaining a mean value equal to 0.512 (in σ units) with a standard deviation equal to 1.943 (we would expect a mean of 0 and a standard deviation of 1 for random fluctuations). We conclude that there exists an additional scatter larger than statistical (of unknown origin) and that the reference observation, which is also the deepest one, estimates a larger extension on average.

The XCLASS catalogue is based on the analysis of archival observations from the XMM-Newton observatory. The XCLASS team processed 2774 high Galactic latitude observations from the XMM archive (as of 2010 May) and extracted a serendipitous catalogue of some 850 clusters of galaxies based on purely X-ray criteria, following the methodology developed for the XMM Large Scale Survey (Pierre et al. 2007). We used the subsample of 422 galaxy clusters available online⁶ to compare the extension and the count rate obtained for the same sources from the two different procedures (i.e. the XCLASS and 3XMM processing). The analytic expression used to represent extended sources in XCLASS was the same as in 3XMM (β-model with β = 2/3) so the numbers should be directly comparable. All 422 clusters are in 3XMM-DR4, but 59 (mostly faint or irregular objects) were classified as point sources.

For the 363 extended sources in common, we compared the extent and the count rate in the [0.5–2.0] keV band obtained by 3XMM and XCLASS. We found that, for both quantities, the 3XMM estimates seem to be biased low with respect to the XCLASS values. The best fit regression on source extent resulted in a slope of 0.7 (Ext_3XMM ≃ 0.7Ext_XCLASS). Excluding clear outliers (difference of extension larger than 20″, typically very faint sources or very bright sources affected by a strong pile-up) the slope increases to 0.85. We conclude that, even excluding these extreme sources, there remains a bias of ≃15% between the extensions estimated by 3XMM and XCLASS.

There exists a similar (a little smaller) bias on the count rate. However Fig. 11 shows that there exists a close correlation between both ratios, implying that only one parameter describes the difference in extent and count rates and that, if the source extents were forced to agree, the count rates would agree too. There is no obvious way to know whether the 3XMM or the XCLASS estimate is better but, together with the inter-3XMM comparison, this result indicates that the purely statistical extension error underestimates the real error.

10. Examples

Thanks to the wide range of parameters provided in the catalogue, sources matching specific criteria can be isolated (for example variability criteria of X-ray hardness ratios). In this section we show some examples of lightcurves (Fig. 12) and spectra (Fig. 13) extracted from the different EPIC cameras. The plots shown are those associated with the on-line catalogue. Both known and new sources are presented. It is immediately obvious from the two Figures that objects with extremely diverse characteristics are found. Variability on very different timescales is seen in Fig. 12, showing short and long flares, slow rises and steady declines in count rate as well as deep eclipses. From visual examination of the strong variability in Fig. 12c, it was quickly obvious that this new X-ray source was a polar (Webb et al. 2016). Figure 12e shows a strong decline in flux, which, when coupled with the hard spectrum observed for this source, suggests that this might be a previously unknown orphan gamma-ray afterglow.

The spectra shown in Fig. 13 are also very varied and originate from a variety of astrophysical objects, ranging from stars, compact objects, galaxies and clusters of galaxies. An unidentified X-ray source is included in Fig. 13a, which also has a highly variable lightcurve, showing a steady decline in count rate, but with a strong flare superposed. The nature of this source is not obvious and more work will be needed to identify its nature. The sources in the full 3XMM catalogue are of course dominated by unidentified objects, emphasising the large discovery space provided by the catalogue.

11. Catalogue access

The catalogue is provided in several formats. Firstly, a Flexible Image Transport System (FITS) file and a comma-separated values (CSV) file is provided containing all of the detections in the catalogue. For 3XMM-DR5 there are 565 962 rows and 323 columns. A separate version of the catalogue (the slim catalogue) is also provided that contains only the unique sources, i.e. 396 910 rows, and has 44 columns, essentially those containing information about the unique sources. This catalogue is also provided in FITS and CSV format. Ancillary tables to the catalogue also available from the XMM-Newton Survey Science Centre webpages⁷ include the table of observations incorporated in the catalogue and the target identification and classification table.

The XMM-Newton Survey Science Centre webpages provide access to the 3XMM catalogue, as well as links to the different servers distributing the full range of catalogue products. These include, the XMM-Newton XSA, which provides access to all of the 3XMM data products, and the ODF data, the XCat-DB⁸ produced and maintained by the XMM-Newton SSC, which contains possible EPIC source identification produced by the pipeline by querying 228 archival catalogues. Finding charts are also provided for these possible identifications. Other source properties as well as images, time series, spectra, fit results from the XMMFITCAT are also provided. Multi-wavelength data taken as a part of the XID (X-ray identification project) run by the SSC over the first fifteen years of the mission are also provided in the XIDresult database⁹. The LEDAS server¹⁰ provides another way to access the 3XMM catalogue and its products, whilst the upper limit server¹¹ allows the user to specify a sky position and obtain upper limits on the EPIC fluxes of a point source at the position if the location has been observed by XMM-Newton but no source was detected. The catalogue can also be accessed through HEASARC¹² and VIZIER¹³. The results of the external catalogue cross-correlation carried out for the 3XMM catalogue (Sect. 7) are available as data products within the XSA and LEDAS or through the XCat-DB. The XMM-Newton Survey Science Centre webpages also detail how to provide feedback on the catalogue.

Where the 3XMM catalogue is used for research and publications, please acknowledge their use by citing this paper and including the following:

This research has made use of data obtained from the 3XMM XMM-Newton serendipitous source catalogue compiled by the 10 institutes of the XMM-Newton Survey Science Centre selected by ESA.

12. Future catalogue updates

Incremental releases (data releases) are planned to augment the 3XMM catalogue. At least one additional year of data will be included with each data release. Data release 6 (DR6) will provide data becoming public during 2014 and 2015 and should be released during 2016. These catalogues will be accessible as described in Sect. 11.

13. Summary

This paper presents the third major release of the XMM-Newton serendipitous source catalogue (3XMM), in its original version (3XMM-DR4) and in the first incremental version (3XMM-DR5). The 3XMM catalogues have been constructed by the XMM-Newton Survey Science Centre and the 3XMM-DR5 catalogue becomes the largest catalogue of X-ray sources detected using a single X-ray observatory. The characteristics and improvements of this catalogue, with respect to previous versions, are outlined as well as how to cite and access the catalogue. This paper serves as the reference for future incremental versions of the same catalogue (3XMM-DR6, etc.), as new XMM-Newton data becomes publicly available.

---

Acknowledgments

Firstly, we extend our thanks to the anonymous referee who raised some very useful points, allowing us to improve this paper. We are also extremely grateful for the strong support provided by the XMM-Newton SOC. We also thank the CDS team for their active contribution and support. The French teams are grateful to Centre National d’Études Spatiales CNES for their outstanding support for the SSC activities. The German teams are grateful to the Deutsches Zentrum für Luft- und Raumfahrt for supporting this activity under grants 50 OX 1101, 50 OX 1401 and 50 OG 1101. The University of Leicester acknowledges the financial support of the UK Space Agency and also ESA. F.J.C. acknowledges financial support by the Spanish Ministry of Economy and Competitiveness through grant AYA2012-31447, which is partly funded by the FEDER programme, and grant AYA2015-64346-C2-1-P. M.T.C. acknowledges financial support from the Spanish Ministry of Science and Innovation through grant AYA2010-21490-C02-1. The Italian team acknowledges financial support during the years from the Ministero dell’Istruzione, dell’Universita’ e della Ricerca (MIUR), from the Agenzia Spaziale Italiana (ASI) and from the Istituto Nazionale di Astrofisica (INAF). This research has also made use of the SIMBAD database, of the VizieR catalogue access tool, and of Aladin, operated at CDS, Strasbourg, France, the TOPCAT/Stilts software written by Mark Taylor of the University of Bristol, UK and the NASA HEASARC FTOOLS package¹⁴ (Blackburn 1995), and made extensive use of the ALICE High Performance Computing Facility at the University of Leicester.

References

Appendix A: Known issues affecting 3XMM-DR4 only

• After the creation of the 3XMM-DR4 catalogue, it was discovered that the raw event files from the ODFs of a number of mosaic mode sub-pointing observations contained corrupted data whereby some of the events in a given sub-pointing ODF were actually from another sub-pointing. Since the raw event positions are specified in detector coordinates and are subsequently mapped to their sky locations during pipeline processing by reference to the observation boresight position, which is specified for the given sub-pointing, the celestial positions of these events are wrong and therefore results in some detections having incorrect celestial coordinates. The problem arose in the algorithm used to split the raw parent ODF into sub-pointing ODFs. In some cases all instruments were affected while in others, only one or both of the MOS instruments was affected. Of the 419 mosaic-mode sub-pointing observations included in 3XMM-DR4, 82 are affected to some extent, involving 4918 detections. The affected observations are listed in the watchout section of the XMMSSC 3XMM-DR4 catalogue web pages¹⁵. For 3XMM-DR5, none of the affected mosaic sub-pointing observations is included in the catalogue.

• The vignetting values provided in the 3XMM-DR4 catalogue (for each instrument, for bands 1 to 5) were found to have been computed for an energy of 0 keV rather than the energy relevant to the band. Thus the values for each band of a given instrument are identical. This error does not affect the count rates or fluxes as the vignetting correction applied to them is computed separately and has been verified as correct. It is only the tabulated values in the vignetting columns of the catalogue that are incorrect in 3XMM-DR4 and they are correct in 3XMM-DR5.

• A significant issue identified after the public release of the 3XMM-DR4 catalogue relates to the error values on various quantities. It was established that the error quantities (i.e. columns containing an _ERR at the end) for the XID band (band 9) count rates and fluxes of a significant number (~42 200) of detections (~10% of the catalogue) were substantially wrong (generally being overestimated by factors up to ~100 but in a few cases, up to 1000). A more detailed investigation found that while all error columns are potentially affected (and therefore also any derived parameters involving error-weighted quantities, such as some of the unique source quantities), the frequency and magnitude of the problem is much worse for the XID band data than any other parameter. It has been established that for other key quantities, such as the statistical positional uncertainty (RADEC_ERR) and the instrument count rates and fluxes in other (non-XID) bands, only about 1.3% of detections are affected and, generally, the scale of the problem is very small. For the positional uncertainty, 1.4% of detections have incorrect RADEC_ERR values and only 0.26% of detections have position errors that differ from their correct values by more than 0.05′′ while for only 89 detections does it differ by more than 0.5′′ (of which, 58 are detected as extended sources and 81 have a non-zero quality flag). Furthermore, for 81% of those detections where the position error is wrong by more than ±0.05′′, the correct position error is smaller than that quoted in the 3XMM-DR4 catalogue. The most extreme deviations of the RADEC_ERR values from their correct values are 32′′ larger and 2.3′′ smaller. For the PN band 2 flux errors, only ~1.1% of detections have values that deviate from their correct values by more than 10^-5, when expressed as a fraction of the correct value. For the errors on the XID band photometric quantities (rates, fluxes, counts) the correct error is generally smaller than that given in 3XMM-DR4. Thus, while there is a significant problem with the error quantities on the XID band photometric data in 3XMM-DR4, the problem is much less severe for other quantities. It is emphasised that the correct error quantities are present in 3XMM-DR5.

Appendix B: Data modes of XMM-Newton exposures included in the 3XMM catalogue

The data modes are described in Table B.1.

Appendix C: Definitions relating to 3XMM-DR5 detections referred to in this work

We describe here some of the important quantities relating to 3XMM-DR5 detections that are frequently referred to in the paper.

RADEC_ERR is the statistical position error, defined as (ra_err² + dec_err²)^1/2, where ra_err and dec_err are the 1-sigma errors in the RA and Dec coordinate directions, respectively, determined during the fitting of the PSF to the source image

SYSERRCC is the estimated 1-sigma error from the rectification process, as defined by Eq. (3) in Sect. 3.4.2.

POSERR is the error representing the quadrature combination of RADEC_ERR and SYSERRCC, i.e. (RADEC_ERR² + SYSERRCC²)^1/2.

Count rates for detections are given in the ⟨ca⟩_⟨b⟩_RATE columns in the catalogue for EPIC camera, ⟨ca⟩, in energy band ⟨b⟩, for bands 1–5 and 9. These are the total integrated counts for the detection, derived from within the whole PSF fitted to the source image, divided by the exposure map value at the source position. The count rate values are corrected to the rate on-axis position. The band-8 rates in each camera are the sum over bands 1–5.

Fluxes are provided in the ⟨ca⟩_⟨b⟩_FLUX columns. These are converted from the count rates via energy conversion factors (ECFs) (see table 2), assuming a power-law spectrum with N_H = 3 × 10²⁰ cm^-2 and a power law photon index of 1.7.

The summary flags, in the SUM_FLAG column, provide a simple overview of the quality of the detection, based on a combination of the automatic flags and flags set during manual (visual) screening. Values are: 0 – identifies the best quality detections, i.e. those with no evident complicating factors; 1 − detections where the source parameters may be affected; 2 − cases where the automatic analysis suggests the detection may be a spurious extended source or associated with known detector features but is not flagged by manual screening; 3 – cases where manual screening has flagged the detection but automatic flags are not set; 4 – detections where both automatic and manual screening flags are set.

Appendix D: Detailed description of the issue known as “missing sources”

Of the ~25 700 missing 3XMM detections, up to 8% are found only in the pn band-1 data. Visual inspection of examples and analysis of the pn detector-image data suggests many of these are probably previously unrecognised MIP features, i.e. spurious detections, in 2XMMi-DR3 (see Sect. 3.2.2), though some may well be real, soft sources. A second, difficult-to-quantify percentage (but ≤7%) of the missing 3XMM detections may have detected counterparts in the 3XMM catalogues but be unmatched within 10′′ due to imperfect astrometry in either the 2XMMi-DR3 and/or 3XMM catalogue. A third component of up to around 3% of the missing 3XMM detections may be detections in 2XMMi-DR3 that are associated with hitherto unrecognised/unflagged detector features – such features become apparent when the missing 3XMM detections are plotted in detector coordinates for each EPIC instrument, after allowing for likely real detections in the same regions that are detected in more than 1 instrument.

Other explanations for the missing 3XMM detections include

• A small number (<1%) are pairs of visually verified close sources that were separated in 2XMMi-DR3 but found as either a single extended or a single unresolved point source in 3XMM.

• A small number of cases are likely spurious detections in the wings of bright sources in 2XMMi-DR3 that were not flagged during the manual screening process for 2XMMi-DR3 and were not detected in 3XMM.

The above-mentioned explanations account for ≤20% of all the clean, point-like missing 3XMM detections. Some 75% of the missing 3XMM detections have EPIC likelihoods in 2XMMi-DR3, L, < 10 (90% have L< 15). It might be thought that the missing 3XMM detections could arise from spurious detections due to random statistical background fluctuations (false positives) in 2XMMi-DR3 – the numbers of such detections, estimated from simulations, was discussed in Sect. 9.4 of Watson et al. (2009). Using the cumulative count rates presented in Fig. 10 of Watson et al. (2009) and the distribution of exposure times for observations in 2XMMi-DR3, we estimate around 7500 detections in the common observations might be false positives. This, however, is probably an overestimate of the contribution of false positives to the missing sources because although there are notable changes to the pipeline processing between the 2XMMi-DR3 and 3XMM catalogues, the input ODFs and associated event data are often the same for the common observations, i.e. the data are not independent. It should be noted that of the ~25 700 missing 3XMM-DR5 detections, ~5200 of them belong to unique sources that comprise at least one other 2XMM-DR3 detection, hinting that at least 20% are probably real.

The distributions of the band-8 likelihood for clean, point-like EPIC detections found in 3XMM-DR5 and not in 2XMMi-DR3 (and vice versa) are very similar and strongly biased to low (6 <L< 10) likelihood values. Both are much more strongly concentrated in this range than the distribution of all clean, point-like detections. Evidently, the issue of the “missing” detections is primarily related to changes affecting detections with likelihoods near the L = 6 threshold used for the catalogue.

It is clear that two of the major improvements to the 3XMM catalogue with respect to the 2XMM catalogue, which are the new empirical PSF, described in Sect. 3.3 and the optimised flare filtering (see Sect. 3.2.3), could have an impact on the detection likelihoods and hence the numbers of detected sources. We investigated the impact of these two improvements. Optimising the flare filtering maximises the signal to noise ratio of the sources but also affects the background level, as described in Sect. 3.2.3. To explore the impact of changing the background, we scaled the 3XMM-DR5 background maps around their original values – raised background model values from 3XMM-DR5 images at the positions of faint sources could reduce the detection likelihood below the threshold of 6. Our analysis was limited to scaling the entire original background map for each available instrument by a common factor (in steps of 2% between 90% and 110%). While this is not adequately representative of real background variations between processings, which would vary across the field (see below), it helps to illustrate the potential effects that may occur.

From a subset of 1854 fields, we find up to ~9700 extra detections may appear if the background is systematically underestimated by 10% and ~6800 fewer detections appear if the background is 10% higher than the original level. However, as noted, the differences in the background maps between the 2XMMi-DR3 and 3XMM processings are much more likely to occur on a spatially localised scale rather than a uniform change across the field of view. To look for indicators that this might be the case, we computed ratios of the background maps (3XMM-DR5/2XMMi-DR3) in each instrument and band and looked for deviations of the ratio (relative to the mean of the ratio image) at the positions of 2XMMi-DR3 detections that are missing in 3XMM-DR5. We observe a spread of up to ±20% in the deviations of the ratio at some source positions but no evidence of systematic background over-estimation in a specific instrument or band. Nevertheless, background enhancements that push the EPIC band 8 detection likelihood below 6 could be arising in a different instrument and/or energy band in each case.

The second effect of the improved flare filtering is the impact on the good time intervals. We investigated the relation between exposure time, the number of counts in the source (count number) and detection likelihood, using only the pn parameters of detections that are in both 2XMMi-DR3 and 3XMM-DR5 (i.e. whose EPIC detection likelihood is >6 in each catalogue). We expect that we would see a similar relation for the combined instrument (EPIC) source parameters if we could include detections with EPIC likelihoods below six in the catalogues (which, by definition, we don’t have). Figure D.1 show the ratio of pn source count numbers (DR5/DR3) plotted against the corresponding ratio of median pn exposure times. The data include only point sources with SUM_FLAG≤ 1 that are isolated and not affected by nearby extended sources, in both catalogues. The red points reflect sources whose pn detection likelihood in 3XMM-DR5, L_pn(DR5), is >6 while their pn detection likelihood in 2XMMi-DR3, L_pn(DR3), is <6 – these are detections that, based on their pn data, would be present in 3XMM-DR5 and not in 2XMMi-DR3. The blue points represent data where L_pn(DR5)< 6 and L_pn(DR3)> 6, i.e. which would be in 2XMMi-DR3 and not in 3XMM-DR5.

To investigate the impact of changing the PSF to the empirical model, we reprocessed the 4921 fields that are common to 3XMM-DR5 and 2XMMi-DR3 using the same source-detection steps of the pipeline, input data and calibration files that were employed to create 3XMM-DR5, but changing the new empirical PSF model to the previous “medium-accuracy” model. We found some 8300 clean, point-like 3XMM-DR5 detections have no matching detection obtained with the medium-accuracy PSF within 5′′ (7100 within 10′′), demonstrating that the new PSF has a non-negligible effect on the source detection. It should also be pointed out that emldetect gives the most reliable results when the PSF model used is similar to the true PSF (Feigelson & Babu 2006), implying that the empirical PSF, constructed from observed source data, should provide more reliable sources. Changing both the background and PSF therefore has an impact on the maximum likelihood determined. Figure D.2 shows the relationship between the maximum likelihood (ML) in 3XMM-DR5 and 2XMMi-DR3 for all sources common to both catalogues. More than half the sources have a higher maximum likelihood in 3XMM-DR5 compared to 2XMMi-DR3, indicating generally better sensitivity in 3XMM-DR5. It should also be noted that, as indicated in the emldetect description¹⁶, the maximum likelihood values provide only a rough estimate of the number of expected spurious sources for low count sources (≲9, Cash 1979). As many as 10% of the catalogue sources have counts <9 in at least one instrument, so sources with a low count rate may have an inaccurate likelihood value attributed.

The changes to the pipeline generating the catalogue sources mean that the maximum likelihood value attributed to each source varies from catalogue to catalogue. The dispersion is high (~2 in ML) for the distribution of ML values in one catalogue, given a specific ML value in the other catalogue. Given that we have chosen a threshold of ML ≥ 6 to indicate a real source, many sources with a ML close to this value in one catalogue will have an ML< 6 in the other, due to this broad dispersion. Indeed, as many as 10 000 sources can be found below this threshold in the other catalogue (2XMMi-DR3 when comparing with 3XMM-DR5 or 3XMM-DR5 when comparing with 2XMMi-DR3), and are therefore considered as missing when considering one catalogue over the other. In reality, it is simply that the ML value has fallen slightly below our chosen threshold and therefore the source is just not included in the catalogue.

In conclusion, the main reason for the missing sources is that the changes in the pipeline processing procedure between 2XMMi-DR3 and 3XMM-DR5 have produced slightly different likelihood values (of mostly real sources), so that detections near the likelihood threshold cross the boundary, in both directions, between catalogues, resulting in different lists of detections. Overall, however, the 3XMM-DR5 procedure is better, resulting in more sources.

Appendix E: Astrometry and the deviation from the Rayleigh distribution

To explore the cause(s) of the deviations from the Rayleigh curve, we first examined whether the outlier pairs in the tail excess might be spurious XMM-quasar associations, though as noted by Watson et al. (2009), the false match rate for quasars is expected to yield far fewer mismatches than the numbers found in the tail excess. To test this possibility we compared the distribution of the 3XMM-DR5 EPIC band 8 flux (F_X) to SDSS (r band) flux (F_r) ratio (i.e. F_X/F_r) for XMM-quasar pairs from x> 3.5 (the region of the excess tail where the Rayleigh function predicts negligible numbers) to that from x< 0.8 (where the data and model match well). While pairs from the tail do have a slightly (25%) higher F_X/F_r ratio on average than those at x< 0.8 it is too small a difference to be explained by systematic mismatching. This conclusion is supported by considering XMM-quasar pairs in the tail whose X-ray detections belong to 3XMM-DR5 unique sources that include one or more other X-ray detections with a quasar counterpart. Amongst 104 such unique sources involving an XMM-quasar pair from the tail, only 13 are cases where all other constituent XMM-quasar pairs have x> 3.5. These 13 cases might reflect mismatches of the XMM detections and quasars. However, the X-ray detections involved represent only 8% of the X-ray detections with quasar counterparts that make up the 104 unique sources, suggesting most of the XMM-quasar pairs from the tail are not mismatches.

We then constructed distributions for many XMM catalogue parameters (e.g. position errors, off-axis angle, count rates, equatorial and galactic location, exposure times, nearest-neighbour distance etc.), comparing the distributions of the data subsets from x> 3.5 and x< 0.8. The position error (POSERR) distributions of the two subsets are very similar while the XMM-quasar separations are markedly different, having an average of 0.4′′ for data with x< 0.8 compared to 5.5′′ for the x> 3.5 subset. There is an indication that the points at x> 3.5 tend to lie at larger off-axis angles. No other trends could be discerned from the distributions for other parameters. To push this further, we also cross-matched the 6614 3XMM-DR5 detections with SDSS quasar counterparts against the Chandra catalogue (Evans et al. 2014a). Within 10′′, 745 XMM detections have one or more matches with Chandra detections – we retained only the nearest match in the few instances where multiple matches were present. The 3XMM-DR5 detections from the tail do tend to be notably more offset from their Chandra counterparts compared to those detections with x< 3.5. Furthermore, although numbers are more limited, for the XMM-quasar pairs in the tail with Chandra counterparts, the error-normalised offsets between the Chandra detections and the SDSS quasars appear to provide a better match to the Rayleigh distribution – there is no evidence of a similar tail excess. This hints at the positions of the 3XMM-DR5 detections in the tail being incorrect. While, alternatively, their position errors may be underestimated, if so, there is no clear evidence the errors are being systematically underestimated, e.g. being incorrectly characterised with off-axis angle. It is worth noting that while the proportion is lower in the central regions of the field of view, even in the 10′<θ< 12′ annulus

(θ is the EPIC off-axis angle), 5.6% of the XMM-quasar pairs have x> 3.5 – this demonstrates there is not a generic problem with sources at higher off-axis angles.

As noted, there is an indication that XMM sources at x> 3.5 tend to lie at higher off-axis angles in the field than those from lower x values. This is illustrated in figure 5 where, alongside the histogram of error-normalised offsets for all the XMM-quasar pairs (red), we show the histograms for data from off-axis angles θ< 5′ (blue), 5′ ≤ θ< 10′ (green) and 10′ ≤ θ< 15′ (grey). For sources near the centre of the field, the distribution peaks too early, at x = Δr/σ_tot ~ 0.65 but better matches the tail at x> 2.5. Conversely, data from 10′ ≤ θ< 15′ peak near x = 1 but account for much of the excess tail. We examined whether this could arise from, for example, an error in the rotation correction of the rectification process (see Sect. 3.4.1) in some observations. If so, for a given field, one might anticipate the quasar counterparts having a systematic offset, either ahead of, or behind, the X-ray detection, in a sector oriented perpendicular to the radial vector, r, from the field boresight to the X-ray position. This is not, however, evident in fields that contain useful numbers (up to 22) of XMM-quasar pairs, one or more of which come from x> 3.5. We also performed a more detailed analysis in which the circularised statistical XMM position error, RADEC_ERR, used previously was replaced with an error derived from an error ellipse: an elliptical error contour should better characterise positional uncertainties arising from the elongated PSF profiles that become evident at larger off-axis angles. Assuming the ellipse is oriented with the major axis tangential to r, the mean geometry of the error ellipse as a function of off-axis angle was obtained via the separate errors in RA and Dec of all 3XMM-DR5 detections (available in the initial emldetect source lists, though only the circularised RADEC_ERR value is provided in the final observation summary source lists). For each XMM-quasar pair, the idealised error ellipse of the X-ray source was scaled to the measured RA and Dec errors and a mean of the major and minor axes was obtained. Using the mean elliptical positional uncertainty to normalise the XMM-quasar offsets, even when combined with the elliptical errors for the rectification correction and the elliptical quasar position errors, still results in a notable excess at x> 3.5.

We conclude that the excess of 3XMM-DR5 detections with error-normalised offsets from their SDSS quasar counterparts >3.5 appears to have a modest dependence on the off-axis location of the detection in the XMM field of view, with a small fraction of detections at higher off-axis angles having either incorrect positions or underestimated errors, while sources near the centre may have slightly overestimated errors. However, no systematic cause is identified.

All Tables

All Figures