© The Authors 2023

1 Introduction

Minor bodies in planetary systems potentially have an intimate relationship with life. Minor bodies might provide complex molecules to planets to facilitate the formation of life (e.g. Oba et al. 2022), while later on impacts can play a significant role in its evolution through climate changes and mass extinctions (e.g. Alvarez et al. 1980). Deepening our understanding of this potential relationship is a high priority on the European science agenda (e.g. Cosmic Voyage 2050¹) and on the US science agenda (e.g. Origins, Worlds, and Life: A Decadal Strategy for Planetary Science and Astrobiology 2023-2032²). Near-Earth objects (NEOs) are a reservoir that surely interacts with Earth’s biosphere and physically represents the closest instance of minor bodies that are at our disposal. NEOs are asteroids or comets whose perihelion occurs at less than 1.3 astronomical units (au), meaning that close approaches with the Earth might occur at some point. The size of these objects ranges from metres to tens of kilometres. Currently, more than 30 000 NEOs are catalogued in the Minor Planet Center (MPC³) and the discovery rate is of the order of thousands per year. While we are certain that the vast majority of the largest NEOs have already been discovered, very little is known about the majority, the ones in the range of a few metres to about 150 metres.

With NEOs, the minor body ↔ life relationship can be approached from a scientific perspective: understanding the role of minor bodies in the origin and long-term evolution of life in a planetary system, such as our Solar System. For example, NEOs might deliver the meteorites that provide the ‘starter set’ for life in the form of prebiotic molecules (Oba et al. 2022). Furthermore, NEO compositions can put constraints on competing formation scenarios for rocky planets like Earth (Burkhardt et al. 2021). Thus, the characterization of the orbits and physical composition of NEOs is valuable to advance our scientific understanding of the minor body ↔ life relationship.

The minor body ↔ life relationship can also be approached from a societal perspective: the impact hazard poses a significant threat to technological civilizations on a planet (Borovička et al. 2013, Popova et al. 2013, Brown et al. 2013) and calls for planetary defence strategies. The Tunguska and Chelyabinsk impact are recent reminders. Advanced technological civilizations on a planet can also use their neighbouring minor bodies to mine (rare) chemical elements and compounds (Hein et al. 2020), possibly as a foraging stop in space exploration (Davis et al. 1993). In conclusion, characterization of the orbits and physical composition of NEOs is valuable both to advance both our scientific understanding of the formation and evolution of planetary systems and life and to serve societal goals related to planetary defence and space-based technological infrastructures.

Small NEOs are only observable from the Earth during their brief close approaches to our planet. Every day, observatories such as Pan-STARRS (Vereš et al. 2015), Catalina (Christensen et al. 2019), ATLAS (Tonry et al. 2018), and the Zwicky Transient Facility (ZTF; Bellm et al. 2019) are discovering new NEO candidates that need observational follow-up for their confirmation. It is not always easy to provide follow-up observations of these objects since their changing observational conditions, usually linked to a growing position uncertainty, make them challenging objects for most telescopes. Particularly challenging are the observations of some subgroups of NEOs such as the interior-Earth objects (IEOs; Sheppard et al. 2022) or Earth companions such as the Trojans (see e.g. Santana-Ros et al. 2022). Once the close approach with the Earth ends, it is impossible to gather any further data on a NEO, until the next close approach happens, which sometimes might take several years. The only way to get any additional information about these bodies once their observational period is over is to rely on data mining. Therefore, it is important to systematically mine and monitor new archival data for the appearance of these objects. This can be done using serendipitous discovery in archival observations not dedicated to NEOs. For example, the Dark Energy Survey (DES; Banda-Huarca et al. 2019; Abbott et al. 2021; Bernardinelli et al. 2022), the Kilo-Degree Survey (KiDS; Mahlke et al. 2021) and the VISTA Hemisphere Survey (VHS; McMahon et al. 2013; Popescu et al. 2016) have been used to search for minor bodies, including NEOs. Similarly, upcoming wide-area surveys such as the ESA Euclid mission (Carry 2018; Pöntinen et al. 2020) and the Legacy Survey of Space and Time (Grav et al. 2016; Jones et al. 2018) and their combination (Guy et al. 2022) will have a role in NEO astrometric and photometric observations. These astronomical surveys have complementary data, and are often deeper and survey further away from the ecliptic compared to NEO-dedicated surveys. In particular, the detection of a known NEO in an observing dataset before its discovery (hereafter precovery) provides information about the NEO’s motion at an earlier epoch at a likely complementary location in its orbit because the orbit uncertainty depends on the fraction of the orbital arc that is covered during discovery. The (p)recovery of one or more points far away from the discovery and immediate follow-up observations can significantly improve the accuracy of the orbital parameters. This is especially true when re-calibrating archival observations to astrometric reference catalogues such as Gaia that postdate the observations.

In this way astrophysical missions not only have a scientific purpose, but can also obtain a societal spin-off. This paper reports an exploratory pilot for that societal spin-off that consists in the re-use of astrophysical imaging data for precovery and also in the re-use of investments put in an information system called AstroWISE to make it generically capable of performing precovery in a wide range of astronomical archives within a single data flow environment and with a single precovery pipeline. This exploratory AstroWISE precovery pilot was driven by the following quantitative and qualitative questions: What fraction of detectable NEOs can we precover? And how automated can we make the precovery? What astrometric and photometric accuracy can be achieved? What level of automation can be achieved in the precovery workflow? What are the major challenges to exploiting various astronomical surveys for NEO space safety purposes? What value do these precoveries have for planetary defence? Do precoveries have benefits for NEO science?

The structure of the paper is as follows. Section 2 describes the data and lists of NEOs used for precovery. In Sect. 3, the pipeline is described in detail. In Sect. 4, the findings and results of NEO precovery are presented. In Sect. 5, we summarize and conclude with the main results of this work and discuss the next steps⁴.

2 Data, software systems, and applications

In this work, we use the archival imaging data of OmegaCAM, the wide-field imaging camera of the VLT Survey Telescope (VST) at ESO’s Cerro Paranal Observatory. In the past OmegaCAM was used for several wide-field surveys including the Kilo-degree Survey (KiDS; Kuijken et al. 2019), the VST-ATLAS surveys (Shanks et al. 2015), the Fornax Deep Survey (FDS; Peletier et al. 2020), and VEGAS (Iodice et al. 2021). OmegaCAM has 32 science CCDs with a field of view (FoV) of about 1^o × 1° on the sky. In its first decade of science operation, OmegaCAM covered a large fraction of the southern hemisphere (Fig. 1). Considering the large volume of the dataset of OmegaCAM, we developed a pipeline to automatize the process of (p)recovery. Here we focus our (p)recovery effort on two datasets of NEOs.

First, the NEO risk list⁵. This list is provided by the ESA NEO Coordination Centre (ESA-NEOCC) and consists of about 1350 known NEOs with a non-negligible chance of impact in the next hundred years. This list is updated regularly based on the most recent observations. We use the list available on 1 February 2022.

Second, the NEO full list. This list consists of all the known NEOs (about 30 000 sources). We used the version available on 1 June 2022.

These data are processed and analysed using several public web tools to collect the predicted properties of NEOs, using the ASTROWISE system (Begeman et al. 2013; McFarland et al. 2013) for data management, image processing, and calibration, and using dedicated software applications for streak detection, astrometry, and photometry. We briefly introduce these software systems and software applications here.

ASTROWISE stands for Astronomical Wide-field Imaging System for Europe. ASTROWISE is an environment consisting of hardware and software; it is federated over institutes over Europe. It was developed to scientifically exploit the ever-increasing avalanche of data produced by astronomical wide-field surveys. ASTROWISE is an all-in-one system: it allows a scientist to process raw data, calibrate data, perform post-calibration scientific analysis, and archive all the results in one environment. The system architecture links together all these commonly discrete steps in data analysis.

Fig. 1 An overview of a decade of observations with OmegaCAM/VST. Top: the sky coverage of the OmegaCAM/VST observations over 10 yr of observations of the southern hemisphere. The ASTRO-WISE archive contains 361977 individual exposures. Bottom: the histogram of exposure times of the OmegaCAM/VST observations in u (54918 exposures), ɡ (79127 exposures), r (96811 exposures) and i (78876 exposures).

3 Precovery pipeline

The precovery pipeline (Fig. 2) consists of four main steps. During these steps it interacts with several public web tools to collect the predicted properties of NEOs and uses the ASTROWISE system for data management, image processing, and calibration while using dedicated software applications for streak detection, astrometry, and photometry. The four steps are described next.

3.1 Preliminary spatiotemporal cross-match to observational archive (step 1)

As the first step, the possible appearance of NEOs from the provided list in exposures in ESO’s OmegaCAM archive was assessed using the Solar System Object Image Search (SSOIS)⁶ web service (Gwyn et al. 2012). We used SSOIS as follows.

First, for the NEOs, after a search by object name, the orbital elements provided by the Canadian Astronomy Data Centre (CADC) itself were used, which is a regularly updated copy of the MPC orbital element database. Second, SSOIS itself then generated an internal ephemerides database using the ORBFIT⁷ software package. Third, the SSOIS’ internal database of (RA, Dec) pointings and observation dates for ESO’s OmegaCAM observations were used. This database was updated with observations until approximately 1 November 2021⁸. Fourth, the ephemerides database in point 2 was cross-matched in the three-dimensional space of (observation date, RA, Dec) by SSOIS itself with the internal database of OmegaCAM observations in point 3. The cross-match algorithm is described in Gwyn et al. (2012). Fifth, uncertainties in input orbital parameters and image pointings are not taken into account in the cross-match procedure. This means that the cross-match only produces a result when an NEO ephemeris for the exact orbital elements specified in step one falls inside the field of view (FoV) of an OmegaCAM exposure for the exact (observation date, RA, Dec) specified in step three. Sixth, we did not ask SSOIS to refine the cross-match to the (X, Y) of a specific detector as that functionality is not available (yet) for OmegaCAM.

To automate our usage of SSOIS we implemented a scripted interface to SSOIS. This interface submits a query to SSOIS. In return, SSOIS provides a list of (raw) imaging data (exposures) for OmegaCAM, date, time, exposure time, and the observed filter. SSOIS outputs for OmegaCAM point to the main raw frames, and do not specify which extension (chip) overlaps with the predicted position of the NEO. The information returned by SSOIS on NEO names and OmegaCAM exposures was stored in a local SQLite file.

3.2 Final spatiotemporal cross-match (step 2)

In Step 2, the pipeline uses the NEO Dynamic Site (NEODyS)⁹ and JPL Horizons¹⁰ web service¹¹ to obtain more accurate predictions of the NEO positions over the duration of exposure and to obtain the predicted angular motion and visual magnitude. We supply to NEODyS/Horizons the start and end time of the observation, the name of the NEO, and the observatory identifier to specify its location. This query to NEODyS/Horizons returns the (RA, Dec), 1σ uncertainties in position, V-band magnitude and the position of the NEO at the beginning and end of the exposure. These values are used to predict the rate and direction of NEO motion relative to the astrometric reference frame.

Based on the returned apparent magnitude we make a preliminary prediction on the signal-to-noise (S/N) for the given filter and exposure time of the candidate (p)recovery observation by assuming its appearance is a point source (i.e. neglecting proper motion) and assuming a solar spectral energy distribution of the NEO. Candidate (p)recoveries for which NEODyS/Horizons confirm the overlap with the exposure FoV and that also have a predicted S/N > 1 are stored in the local SQLite database. To automate our usage of NEODyS/Horizons we implemented a scripted interface to NEODyS/Horizons, using the same approach as for SSOIS.

Fig. 2 Four-step precovery pipeline. The pipeline gets a text file containing NEO names as the input and produces a CSV table at the end containing the NEO precoveries. The intermediate outputs are stored in an SQLite database, which communicates with the pipeline in many instances throughout the process.

3.3 Data processing and astrometry (step 3)

In step 3, we apply several filters and criteria to the precovery candidates resulting from step 2 to narrow down the search to sources that are likely to be detectable. These filters are described below.

• An upper limit on the angular separation (RA, Dec) between SSOIS and NEODyS/Horizons predictions;

• An upper limit on the 1σ uncertainty in the position (RA, Dec);

• Availability of the raw data (publicly available);

• The exact position of NEOs on the camera;

• A lower limit on the predicted S/N.

The imaging data used for precovery were identified by SSOIS and were based on its predictions. However, once we refine these predictions using NEODyS/Horizons, the images may not overlap with the refined position. Therefore, if the angular separation between these predictions is larger than a certain limit (for example larger than the FoV), we might assume that the object is not within the FoV of the camera. Then, for cases where the uncertainties in position are too large, the possibility that the NEO is within the FoV is very small. The pipeline skips these frames. Additionally, given the coordinates of the object, the pipeline inspects which CCD is overlapping with the corresponding position of NEOs and if there is no overlap, the pipeline excludes these frames. In the end, cases with an S/N lower than the given limit are removed to exclude very faint and non-detectable objects.

All of these criteria are applied to both NEO samples (i.e. the risk list and the full list). Additionally, for the full list, several other filters are used: a lower limit on the 1σ uncertainty in the position, with which we can focus the precovery search on objects without precise positional information; a lower limit on the predicted length that considers the observational difficulties for detecting and matching short streaks (step 4, described later in the text) and allows the pipeline to skip frames with NEOs shorter than this limit; and objects without calibrated data for the rare cases when the pipeline fails to reduce and/or calibrate raw frames, and therefore no calibrated frame is available for further analysis (the pipeline skips these frames).

The thresholds used for these filters are given in Table 1. Once a frame passes all the filters and criteria (referred to as a precovery candidate), we then use ASTROWISE to produce astrometrically and photometrically calibrated pixel images for the candidate precovery images returned by NEODyS/Horizons.

If calibrated detector images already exist in the database, they were downloaded from ASTROWISE. If not, the raw data was ingested from the ESO archive into the ASTROWISE system as needed, and the ASTROWISE optical image pipeline was used to process that raw data. To obtain the optimal astrometry, we manually astrometrically recalibrated the detector images downloaded from ASTROWISE using SCAMP 2.10 (Bertin 2006). This version of SCAMP uses the coordinates of stars in the Gaia EDR3 catalogue as astrometric reference objects. The final astrometry was extracted from the resampled and background subtracted pixels using SWARP (Bertin et al. 2002). We note that the majority of OmegaCAM data residing in ASTROWISE have been astrometrically calibrated using the near-IR 2MASS catalogue (Skrutskie et al. 2006) as the astrometric reference catalogue.

To improve the astrometric solution of our dataset and to achieve better astrometry of the precovered NEOs, we implemented astrometric calibration using the Gaia EDR3 catalogue (Gaia Collaboration 2021) as the astrometric reference. It allows us to (re)calibrate OmegaCAM images astrometrically to Gaia. It improves the external astrometric accuracy from a typical RMS 0.3″ to RMS 0.04″(a factor of 7 improvement). This accuracy in astrometric calibration brings down the RMS of the external astrometric residuals below the size of most observed offsets between predicted and precovered astrometry.

3.4 Streak setection and association (step 4)

In step 4, the pipeline deploys STREAKDET (Virtanen et al. 2016; Pöntinen et al. 2020) on the photometrically and astrometrically calibrated images to extract candidate streaks. STREAKDET is a package that was developed to identify streaks in ground-based and space-based imaging data. STREAKDET detects streaks in the images in four steps: segmentation of images, detection of streak-like objects, classification of objects to separate streaks from other astronomical objects, and derivation of their parameters (such as coordinates, total flux and length) by fitting a model. For the precovery pipeline, we optimized the STREAKDET configuration parameters for OmegaCAM.

Once streak detection is done, then the pipeline searches for the best match between the streaks found within the 3σ uncertainty ellipse of NEOs and their predicted properties. The predicted properties used include the NEO length, the direction of motion, and the magnitude. This procedure is illustrated in Fig. 3. The matching procedure (alternatively called ‘association’) is done in two stages.

In the first step, the pipeline filters out (excludes) the detected streaks with properties different from the predicted properties of the NEOs. The applied filters are

$$\begin{array}{l}-2\text{\hspace{0.17em}mag\hspace{0.17em}\hspace{0.17em}<\hspace{0.17em}\hspace{0.17em}}{m}_{\text{det,\hspace{0.17em}streak}}-m_{\text{pred}}<2\text{\hspace{0.17em}mag,}\hfill \\ \dots \text{\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}\hspace{0.17em}}0.5\text{\hspace{0.17em}\hspace{0.17em}<\hspace{0.17em}\hspace{0.17em}}{l}_{\text{det,\hspace{0.17em}streak}}/l_{\text{pred}}<2,\hfill \\ \dots \text{\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}\hspace{0.17em}}0.5\text{\hspace{0.17em}\hspace{0.17em}<\hspace{0.17em}\hspace{0.17em}}{\sigma }_{\text{det,\hspace{0.17em}streak}}/{\sigma }_{\text{pred}}<1.5,\hfill \\ \dots \text{\hspace{0.17em}\hspace{0.17em}and\hspace{0.17em}\hspace{0.17em}}(-30°<{\theta }_{\text{det,\hspace{0.17em}streak}}-{\theta }_{\text{pred}}<30°,\hfill \\ \dots \text{\hspace{0.17em}\hspace{0.17em}or\hspace{0.17em}\hspace{0.17em}}150°<{\theta }_{\text{det,\hspace{0.17em}streak}}-{\theta }_{\text{pred}}<210°),\hfill \end{array}$$

where m_{det, streak}, l_{det, streak}, σ_{det, streak}, and θ_{det, streak} are respectively the observed apparent magnitude (in the given filter), length, width, and position angle of motion of the detected streak, while m_pred, σ_pred and θ_pred are the same properties but predicted for the NEO.

In the second step, for the detected streaks that satisfy the first step (if any), the pipeline estimates a difference score, which is the standardized Euclidean distance between the observed properties of a streak and the predicted properties of the NEO. The parameters used for measuring the distance are the angular length and the position angle of motion. These parameters are normalized by the observed length and the seeing (FWHM) during observations as follows¹²

$$\begin{array}{l}{\theta }_{\text{det,\hspace{0.17em}streak,\hspace{0.17em}norm}}={\theta }_{\text{det,\hspace{0.17em}streak}}/Arctan\left(FWHM/l_{\text{det,\hspace{0.17em}streak}}\right)\hfill \\ l_{\text{det,\hspace{0.17em}streak,\hspace{0.17em}norm}}=l_{\text{det,\hspace{0.17em}streak}}/\text{FWHM}\hfill \end{array}$$

where θ_{det, streak, norm} and l_{det, streak, norm} are the normalised position angle of motion and length of the detected streak. Correspondingly, the difference score is estimated using

$$score=\sqrt{{\theta }_{\det \text{,\hspace{0.17em}streak,\hspace{0.17em}norm}}{}^2+l_{\det \text{,\hspace{0.17em}streak,\hspace{0.17em}norm}}{}^2}.$$

In the end, the detected streak with the lowest difference score is picked as the best match (indicated by the green circle in the fourth column of Fig. 3). Finally, those frames with a matched streak are visually inspected to check whether it is a true precovery.

4 Results

This section presents the outcome of NEO precovery and discusses the performance of the precovery pipeline for the detection, astrometry and photometry of NEOs.

4.1 Accuracy of astrometry and photometry

The output catalogues of STREAKDET provide the coordinates (X,Y) of the centre and the total flux of the detected streaks. However, they do not provide uncertainties of these measurements. To estimate the centroiding accuracy of STREAKDET, we ran the pipeline for NEOs with predicted positional uncertainty of less than 0.02″(1σ uncertainty ellipse). This is about 10 times smaller than the pixel size of the instrument. The offset between the predicted coordinates of these NEOs (RA, Dec) and the derived coordinates using STREAKDET gives an upper limit of the total astrometric accuracy (i.e. the aggregate of calibration and centroiding accuracies). The average value of this offset, as shown in Fig. 4, is about ~0.12″.

A preliminary estimate of the astrometric calibration accuracy is ~0.05″. It is reasonable to assume that the centroiding and calibration astrometric errors are independent because (i) the centroiding accuracy of the bright Gaia stars is likely ≪0.05″and (ii) the same or at least very similar ensemble of Gaia stars are used for the astrometric calibration of the exposures for one apparition of an NEO. We note that the absolute astrometric accuracy of stars in the Gaia EDR3 is ≪0.05″. Thus, we draw the preliminary conclusion that the total absolute astrometric accuracy of our NEO positions is typically ~0.12″. This means offsets of >0.2″ (i.e. the angular scale of 1 OmegaCAM pixel) between extracted and predicted positions are at typically 1.5σ significance before taking prediction uncertainties into account in the significance calculation. In individual cases the accuracy can be worse (see notes on individual recoveries below). Moreover, we also assessed the impact of resampling in step 3 of the pipeline (using SWARP) in the astrometry and found a deviation of less than 0.01 ″ (Fig. 5).

In the absence of any indication of photometric accuracy of STREAKDET, we made an estimation using the output catalogue of SEXTRACTOR. The uncertainty in the measured magnitudes of streaks (within their magnitude range) is on average 0.1 mag.

Fig. 3 Results of the streak detection using STREAKDET for three examples. From left to right: the first column shows the calibrated cutout around the predicted position of NEOs. The second column shows the 3σ uncertainty ellipse (in red) and the predicted appearance of the NEO considering its speed and direction of motion (red line). The third column shows the detected streaks across frames by STREAKDET (grey circles). The fourth column shows the streaks that match the predicted properties of the NEOs (green circles).

Fig. 4 Upper limit for the centroid estimation accuracy of STREAKDET in a range of magnitude, length, S/N, and seeing. The astrometric accuracy is within 0.12″ and does not show any dependence on the observed properties of NEOs.

4.2 Precovery of NEOs in the ESA risk list

Table 2 summarizes the down selection of precovery candidates during the deployment of this pipeline on the NEOs in ESA’s risk list in combination with ESO’s OmegaCAM archive. This archive encompasses about 10 yr of OmegaCAM exposures with a total number of exposures of the order of 400 000.

The query to SSOIS returned 10 345 exposures (i.e. candidate precoveries for about 1 in 40 exposures). Unfortunately, for 96% (9904 exposures) the upper limit to the predicted S/N of the appearance was < 1, leaving only 441 candidate precoveries. Subsequently, NEODyS made a refined positional prediction for these 441, which in 33% of the cases was >1° away from the centre of the exposure (i.e. the RA, Dec of the pointing), which itself is 1°× 1°.

We assumed that this made it too unlikely that the NEO appearance would be inside the exposure, and removed them from the candidate list. This left 295 candidates. For 15% of these candidates the 1σ semi-major axis of the ellipse was >1°, and thus again we deemed it too unlikely for now that the NEO appearance would be inside the exposure. This left us with 251 precovery candidate exposures. For 15 of these the data could not be retrieved in ASTROWISE: 8 were found to belong to other OmegaCAM surveys (we are in the process of obtaining access) and the remaining 7 have not yet been publicly released by ESO. For 28% of the remaining candidates (236 precovery candidates) the actual predicted position of the NEO landed outside the pixels of the detectors (e.g. in gaps or just outside the FoV), and given the typically small size of the error ellipse we deemed it too unlikely that a precovery could be made. This left 170 precovery candidates. At this step, we refined the S/N estimate by taking into account the proper motion. For 60% of the 170 candidates the predicted S/N was then less than 2, and we removed them from the list of candidate precoveries. This left 68 precovery candidates on which precovery was attempted with STREAKDET. If STREAKDET failed, manual precovery was attempted. This was successful for 57% of the cases, so 49 precoveries (listed in Table 3), with 26 being successful via STREAKDET and 23 being successful via subsequent manual analysis. The sky distribution and image thumbnails of these 49 precoveries are shown in Figs. 6 and 7. Figure 8 shows the predicted properties of detected and non-detected cases. As is seen in this figure, for NEOs with a predicted length smaller than 2″, we could only detect NEOs by visually examining several frames available. These objects would be missed completely by the pipeline.

Fig. 5 Offset between obtained RA and Dec of all the sources extracted (using SEXTRACTOR) in frames with and without resampling and background subtraction (using SWARP) in step 3 of the pipeline

Fig. 6 Sky coverage of the OmegaCAM/VST observations (green area), the 10 345 frames resulting in a chance of overlapping with an NEO resulting from SSOIS (blue points), 19 precovery candidates without a detection (red points), and 49 precovered NEOs (black points). Of the 49 precoveries, 27 were identified using the pipeline and 22 by visual inspection after step 3 of the pipeline. The close sky proximity of observations at the resolution of the figure means that multiple observations can appear to be a single data point.

4.3 Precovery of NEOs in the full list

In Table 4, we summarize the results from precovery of the full list of NEOs. Queries to SSOIS led to identifying 186 476 frames. In the subsequent step, the query to Horizons removed more than two-thirds of the frames, due to the low S/N (S/N < 1) of the NEOs, which left 55 692 cases. Other filters were applied and in each step, a fraction of the precovery candidates were removed. The filters applied to the full list of NEOs were modified slightly to avoid the extra computation time for cases where the pipeline most likely does not detect any NEOs. For the full list of NEOs, we applied an extra filter on the predicted length of NEOs and did not attempt recovery of objects with predicted lengths smaller than 3″. These objects, as was seen earlier, are the trickiest to detect (using STREAKDET), and they have a similar appearance to other sources (e.g. extragalactic objects and blended objects), which makes the matching process harder. Moreover, a lower limit on the 1σ uncertainties in position was introduced to focus our search on objects with a less certain positional prediction.

After applying all these filters, 968 cases remained, about 0.5% of the initial number of frames (from SSOIS). Out of 968, after visual inspection of the results, the pipeline successfully detected an NEO in 196 cases (listed in Table 5), about one-fifth of the total cases. Out of these 196 NEO appearances, 114 are true precoveries. Figure 9 shows the histogram of the time intervals for these true precoveries, which ranges between 1 day and 3653 days (for 2021 WO), with the majority identified between 1 and 200 days before the discovery date.

The pipeline also resulted in 157 false positives (about 44% of automatic precoveries), of which 75% have a predicted S/N < 5. Additionally, for long NEOs, the precovery rate decreased because of the limitations in streak detection: STREAKDET segments long streaks (low and high S/N) as two separate streaks, and the resulting positional information is normally not valid. Therefore, in these cases, while the pipeline produced a precovery that almost matches that of the NEO, we do not take it into account as a successful precovery.

The bottom graph in Fig. 8 shows the predicted S/N and length of the detected and undetected precovery candidates. Out of 772 cases without a precovery, 311 have a predicted S /N < 5 (Fig. 10), and 94 have a 01σ positional uncertainty larger than 200″ (Fig. 11). The precovery rate of 20% does not mean that the pipeline cannot identify 80% of the precovery candidates. Visual inspection shows that for the majority of cases without a recovery, no NEO is visible in the frames.

To examine the possible complementary role of other existing tools for streak detection, we searched for streaks using SEXTRACTOR (Bertin & Arnouts 1996) for the cases without a true detection. SEXTRACTOR could identify an extra 26 cases. However, when only SEXTRACTOR is used, the number of detection drops to only 58 (compared to 196 for STREAKDET), which shows that SEXTRACTOR cannot substitute STREAKDET. SEXTRACTOR should be used with caution, considering its limitation to properly segment the streaks, which would result in an inaccurate estimate of the centroid of streaks (within a 01–2″). The uncertainties of SEXTRACTOR centroiding (ERRX_WORLD and ERRY_WORLD parameters in the output tables) is on average 0.88″, in the same range as the FWHM of the images.

To explore further the accuracy of the predicted properties of NEOs, in Fig. 12, we compare the predicted and observed (measured from data) length (angular size) and apparent magnitude of the precovered NEOs. There is a general agreement between the predicted and observed properties, in particular for the shorter streaks, while the observed NEOs are on average about 0.5 mag fainter. We explore this magnitude offset further in Fig. 13 where the predicted and observed lengths and magnitudes of the pr-ecovered NEOs are shown. While there is no clear pattern in the magnitude offset between filters, the predicted and observed magnitudes in u and g show larger offsets. There are two possible sources for this offset: overprediction of the magnitudes of NEOs and the simplistic assumption on the colour of NEOs (solar colour) for magnitude transformation between the V band and the observed filters.

Fig. 7 Precovered risk list NEOs (49) found using the precovery pipeline: 26 precoveries were detected (fully) automatically and 23 were detected by visual inspection of the frames.

Fig. 8 Predicted S/N and predicted length for precovered NEOs. Left: NEOs of the risk list, precovered automatically using the pipeline (red points) or manually by visual inspection of frames (yellow points), and non-precovered risk list NEOs in the OmegaCAM data (grey points). Right: NEOs of the full list, precovered automatically using the pipeline and STREAKDET (red points) and SEXTRACTOR (blue points), and non-precovered full list NEOs in the OmegaCAM data (grey points).

Fig. 9 Time interval between precovery and discovery of the 196 NEO appearances (NEO full list). The upper and lower panels show the NEO appearances after the discovery (recovery) and before the discovery (true precovery) of the NEOs, respectively.

Fig. 10 Histogram of the S/N distribution of the 968 precovery candidates after detectability filtering (in step 3) in u, ɡ, r, and i.

Fig. 11 Histogram of the 1σ positional uncertainties of NEO apperances. Left: positional uncertainties of the 968 precovery candidates after step 3 of the pipeline. Right: positional uncertainties for the successful NEO precoveries by the pipeline.

Fig. 12 Predicted vs. observed properties of the precovered NEOs for the full list of NEOs. Left: predicted vs. observed apparent magnitude of the precovered NEOs by STREAKDET (red points) and SEXTRACTOR (blue points). Right: predicted vs. observed length of the precovered NEOs by STREAKDET (red points) and SEXTRACTOR (blue points).

Fig. 13 Same as Fig. 12, but for precoveries done by STREAKDET in each filter (ugri).

5 Conclusions and outlook

In the Introduction of this paper we outlined our project objectives, and to quantify the results from our exploratory study we defined our main objective to answer six questions. In this section, we answer these questions and add the outlook on possible next steps.

• What fraction of detectable NEOs can we precover? And how automated can we make the precovery? The detectability and the precovery rates vary as a function of the chosen threshold of S/N. The detectability rate is estimated to be ~0.05 per NEO at an S/N higher than 3 for NEOs on the risk list and for the full list of NEOs. The precovery rate for S/N > 3 is 40% for NEOs on the risk list and 20% for the full list of NEOs. The precovery rate increases to about 50% for S/N > 10. In other words, currently the majority of NEOs with a predicted 3 < S/N < 10 remain undetected, even after visual inspection. Assessing whether the failed precoveries are consistent with predicted errors on the predicted locations and brightnesses is an important next step. This step is beyond the scope of this paper. If inconsistent with predictions, it might give new insight into the limitations of those predictions or insights into how to improve the detection– precovery process.

• What astrometric and photometric accuracy can be achieved? The astrometric and photometric accuracies are 0.12″ (15% of the average FWHM of OmegaCAM/VST images of about 0.8″) and 0.1 mag. Improvements in astrometric accuracy are expected from propagating the proper motions in the Gaia astrometric reference catalogue to the observation date of the science image. Improvements in photometric accuracy can come from more sophisticated modelling of spectral energy distribution (SED), observational configuration, and NEO shape modelling. The Solar System Open Database Network (Berthier et al. 2023) might facilitate this.

• What level of automation can be achieved in the precovery workflow? The precovery pipeline described here works automatically through all the steps. Thanks to the common data model for calibrated observations in ASTROWISE (McFarland et al. 2013) it can be deployed straightforwardly on calibrated observations for many other instruments available in the ASTROWISE archive. However, after the last step (streak detection and matching) and before reporting the recoveries, they must be inspected by an expert to confirm or reject the precoveries. Another challenge is that precise photometric calibration for a range of instruments is hard to fully automate because the derivation of the solution uses reference stars sometimes inside the science images and sometimes in calibration observations. A potential solution would be to construct a photometric reference catalogue that spans the entire sky observable by OmegaCAM with sufficient stellar density. This appears possible by aggregating information from the multiple large-scale surveys of the southern sky.

• What are the major challenges to exploiting various astronomical surveys for NEO space safety purposes? In addition to the automation already discussed, a main challenge is robust NEO detection and segmentation. This is also the main reason behind the limitations of precovery in this paper. STREAKDET is a great tool for detecting high S/N streaks with sizes between 5″ and 20″. However, its performance drops for faint and long streaks. Deep learning might be a solution to improve streak detection and ultimately NEO precovery.

• What value do these precoveries have for planetary defence? The precovery of NEOs provides valuable positional information about NEOs before their discovery. In this paper we could precover three NEOs, and as a result these three NEOs were removed from the risk list of NEOs.

• Do recoveries have benefits for NEO science? Precision astrometric information of a sizeable ensemble of recovered NEOs can provide a better understanding of non-gravitational effects on NEO orbits (e.g. the Yarkovsky effect). We also note that performing additional serendipitous discovery in astronomical surveys, especially at high ecliptic latitudes, might provide interesting constraints on the orbital demography of NEOs, complementary to NEO dedicated surveys. Collecting the optical– near-infrared SED information from the recovery of a sizeable NEO sample can be complementary to existing astrometric and SED information and hence provide valuable information about their orbit and composition, shapes, and rotational properties.

This exploratory study has shown that it is possible to adapt ASTROWISE to contribute to NEO precovery, and that it achieves the required astrometric accuracies. The current study indicates two major advantages. First, ASTROWISE allows a systematic precovery process that scales up to the Petabyte regime. At face value it appears that this is complementary to current excellent manual operations in that it could allow us to obtain risk assessment information also from null results on attempted precovery. The second major advantage is that it makes it feasible to improve efficiency or large data set astrometric accuracy and photometry, when improved calibration methods, improved insights in survey instrument behaviour, or improved reference catalogue data become available.

Acknowledgments

We would like to thank the referee for their comments and suggestions to improve the quality of this work. We are happy to acknowledge and are grateful to Angela Maria Raj, Katya Frantseva, Migo Müller, Stephen Gwyn, Ylse de Vries, Konrad Kuijken, Danny Boxhoorn, Willem-Jan Vriend, KiDS DR5 production team, Michiel Rodenhuis, Thomas Wijnen, Edwin Valentijn and the Kapteyn institute. This work was executed as part of ESA contract no. 4000134667/21/D/MRP (CARMEN) with their Planetary Defence Office. The Big Data Layer of the Target Field Lab project “Mining Big Data” was used. The Target Field Lab is supported by the Northern Netherlands Alliance (SNN) and is financially supported by the European Regional Development Fund. The data science software system ASTROWISE runs on powerful databases and computing clusters at the Donald Smits Center of the University of Groningen and is supported, among other parties, by NOVA (the Dutch Research School for Astronomy). TSR acknowledges funding from the NEO-MAPP project (H2020-EU-2-1-6/870377). This work was (partially) supported by the Spanish MICIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe” by the “European Union” through grant PID2021-122842OB-C21, and the Institute of Cosmos Sciences University of Barcelona (ICCUB, Unidad de Excelencia “María de Maeztu”) through grant CEX2019-000918-M. This research has made use of the Aladin sky atlas (Bonnarel et al. 2000; Boch & Fernique 2014) developed at CDS, Strasbourg Observatory, France and SAOIm-ageDS9 (Joye & Mandel 2003). This work has been done using the following software, packages and PYTHON libraries: Astro-WISE (Begeman et al. 2013; McFarland et al. 2013), NUMPY (van der Walt et al. 2011), SCIPY (Virtanen et al. 2020), ASTROPY (Astropy Collaboration 2018).

References

All Tables

All Figures

	Fig. 1 An overview of a decade of observations with OmegaCAM/VST. Top: the sky coverage of the OmegaCAM/VST observations over 10 yr of observations of the southern hemisphere. The ASTRO-WISE archive contains 361977 individual exposures. Bottom: the histogram of exposure times of the OmegaCAM/VST observations in u (54918 exposures), ɡ (79127 exposures), r (96811 exposures) and i (78876 exposures).
In the text

	Fig. 2 Four-step precovery pipeline. The pipeline gets a text file containing NEO names as the input and produces a CSV table at the end containing the NEO precoveries. The intermediate outputs are stored in an SQLite database, which communicates with the pipeline in many instances throughout the process.
In the text

Fig. 3 Results of the streak detection using STREAKDET for three examples. From left to right: the first column shows the calibrated cutout around the predicted position of NEOs. The second column shows the 3σ uncertainty ellipse (in red) and the predicted appearance of the NEO considering its speed and direction of motion (red line). The third column shows the detected streaks across frames by STREAKDET (grey circles). The fourth column shows the streaks that match the predicted properties of the NEOs (green circles).

In the text

	Fig. 4 Upper limit for the centroid estimation accuracy of STREAKDET in a range of magnitude, length, S/N, and seeing. The astrometric accuracy is within 0.12″ and does not show any dependence on the observed properties of NEOs.
In the text

	Fig. 5 Offset between obtained RA and Dec of all the sources extracted (using SEXTRACTOR) in frames with and without resampling and background subtraction (using SWARP) in step 3 of the pipeline
In the text

Fig. 6 Sky coverage of the OmegaCAM/VST observations (green area), the 10 345 frames resulting in a chance of overlapping with an NEO resulting from SSOIS (blue points), 19 precovery candidates without a detection (red points), and 49 precovered NEOs (black points). Of the 49 precoveries, 27 were identified using the pipeline and 22 by visual inspection after step 3 of the pipeline. The close sky proximity of observations at the resolution of the figure means that multiple observations can appear to be a single data point.

In the text

	Fig. 7 Precovered risk list NEOs (49) found using the precovery pipeline: 26 precoveries were detected (fully) automatically and 23 were detected by visual inspection of the frames.
In the text

Fig. 8 Predicted S/N and predicted length for precovered NEOs. Left: NEOs of the risk list, precovered automatically using the pipeline (red points) or manually by visual inspection of frames (yellow points), and non-precovered risk list NEOs in the OmegaCAM data (grey points). Right: NEOs of the full list, precovered automatically using the pipeline and STREAKDET (red points) and SEXTRACTOR (blue points), and non-precovered full list NEOs in the OmegaCAM data (grey points).

In the text

	Fig. 9 Time interval between precovery and discovery of the 196 NEO appearances (NEO full list). The upper and lower panels show the NEO appearances after the discovery (recovery) and before the discovery (true precovery) of the NEOs, respectively.
In the text

	Fig. 10 Histogram of the S/N distribution of the 968 precovery candidates after detectability filtering (in step 3) in u, ɡ, r, and i.
In the text

	Fig. 11 Histogram of the 1σ positional uncertainties of NEO apperances. Left: positional uncertainties of the 968 precovery candidates after step 3 of the pipeline. Right: positional uncertainties for the successful NEO precoveries by the pipeline.
In the text

	Fig. 12 Predicted vs. observed properties of the precovered NEOs for the full list of NEOs. Left: predicted vs. observed apparent magnitude of the precovered NEOs by STREAKDET (red points) and SEXTRACTOR (blue points). Right: predicted vs. observed length of the precovered NEOs by STREAKDET (red points) and SEXTRACTOR (blue points).
In the text

	Fig. 13 Same as Fig. 12, but for precoveries done by STREAKDET in each filter (ugri).
In the text