© ESO, 2012

1. Introduction

Understanding planetary systems formation and evolution has become one of the biggest challenges of astronomy, since the imaging of a debris disk around β Pictoris in the 80’s (Smith & Terrile 1984) and the discovery of the first exoplanet around the solar-like star 51 Pegasi during the 90’s (Mayor & Queloz 1995). Today, about 25 debris disks have been imaged at optical, infrared, or submillimetric wavelengths¹. Debris disks trace stages of system evolution where solid bodies with sizes significantly larger than the primordial dust size (larger than meters or km sized) are present to account for through collisions, the presence of short lived dust. They are thought to be privileged places to search for planets. This is particularly true for those showing peculiar structures (e.g. rings with sharp edges) or asymmetries, spirals even though other physical effects not involving planets could also lead to the formation of similar structures. Takeuchi & Artymowicz (2001) for instance showed that relatively small amounts (typ. 1-a few Earth masses) of gas can shape the dust disk through gas-dust interactions into rings (see below). It is remarkable however that all the stars around which relatively close (separations less than 120 AU) planets have been imaged are surrounded by debris disks: a  ≤ 3-M_Jup planetary companion was detected in the outskirts of Fomalhaut’s debris disk (119 AU from the star; Kalas et al. 2008), four planetary companions of 7–10 M_Jup were imaged at 15, 24, 38 and 68 AU (projected separations) from HR 8799 (Marois et al. 2008; Marois et al. 2010). Using VLT/NaCo L′-band saturated images, we detected a 9 ± 3 M_Jup planet in the disk of β Pictoris ( ≃ 12 Myr) with an orbital radius of 8–12 AU from the star (Lagrange et al. 2010; Chauvin et al. 2012). More recent studies at Ks show that β Pic b is located in the inclined part of the disk (Lagrange et al. 2012), conforting the link between disk morphology and the presence of a planet (Mouillet et al. 1997; Augereau et al. 2001). β Pic b also confirms that giant planets form in a timescale of 10 Myr or less. Interestingly, β Pic b and maybe also HR 8799e could have formed in situ via core accretion, in contrast with the other, more remote, young companions detected with high-contrast imaging. If formed in situ, the latter probably formed through gravitational instabilities within a disk, or through the fragmentation and collapse of a molecular cloud.

There are many exciting questions regarding disks and planets: could different planet formation processes be at work within a given disk? Disks and planets are known to exist in binary systems; (how) do massive companions impact on the dynamical evolution of inner planets and disks? In a recent study, Rodriguez & Zuckerman (2012) showed that for a binary system to have a disk, it must either be a very wide binary system with disk particles orbiting a single star or a small separation binary with a circumbinary disk. Such results can help the search for planets if one relates debris disks with planet formation. However, another question, already mentioned, is to which extent and how can debris disks indicate the presence of already formed planets?

A particularly interesting system in the present context is HR 4796A, consisting of an early-type (A0), young, close-by star (see Table 1) surrounded by dust, identified in the early 90’s (Jura 1991) and resolved by Koerner et al. (1998) and Jayawardhana et al. (1998) at mid-IR from the ground, and at near-IR with NICMOS on the Hubble space telescope (HST) (Schneider et al. 1999) as well as from the ground, coupling coronagraphy with adaptive optics (Augereau et al. 1999). The resolved dust shapes as a narrow ring, with steep inner and outer edges. The steepness of the inner edge of the dust ring has been tentatively attributed to an unseen planet (Wyatt et al. 1999); however, none has been detected so far. The disk images + SED modeling required at least two populations of grains, one, narrow (a few tens AU) cold ring, located at  ≃ 70 AU, and a second one, hotter and much closer to the star (Augereau et al. 1999), although the existence of an exozodiacal dust component is debated (Li & Lunine 2003). Wahhaj et al. (2005), argued that in addition to the dust responsible for the ring-like structure observed at optical/near IR wavelengths, a wider, low-density component should be present at similar separations to account for the thermal IR images. Recently, higher quality (SN/angular resolution) data were obtained with STIS (Schneider et al. 2009) and from the ground (Thalmann et al. 2011), the latter using performant AO system on a 8 m class telescope, as well as Angular Differential Imaging (ADI, see below). With the revised distance of HR 4796 with respect to earlier results, the ring radius is at about 79 AU, and has a width of 13 AU (STIS 0.2–1 μm data). Furthermore, these authors show a 1.2–1.6 AU physical shift of the projected center of the disk wrt the star position along the major axis and Thalmann et al. (2011) moreover measures a 0.5 AU shift along the disk minor axis. Finally, and very interestingly in the context of planetary system formation, HR 4796 has also a close-by (7.7 arcsec, i.e. about 560 AU projected separation), M-type companion (Jura et al. 1993), plus a tertiary one, located much further away (13 500 AU in projection; Kastner et al. 2008). The closest companion may have played a role in the outer truncature of the disk, even though, according to Thebault et al. (2010), it alone cannot account for the sharp outer edge of HR 4796 as observed by STIS.

ADI (Marois et al. 2006) is a technics that has proved to be very efficient in reaching very high contrast from the ground on point-like objects. It has also been used to image disks around HD 61005 (the Moth disk, Buenzli et al. 2010), HD 32297 (Boccaletti et al. 2012), and β Pic (Lagrange et al. 2012), but it should be used with care when observing extended structures, as the morphology of these structures may be strongly impacted by this method (Milli et al. 2012).

We present here new high contrast images of HR 4796 obtained with NaCo on the VLT at L′ band, both in ADI and Sparse Aperture Mode (SAM, Tuthill et al. 2010) aiming at exploring the disk around HR 4796A, and at searching for possible companions around HR 4796A as well as around HR 4796B. SAM and ADI are complementary as the first give access to regions in the 40–400 mas range from the star, and ADI further than typically 300 mas. Pedagogical examples of SAM performances and results are reported in Lacour et al. (2011).

2. Data log and reduction

2.1. Log of observations

VLT/NaCo (Lenzen et al. 2003; Rousset et al. 2003) L′-band data were obtained on April, 6 and 7th, 2010, in ADI mode and with SAM. We used the L27 camera which provides a pixel scale of  ≃ 27 mas/pixel. The log of observations is given in Table 2.

The precise detector plate scale as well as detector orientation were measured on an θ 1 Ori C field observed during the same run, and with HST (McCaughrean & Stauffer 1994 (with the same set of stars TCC058, 057, 054, 034 and 026). We found a platescale of 27.10 ± 0.04 mas per pixel and the North orientation offset was measured to be  −0.26 ± 0.09° if we do not consider a systematics in the North position, or  − 0.26 ± 0.3° otherwise (see Lagrange et al. 2012, for a detailed discussion on the absolute uncertainty on the detector orientation).

2.1.1. ADI data

The principle of ADI imaging is given in Marois et al. 2006 (see also Lafrenière et al. 2007). Here, a typical ADI sequence consisted in getting sets of saturated images (datacubes of NDIT images) at different positions on the detector, followed and precedented by a series of un-saturated point spread function (PSF) images recorded with a neutral density filter (ND Long). These unsaturated images are used to get an estimate of the PSF shape for calibration purposes (photometry, shape), and fake planet simulation. On April 5/6, a few tests were made with different offsets patterns (star centered on either 2 or 4 positions on the detector) so as to test the impact of HR 4796B (which rotates on the detector during the ADI observations) on the final image quality². This is important as the field of view (FoV) rotation was fast. On April 6/7, the saturated images were recorded with two offsets corresponding to two opposite quadrants on the detector. Both nights, the atmospheric conditions were good on average, but variable, and the amplitude of field rotation was larger than 80° (see Table 2). The comparison between the PSFs taken prior and after the saturated images does not reveal strong variations.

2.1.2. SAM data

Sparse aperture masking is obtained on NaCo by insertion of a mask in the cold pupil wheel of the camera (Lacour et al. 2011). The mask acts as a Fizeau interferometer. It forms in the focal plane of the camera interference fringes which are used to recover the complex visibilities of the astronomical object. Of the four available masks (Tuthill et al. 2010), we used the 7 holes mask which gives the highest throughput (16%). It is made of 1.2 m wide circular apertures (scaled on M1) positioned in a non-redundant fashion over the pupil. Minimum and maximum baseline lengths are respectively 1.8 and 6.5 m. Each mask offered by SAM can be used in addition to almost all the spectral filters offered by Conica.

The principle of SAM is based on its ability to facilitate the deconvolution of phase aberrations. Phase errors are introduced by i) atmospheric residuals and ii) instrumental aberrations (also called non-common path errors). We used integration times of the order of the typical coherence time of the phase errors. It permits a partial deconvolution of the remaining atmospheric perturbation not corrected by the AO. But most importantly, it gives an excellent correction of the slowly changing instrumental aberrations. This later point is the important factor which makes aperture masking competitive with respect to full aperture AO.

In practice, L′ SAM data on HR 4796 were obtained on June 2011. The adopted DIT was 100 ms, equivalent to a few τ₀ in the L′ band. Each set of observation consists in 8 ditherings of the telescope to produce 8 datacubes of 500 frames on 8 different positions on the detector. Each dither moves the star by 6 arcsec in X or in Y on the windowed detector (512 by 512 pixels, equivalent to 14 arcsec on sky). After 8 dithers, the telescope is offset to the K giant star HD 110036 for calibration, where the very same observation template is repeated. Four star-calibrator pairs were obtained totalizing 64 datacubes, requiring a total observation time of 2 h (including overheads). Over this time, the object has rotated by 50 degrees (the variation of the parallactic angle).

2.2. Data reduction

2.2.1. ADI data

Each individual ADI image was bad pixel-corrected and flat-fielded as usual. Background subtraction was made for each cube using the closest data cubes with the star at a different offset. Data selection was also made, within each data cube and also for each data cube. Recentering of the images was done using the offsets measured by Moffat fitting of the saturated PSF. The data cubes were then stacked (averaged) and then reduced with different procedures that are described in details in Lagrange et al. (2012) and reference there-in: cADI, sADI, rADI and LOCI. These procedures differ in the way the star halo is estimated and subtracted. We recall here the differences between these various procedures, as well as new ones developed to limit the disk self-subtraction in cADI and/or LOCI:

• In cADI, the PSF is taken as the mean or median of all individualrecentered ADI saturated images.

• To remove as much as possible the contribution of the disk from the PSF in the cADI images, we tested two slightly modified cADI reductions. In the first one, we start as usual, i.e. build a PSF from the median of all data, subtract this PSF to all data and rotate back the obtained residual images to align their FoV. The data are then combined (median) to get a first image of the disk. Then, to remove the disk contribution to the PSF, we rotate the disk image back to the n different FoV orientations corresponding to those of the initial images and subtract the median of these rotated disk images to the PSF. We obtain thus a PSF corrected (to first order) from the disk contribution. This disk-corrected PSF is then subtracted to the individual initial images; the individual residuals are then rotated back to be aligned and stacked (median) to get a new disk image (corresponding to one iteration). This ADI procedure is referred to as cADI-disk. In the second one, we mask the disk region in each file when used to compute the reference³. We will call this method mcADI. This method will be described in details in a forthcoming paper (Milli et al. 2012).

• The rADI procedure (identical to Marois et al. 2006, ADI) selects for each frame a given number of images that were recorded at parallactic angles separated by a given value in FWHM (the same value in FWHM for each separation), to build a PSF to be subtracted from the image considered.

• In the LOCI approach, for each given image and at each given location in the image, we compute “parts” of “PSFs”, using linear combinations of all available data, with coefficients that allow to minimize the residuals in a given portion of the image.

• To limit the impact of the disk self-subtraction on the LOCI images, we also modified our LOCI approach, masking the disk in each file whenever the disk appears in the optimization zone (see Milli et al. 2012). We will call this method mLOCI.

The parameters used for the rADI and LOCI procedures are the following:

• LOCI/mLOCI Δr = 1.4 ×    FWHM below 1.6′′ and 5.6  ×    FWHM beyond (radial extent of the subtraction zones); g = 1 or 0.5 (radial to azimuthal width ratio), N_A = 300; separation criteria 1 ×    FWHM.

• rADI: separation criteria: 1.5 ×    FWHM; number of images used to compute each “PSF”: 20, Δr = 1.4 ×    FWHM below 1.6 arcsec and 5.6  ×    FWHM beyond (radial extent of the psf reconstruction zones).

For comparison purposes, we also performed a zero-order reduction (hereafter referred to as “nADI”) which consists in, for each image, 1) computing an azimuthal average of the image (with the star position as the center of the image); we get then a 1-D profile, 2) circularizing this 1-D profile to get a 2-D image centered on the star position, 3) subtracting the obtained image to the initial image to get corrected image. We then derotate and stack all the “corrected” images. nADI clearly does not benefit from the pupil stabilization and is not to be considered as a real ADI reduction procedure, but can help in some cases disentangling artefacts produced by ADI reductions from real features.

The data obtained on the 6th and 7th were reduced separately and then averaged. As they happened to have similar signal-to-noise ratio (S/N), a simple averaging was made.

2.2.2. SAM data

The first step to reduce the SAM data is to clean the frames. This can be done in the same way as any classical imaging method in the infrared. In practice, we flatfielded the data and subtracted the background. The background was estimated by taking the median value of the 8 datacubes of a single observation set.

As any interferometric facility, the observable parameters of SAM are fringes. The information lays in the contrast (which, once normalized, is called visibility) and the phase. Contrasts and phases are obtained by least square fitting of the diffraction pattern. Since the fitting of sinusoidal curves is a linear least square problem, a downhill algorithm to find the maximum likelihood was not required. Instead, inversion was done by projection of the datacubes on a parameter space defined by each complex visibility fringes. The matrix used for projection is determined by singular value decomposition of a model of the fringes. In the end, we checked that it gives exactly the same result as a least square minimization algorithm of the kind of conjugate gradient (but much faster). The fringes are modeled by cosines of given frequency multiplied by the PSF of the Airy pattern of a single hole. Wavelength bandwith is accounted for by smearing the pattern over the filter bandpass.

As a result, we get a single complex value for each baseline and each frame. They are used to compute the bispectrum, which is summed over the 8 datacubes which correspond to a single acquisition. Then, the closure phases are obtained by taking the argument of the bispectrum. One set of closure phase is obtained for an observation set which takes around 8 min. Over that time, the parrallactic angle changes less than 6 degrees, which effect is neglected (baselines rotation during an observation set is not accounted for). The final step consists in calibrating the closure phase of HR 4796 by subtracting the corresponding values obtained for the red giant (HD 110036).

3. Data simulations

Obviously, ADI affects the resulting disk shape because of disk self-subtraction. This effect is expected to be more important as the disk inclination with respect to line-of-sight decreases. Also, the different ADI reduction procedures will impact differently the disk shape. A general study of the impact of ADI on disk reductions will be presented in a forthcoming paper (Milli et al. 2012). In this paper, we concentrate on the HR 4796 case and we monitor this impact using fake disks, as done in Lagrange et al. (2012).

3.1. Assumptions

To simulate the HR 4796 disk, we assumed, following Augereau et al. (1999) a radial midplane number density distribution of grains  ∝ ((. We chose r₀ = 77.5 AU, α_in = 35 to ensure a very sharp inner edge, and α_out =  −10, as assumed by Thalmann et al. (2011). The vertical distribution is given by: where the height scale is 1 AU at 77.5 AU. The disk flaring coefficient is β = 1 and the coefficient γ = 2 ensures a gaussian vertical profile. The disk is inclined by 76 degrees (a pole-on disk would have an inclination of 0 degree), and we assumed an isotropic scattering (g = 0), as Hinkley et al. (2009) polarimetric measurements indicate a low value for g (0.1–0.27). The disk was simulated using the GRaTer code (Augereau et al. 1999; Lebreton et al. 2012). It will be referred to as HR 4796SD. The ring FWHM thus obtained is 0.14′′ (before reduction) under such hypothesis. We also considered another disk, with all parameters identical to those of HR 4796SD, but with α_out =  −4; this disk (referred to as HR 4796blowoutSD) is representative of the outer density distribution that would be observed if the outer brightness distribution was dominated by grains expelled by radiation pressure as in the case of β Pic (Augereau et al. 2001).

3.2. Simulated disk images

The flux of the simulated disk is scaled so as to have the same number of ADU (at the NE ansae) as in the real disk, once both simulated and real data are reduced by cADI. When brighter disk are needed, a simple scaling factor is applied. The simulated projected disks are then injected in a datacube, at each parallactic angle, corresponding to each real data file, and are then convolved either by a theoretical PSF matching the telescope and instrument response, or the average of the real PSFs taken prior to and after the saturated images. Each image is added to each frame of the original data cube, with a 130° or 90° offset in PA with respect to the real disk, so as to minimize the overlap between both disks. The datacubes are then processed by nADI, CADI, mcADI, LOCI and mLOCI.

4. The HR 4796A disk

4.1. Disk images: qualitative view

Figure 1 shows the images obtained when combining the data obtained on the 6th and 7th of April. Images resulting from the ADI reductions described above are showed: cADI, cADI-disk, mCADI, rADI, LOCI, and mLOCI. We also show for comparison the image resulting from nADI reduction.

Fig. 1 HR 4796A disk at L′ (linear scale). The pixel scale is 27 mas/pixel. From top to bottom, the same data reduced with nADI, cADI, cADI-disk, mcADI, rADI, LOCI, and mLOCI (see text). Note that the color codes are identical for all cADI reductions reductions on the one hand, and LOCI reductions on the other hand, to enable comparisons within a given method, but different cuts are used for cADI, rADI and LOCI reductions.

The disk is clearly detected at “large” separations from the star with the nADI reduction, and is, expectedly, lost in the Airy rings closer to the star. The parts closer to the star in projection are revealed only by the real ADI reductions, and actually, the disk is more completely detected than in previously published images, in particular the west side is almost continuously detected. The masking greatly improves the image quality of the LOCI image; the impact of mcADI with respect to cADI is, expectedly, less important, even though the flux restitution is increased. Nevertheless, the dynamical range of our images is lower than that of the recently AO published images. This is because the present data are obtained at L′, with a higher Strehl ratio, whereas the previous ones were obtained at shorter wavelengths, with lower Strehl ratio, but with detectors that have much lower background levels.

The ring appears very narrow in our L′-data, barely resolved: the FWHM measured on the PSF is 4.1 pixels (0.11′′), while the ring FWHM is  ≃ 5.7 pixels (0.15′′) (NE) and resp. 5.0 pixels (0.13′′) (SW) on the cADI data. We will however see below that the ADI reduction has an impact on the observed width.

Thalmann et al. (2011) report a relative enhancement of the disk brightness in the outer part of the ring, along the semi-major axis, that they describe as streamers emerging from the ansae of the HR 4796 disk. Our images do not show such strong features. To test whether such features could be due to the ADI reduction and/or data characteristics, we built cADI and LOCI images of a bright⁴ HR 4796SD-like disk, inserted into our data cube, convolved by a theoretical PSF and, in the case of LOCI, reduced with Thalmann et al. LOCI parameters, assuming a 23 degrees FoV rotation. The input disk images are shown in Fig. 2, as well as the reduced ones. The reduced images clearly show features similar to those observed by Thalmann et al. (2011) in their Figs. 1 and 3. We also show in Fig. 2 the same simulations of a similar disk inserted into a data cube matching the parallactic angle excursions, but assuming no noise. Note that for the LOCI reduction of noiseless data, we took the coefficients derived from the previous LOCI reduction (taking noise into account); this is necessary to avoid major artefacts when using LOCI on noiseless data. The later simulations (no noise) highlight the reduction artefacts. Other simulations are provided in Milli et al. (2012). Hence, we conclude that the features indicated as “streamers” are probably artefacts due to the data characteristics (FoV rotation amplitude, number of data, SN) and data reduction. The fact that we do not see them in the present data is due to our relatively lower dynamical range, and to the larger FoV rotation amplitude. To check this, we injected a fake disk (HR 4796SD), with a flux similar to the observed one in the actual datacubes (with a 130° offset in PA to avoid an overlap of the disks). We processed the new data cubes as described above with cADI, mcADI, LOCI and mLOCI. The resulting images are shown in Fig. 3; the artefacts are not detectable. This is also true when considering a fake disk with blowout (HR 4796blowoutSD).

Fig. 2 (Left) From top to bottom 1) simulated HR 4796SD-like disk (projected, no noise); 2) simulated cADI reduction of this disk once inserted into our data cube, and convolved by a theoretical PSF. We assumed a 23 degrees FoV rotation as Thalmann et al. (2011), 3) idem with LOCI reduction. (Right) Idem without noise.

Fig. 3 Top: simulated disk (model HR 4796SD) inserted into the data: images (linear scale) reduced with cADI, mcADI, LOCI and mLOCI, for a rotation angle amplitude of 85 deg similar to that of our data. Bottom: idem for a disk model assuming αout =  −4 (model HR 4796blowoutSD). North is up and East is to the left. The real disk is NE-SW oriented; the simulated one is NW-SE oriented.

Finally, we note a small distorsion in the SW disk towards the inner region of the cADI and LOCI images, at (r, PA) between (19 pix, 235°) and (28 pix, 220°). The feature, indicated by a green arrow in Fig. 1, is however close to the noise level. If we go back to the individual images taken on April, 6th and April, 7th, we see that this feature is barely detectable on the April 6th eventhough in both cases, there is a very faint signal inside the disk ellipse (see Fig. 4 for the cADI images). Hence, in the present data, this feature could be due to noise. However, it seems to be at the same position as that pointed by Thalmann et al. (2011) in their data as well as Schneider et al. (2009) data as well as, in L′-data obtained at Keck by Marois & McIntosh (priv. comm.). In Thalmann et al. (2011) data, it appears as a loss of flux in the annulus. In the L′ data, we see rather a distorsion in the disk and a possible very faint additional signal at the inner edge of the disk. However, the ring does not appear as azimuthally smooth in the Thalmann et al. (2011) or our data, due to ADI reduction and limited SN, so it is not excluded at this stage that this feature might be an artefact. Clearly, new data are needed to confirm this structure, and if confirmed, to study its shape as a function of wavelength. If confirmed, its origin should be addressed. In the context of the HR 4796 system, an interesting origin to be considered is the presence of a planet close to the inner edge of the disk. To test the impact of the ADI reduction procedure on a disk + close planet system at L′, we inserted a fake point-like source close to our fake disk HR 4796SD (rotated by 130° with respect to the real disk, convolved by the observed PSF, and inserted in the datacube) inner edge, and processed the data as described previously. For these simulations, we assumed a disk about 10 times brighter than the real one. We run several simulations with different values of planet fluxes and distances to the disk. For some values of the planet position and flux, we were able to reproduce a disk appearent distorsion, especially in the LOCI images. A representative example is given in Fig. 5; in this case, the planet flux would correspond to a 2 M_Jup mass for a disk brightness similar to the HR 4796 one.

Fig. 4 Top: cADI image of HR 4796 on April, 6th. Bottom: idem on April, 7th.

Fig. 5 cADI, mcADI, rADI, LOCI and mLOCI images of simulations of fake disk + fake point-like source (indicated by a green arrow).

4.2. Disk properties

4.2.1. Disk geometry

We fitted the observed disk by an ellipse, using the maximum regional merit method, as in Buenzli et al. (2010) and in Thalmann et al. (2011). The resulting semi-major axis a, semi-minor axis b, disk center position along the semi-major axis (xc), and the semi-minor axis (yc), and inclination (defined as arccos(b/a)) are provided in Table 3, and the fit is showed in Fig. 6. These parameters are derived from the selection of the best fits, defined as those with parameters within 5% of the best fit (best merit coefficient). The uncertainties associated to these measurments take into account the dispersion within this 5% range, and the other sources of uncertainties that are described hereafter.

Fig. 6 Best elliptical fit to the L′ cADI data. NE is to the left and SW to the right.

To estimate the impact of the PSF convolution and ADI process on the ellipse parameters, we used our simulated disk HR 4796SD (without noise) and fitted the disk with an ellipse before and after the PSF convolution and the ADI reduction. For (a, b), differences of (–0.03; 0.12) pixels were found with cADI, and (–0.06; –0.3) pixels with LOCI. For (xc, yc), no significant differences were found with cADI and very small with LOCI. No significant difference was measured on the inclination with cADI while a difference of –0.4° was found with LOCI. Finally, no significant differences were found on the PA. We corrected the measured values on the disk from these biases. We also inserted a model disk HR 4796SD in the data cube (at 90 degrees), and processed the data. The differences found between the parameters of the injected disk and the ones of the recovered disk are compatible with the ones obtained in the case “witout noise”.

Besides, the imperfect knowledge of the PSF center may also affect the results. To estimate this impact quantitatively, we first estimated the error associated to the PSF center, as in Lagrange et al. (2012). The error was found to be [0, 0.27] pixel on the x-axis and [–0.06, +0.04] pixel on the y-axis of the detector. It appears that this imperfect knowledge on the PSF center does not significantly affect the values of (a, b). It impacts the uncertainty of the ellipse center by up to 0.2 pixel along the major and minor axis, and the disk PA (0.24°). The uncertainty on the PA measurement is found to be 0.15° in cADI.

Finally, the PA measurement is also impacted by the uncertainty on the true North Position (0.3°; see a discussion related to this last point in Lagrange et al. 2012).

Our data show an offset from the star center of  ≃ 22 mas on the cADI images ( ≃ 20 mas on the LOCI data) of the center of the fitted ellipse along the major axis, and to the South. Given the uncertainty associated to this measurement, 7 mas, we conclude that the observed offset is real. This offset along the major axis is in agreement with previous results of Schneider et al. (2009) (19 ± 6 mas) and Thalmann et al. (2011) on their LOCI images (16.9 ± 5.1 mas). The latter detected moreover an offset of 15.8 ± 3.6 mas along the minor axis, which was not detected by Schneider et al. (2009); the measured offset on our cADI images is about 8 mas  ± 6 mas; hence very close to the error bars, so the present data barely confirm the offset found by Thalmann et al. (2011).

4.2.2. Brightness distribution

Fig. 7 Observed SBD for the HR 4796A disk at L′ along the major axis (after a binning of 5 pixels perpendicular to the major axis): cADI and mcADI reductions (left); LOCI and mLOCI reductions (right). We note that the peaks of the SBD on the NE and SW do not coincide which is due to the shift of the ellipse center along the major axis.

Figure 7 shows the observed radial surface brightness distributions (SBD) for the HR 4796A disk at L′ along the major axis after cADI, and LOCI reductions. The SBD extraction was made using a 5 pixel vertical binning. The dynamical range of our data is small (factor of 10); it is improved with masking technics, thanks to a lesser disk self-subtraction (see also Fig. 7). Yet, the disk being only slightly resolved, we cannot perform meaningful slope measurements on our data. Indeed, the slope of the surface brightness distribution depends on several parameters: the PSF, the amplitude of the FoV rotation, the ADI procedure, the binning used for the extraction of the SBD, and the separation range on which the slopes are measured. In the present case, the separation range is too small to allow a proper measurement of the slope: we run simulations of the HR 4796SD disk without noise and checked that indeed, measuring the slopes between the maximum flux and the threshold corresponding to the noise on the actual data gave slopes very different (much higher) from the slope measured on a larger separation range.

The ring shape seems nevertheless to indicate a sharp outer edge, but we need to check the impact of the ADI reduction procedure on the final shape of the disk. Figure 8 illustrates the evolution of the SBD along the semi-major axis, starting from the fake disk HR 4796SD, then once the disk is inserted in the real data cubes at 130° (see the corresponding images in Fig. 3), convolved by the observed PSF, and finally when the datacube is reduced with cADI and LOCI. The SBD shape is clearly impacted. We note that the effect is stronger in the inner region that in the outer one. To test whether we can discriminate between a steep and a less steep outer profile, we consider the fake disks HR 4796SD and HR 4796blowoutSD inserted in the real data cubes at 130° (see the corresponding images in Fig. 3), and convolved with the real PSF. The SBD profiles after convolution and reduction along the semi-major axis are given in Fig. 9. We note that the slight shift between the observed and simulated disks SBD is due to an unperfect assumption on the ring position, and is not relevent here. The observed SBD profile appears to be more similar in shape to the ones corresponding to the HR 4796SD case rather than the HR 4796blowoutSD one. We conclude then that even when taking into account the possible biases, the data indicate a very steep outer edge, compatible with α_out =  −10, as found by Thalmann et al. (2011) rather than a less steep one. We cannot provide precise values to the outer NE and SW slopes with the present data, but they are in any case different from the ones measured in the case of the other A-type stars such as β Pictoris (typ. between –4 and –5; Golimowski et al. 2006) and HD 32997 (Boccaletti et al. 2012), and that are expected from a disk which outer part is dominated by grains blown out by radiation pressure from an A-type star (see below).

Fig. 8 Top: from left to right: simulated disk, simulated disk after convolution, after cADI, and after LOCI reductions. Bottom: corresponding SBDs, showing the evolution of the profile after different steps of reduction. See text.

4.2.3. Disk width

Our data indicate a ring width of about 0.154′′ after cADI reduction for the NE side and 0.136′′ for the SW side (data binned over 5 pixels). With mcADI, these values are only marginally changed: 0.147′′ and 0.135′′ respectively. However, as seen above, the SBD, especially inner to the ring is impacted by the PSF convolution, the amplitude of the FoV rotation, the ADI procedure, the binning used for the extraction of the SBD, the noise level as well as the zero flux level after reduction. We made several test with fake disks to estimate the impact of these steps on the FWHM. Also, we tested the impact of the evaluation of the zero level after reduction. It appears that the disk width is mainly affected in the present case by the PSF convolution and the zero level. Taking all these parameters into account, we cannot conclude that the disk is significantly narrower than the size found by Schneider et al. (2009) 0.197′′, which once corrected from the broadening by the STIS PSF became 0.18′′ (13 AU) at shorter wavelengths.

5. Companions around HR 4796A

5.1. SAM detection limits

The detection limits from the SAM dataset are derived from a 3D χ2 map. This map has on each axis the three parameters used to model a binary system: the separation, the position angle, and the relative flux. This model, fitted on the closure phase, is detailed in Lacour et al. (2011). Visibilities are discarded. Figure 12 is showing the detection limits as a function of right ascension and declination. It is obtained by plotting the 5σ isocontour of the 3D map (the isocontour level is given by a reduced χ² of 25).

We did not account for the presence of the disk in the model fitted. We considered that it did not affect the visibilities (because very faint), and did not affect the closure phase (because quasi point-symmetric). Nevertheless, it is not impossible that some of the structures present in Fig. 12 may be caused by a second order effect of the disk on the closure phase. Thus, neglecting the influence of the disk means that we are conservative on the detection limit map.

Given these values, and assuming V − L′ = 0 for this A0-type star, and an age of 8 Myr for the system, we derive the 2D detection limits expressed in Jupiter masses, using DUSTY models (Fig. 12; right). At a separation of about 80 mas (6 AU), we exclude the presence of companions with masses larger than 29 M_Jup. At a separation of 150 mas, the limit becomes M = 40 M_Jup (DUSTY). In both cases, COND03 models give similar limits within 1 M_Jup. At 60 mas, the detection limit is M = 50 M_Jup (DUSTY) and would be 44 M_Jup with COND03. Such values fall in the mass range of brown dwarfs and represent unprecedented mass limits for this range of separations.

Fig. 9 Left: simulated radial brightness distributions for the HR 4796ASD disk (green) and HR 4796ASBD disk with blowout (red) at L′ along the major axis (log scale) once inserted in the data cube and after cADI reduction has been applied. For comparison, observed SBD (black). Right: simulated radial brightness distributions for the HR 4796ASD disk (green) at L′ along the major axis (log scale) once inserted in the data cube and after LOCI reduction has been applied. For comparison, observed SBD (black).

Fig. 10 Left: 2D map of 5-σ detection limit of point-like structures towards HR 4796A, using SAM data. Isocontours for Δmag = 3.5, 4.5, 5.5, and 6.5. Right: detection limits expressed in Jupiter masses using COND03 models; isocontours for masses of 20, 40, 60, 80 and 100 MJup. In both images, North is up and East to the left.

5.2. ADI detection limits

We computed the detection limits using the data obtained on April 6th and 7th, with different reduction methods. To estimate them, we took into account the flux losses due to the ADI reductions, either injecting fake planets in cubes of empty frames obtained with similar FoV rotation for the cADI procedure, or injecting fake planets in the real data cubes in the case of the LOCI procedure. The noise was estimated using a sliding 9 pixel wide box along any given PA. We checked the obtained detection limits by inserting fake planets with fluxes corresponding to the 5σ limit at different separations, and processed again the data, and mesured the resulting S/N on the planets. The S/N were close (or sometimes slightly larger than 5) which shows that our limits are properly estimated. The 2D-detection limits are shown in Fig. 12 for cADI, expressed either in contrasts or in masses, using the COND03 models (Baraffe et al. 2003) or BT-settl models (Allard et al. 2012) and assuming an age of 8 Myr. Similar (not better) limits were obtained with rADI and LOCI. To check the robustness of the detection limits obtained, we injected fake planets with fluxes corresponding to the 5σ level as well as the fake disks and processed the data cubes as before. The resulting images (see Fig. 13) revealed the planets with at least a 5σ level.

The 1-D limits along the major axis, at a PA of 26 degrees (NE side of the disk) are showed in Fig. 11. Similar values are obtained in the SW direction. A few values expressed in jovian masses are given in Table 4. The detection limits are better than the SAM ones further than  ≃ 0.25–0.3′′, with a value of about 7.5 M_Jup at 0.25–0.3′′; they are below 3.5 M_Jup for separations in the range 0.5–1′′, and well below further than 1.5′′. Alltogether, these are to our knowledge the best detection limits obtained in the close surroundings of HR 4796A.

Fig. 11 5-σ detection limit of point-like structures around HR 4796A, along the major axis in the NE and SW directions using SAM and ADI data.

6. Companions around HR 4796B

HR 4796B has the following magnitudes: V = 13.3, H = 8.5, K = 8.3. Using the observed contrast between HR 4796A and HR 4796B (2.6 mag) on the present data, we find L′ ≃ 8.4 for HR 4796B, hence L′abs  ≃  4.1. This value is in agreement with the Lyon’s group model (Baraffe et al. 2003), which, given the near-IR colors, predicts an absolute L′ magnitude of 3.8. We give in Fig. 14 the 2D map of the detection limits (5σ), both in terms of contrast magnitudes and Jupiter masses. At 0.3′′, masses as low as 2 M_Jup could be detected, and at 0.5′′, the detection limit is below 1 M_Jup.

7. Discussion

7.1. The inner disk sharp edge

One of the most remarkable features of the HR 4796 disk is certainly the offset of the disk center with respect to the star (also observed in the case of HD 141569, and Fomalhaut). Two explanations are 1) the presence of a close, fainter companion (in such a case, the disk would be a circumbinary disk and orbit around the binary center of mass), and 2) the presence of a companion close to the disk inner edge on an eccentric orbit that induces a forced eccentricity to the disk ring by secular gravitational interaction, an explanation which was proposed to explain the eccentricity of the Fomalhaut disk (Quillen et al. 2006; Kalas et al. 2005, 2008; Chiang et al. 2009).

We first investigate whether this offset could be due to the presence of a close companion. In such a case, the ellipse center would mark the center of mass of the binary system. Using the center of mass definition, it appears that the mass of a body necessary to shift the center of mass at the observed position of the ellipse center would be much larger than the detection limit obtained with SAM between 40 and 400 mas or between 400 mas and 1 arcsec (ring position) with the ADI data. A companion located between 23 and 40 mas would have a mass larger or comparable to that of HR 4696A; such a scenario must be excluded as under such conditions, the photometric center of the system would also be shifted. Then, the most plausible explanation to the offset is a light eccentric planet close to the inner edge of the disk.

We now try to use the detection limits found in this paper to constrain the properties of an inner planet that could be responsible for the steep inner edge observed with HST/STIS data, which, conversely to ADI data, is not impacted by ADI reduction effects. Wisdom (1980) showed that in the case of a planet and particles on circular orbits, we have the relation δa/a = 1.3(M_p/M_s)^2/7 where M_p and M_s are the planet and star masses, a is the orbital radius of the planet and δ a the distance between the planet and the disk inner edge. Hence if a planet sculpts the inner edge, its mass and distance from the inner edge must satisfy this relation. Assuming an inner edge located at 77 AU, we can derive the mass of the planet necessary to produce this sharp edge, as a function of its distance to the edge, and test whether such a planet would have been detected or not. This is done in Fig. 15 where we show the region, inside the yellow ellipse, that, given the present detection limits, have to be excluded. Hence the only possible location of the planets responsible for the inner edge is between the yellow ellipse and the red one (which traces the inner edge of the disk). We see that along the major axis, only the planets closest to the inner edge (less than  ≃ 10 AU) remain out of the present detection capabilities. Hence if a planet is responsible for the inner edge sculpting and is located along or close to the major axis, then it should be a low mass planet, and located further than 63 AU, i.e. within about 15 AU from the edge. Along the minor axis, due to the projection effects, the presence of planets is much less constrained: only planets at more than 26 AU from the edge would have been detected.

The previous constrains were obtained assuming the planet and the perturbated bodies are both on circular orbits. The actual disk eccentricity beeing very small, about 0.02, this assumption is reasonable. As an exercice, we investigate the impact of a higher eccentricity, using the results of Mustill & Wyatt (2012), who revisited this scenario, assuming the perturbating bodies were on an eccentric orbit; the relation becomes: δa/a = 1.8  e^1/5(M_p/M_s)^1/5. With the same reasonning, and assuming an eccentricity of 0.1, we provide in Fig. 15 the possible locations of a planet responsible for the inner edge, with the same color conventions as in the circular case. Again, comparison with Fig. 12 shows that if a planet was responsible for the inner edge sculpting, and located along or close to the semi-major axis, then it would have to be located less than  ≃ 25 AU from the edge of the disk. The location of planets along or close to the minor axis would not be significantly constrained. This case, even though not adapted to the present case as the disk eccentricity is very small, illustrates the impact of this parameter on the planet detection capabilities.

Fig. 12 2D map of 5-σ detection limit of point-like structures towards HR 4796A, using all available data (CADI). Left: expressed in contrasts (isocontours from 7 to 11 mag, steps of 1 unit). Right: translated in masses using COND03 models (isocontours from 2 to 10 MJup, steps of 1 MJup).

7.2. The outer disk sharp edge

Another striking feature of the system is the very steep disk outer edge. Radiation pressure from A-type stars induces surface brightness distributions in the outer part of disks with slopes of –3.5 to –5, depending on the assumptions related to the production laws of small grains (see for instance Lecavelier Des Etangs & Vidal-Madjar 1996; Thebault & Wu 2008, and references there-in). In particular, Thebault & Wu (2008) modeled the outer parts of collision rings with an initially steep outer edge, following the motion of the small grains produced through collisions and submitted to radiation pressure and showed that, as already proposed, the profile of the resulting SBD in the outer part of the disk followed a r^-3.5 law. They showed that unless the disks are extremely and unrealistically dense, and prevent the small grains from escaping, the disks have to be extremely “cold” (with an average free eccentricity of  ≤ 0.0035) to explain an outer power law of r^-6 (which was the value adopted at this time for the HR 4796 profile). AO data suggest that the situation could be even more radical with an even steeper outer edge than previously thought. Also, if confirmed, the fact that we possibly find at 3.8 μm a disk width different from that found in the optical (0.2–1 μm) with STIS would argue against an extremely dense disk, as, in such a case, large bodies and small grains would be in the same regions. The cold disk scenario can nonetheless be also problematic as, in such a case small grains should be underabundant (Thebault & Wu 2008) and the optical/near-IR fluxes would be produced by large particules (typ. sizes 50 μm). We should expect then a color index different from that observed (see Debes et al. 2008). The latter rather predict that the scattered light flux is dominated by 1.4 μm dust, which seems difficult to explain within the dynamically-cold-disk scenario.

Fig. 13 Fake planets inserted at 0.3, 0.5, 0.8, and 1′′ on both sides of the disk, with fluxes corresponding to the 5-σ detection levels, and reduced with cADI and LOCI.

Could gas be responsible for the observed steep outer edge? Takeuchi & Artymowicz (2001) investigated the impact of the gas in such debris disks, and showed that even small amount of gas, 1- a few Earth masses, could partly balance the effects of gravitation, radiation pressure and Poynting-Robertson drag and alter the grains dynamics differentially, and lead to grains spatial distributions, that, depending on the grain sizes, could be different from those expected in a disk-free gas. Under such processes, the gas could be responsible for ring-like structures at distances depending on the dust size considered. In their attempt to investigate disks roughly similar to HR 4796 and HD 141569, assuming 1 M_Earth gas, they showed that, conversely to large grains which occupy the whole gas disk, grains with sizes ( ≃ 10–200 μm) tend to concentrate in the outer gas disk, where the gaseous density sharply decreases. Hence these grains would form a narrow ring which position traces the change in the radial distribution the gaseous disk. Under this a priori attractive scenario, grains with sizes 1–10 μm would still be blown away. The main problem with this hypothesis is that it requires a gas disk with a relatively sharp outer edge, and thus an explanation for such an edge. Another issue is that Takeuchi & Artymowicz (2001) did not explore the SBD profile beyond the main dust ring, so that it remains to be see whether slopes in the –10 range are possible. Finally, it is worth noting that so far no circumstellar gas has been detected either in atomic species through absorption spectroscopy (but the system being inclined, the non detection is not a strong constraint) or molecular species, either CO (Liseau 1999) or H2 (N(H2) ≤ 10¹⁵ cm^-2, Martin-Zaiedi et al. 2008). In any case, such a scenario can be tested in the forthcoming years with high angular resolution observations on a wide range of wavelengths.

Fig. 14 2D map of 5-σ detection limit of point-like structures around HR 4796B (April 6th), expressed in contrast (left), with contour-levels of 8 to 11 mag (step = 1 mag); and in Mass (right), with contour-levels of 1 to 6 MJup (step = 1 MJup).

Fig. 15 Possible remaining locations of a planet inner to the disk that could produce an inner sharp edge at 77 AU given the present detections limits (left: case of a circular orbit; right: case of an elliptical orbit with e = 0.1). The possible locations are restricted to the region between the yellow curve and the red ellipse that mimics the disk itself. North is up, East to the left.

Fig. 16 Synthetic surface brightness profiles obtained, using Thebault (2012)’s numerical model, for 3 different masses of a putative perturbing outer planet: 8   MJup, 5   MJup and 3   MJup. For each case, the planet has a circular orbit and is placed as close as possible to the main ring of large parent bodies in order to truncate it at about 75 AU without destroying it (see text for more details). The observed profile, derived from Schneider et al. (2009), is shown for comparison. For each planet mass, we show the radial cut (i.e., for one position angle along the disk) that provides the best fit to this observed profile. The horizontal line delineates approximately the part (above this line) of the SBD accessible to the observations assuming a dynamical range of 10.

We finally study the possibility that the outer disk is sculpted by massive bodies. The first candidate we might think of is HR 4796B. Thebault et al. (2010) investigated the possibility that the disk could be sculpted by HR 4796B, if orbiting on a rather eccentric orbit (e ≥ 0.45) but again showed that, even under such conditions the outer profile would not be so steep. This is mainly because the companion star is not able to dynamically remove small grains from the outer regions at a pace that can compensate for their steady collisional production in the parent body ring.

An alternative explanation could be the presence of a close, unseen outer planet. We investigate this scenario using the new code developed by Thebault (2012) to study perturbed collisionally active debris disks. The code computes the motion of planetesimals submitted to the gravitational perturbation of a planet; and follows the evolution of small dust realeased through collisions among the planetesimals and submitted to radiation pressure and Poynting-Robertson effect (note that in the present case, radiation pressure largely drives the grains dynamics once produced). The configuration we consider is a narrow ring of large parent bodies, a birth ring of width  ~8 AU centered on 71 AU. The collisional production and destruction rates of small grains (those which contribute to the luminosity beyond the main ring) in this parent body ring is parameterized by the average vertical optical depth within the main ring, taken to be τ = 5 × 10^-35. The resulting SBD (case face-on) is derived assuming grey scattering. It can be directly compared to the curve presented in Fig. 6 of Schneider et al (2009) paper (de-projected curve⁶).

For the perturbing planet’s mass, we consider 3 different values: 8   M_Jup, 5   M_Jup and 3   M_Jup, which are consistent with the constraints imposed by our observational non-detection. Note that 8   M_Jup is only marginally possible in a very narrow region along the disk’s semi-minor axis, but we have to keep in mind that the detection limits are derived from masses-brightness relationships that are debated at young ages. We assume a circular orbit for the planet (the most favorable case for cleaning out the region beyond the ring, see Thebault 2012) and place it as close as possible to the parent body ring, i.e., so that the outer edge of the observed ring (around 75 AU) corresponds to the outer limit for orbital stability imposed by planetary perturbations. This places the planet at a distance to the central star comprised between 92 and  ~99 AU depending on its mass.

In Fig. 16, we show the SBD obtained for such a configuration. Note that, for each planet mass, we are not showing an azimutal average but the “best” radial cut, i.e. the one that gives the closest match to the deprojected NE side SBD obtained in Fig. 6 of Schneider et al. (2009). As can be clearly seen, the 8   M_Jup case provides a good fit to the observed profile: the maximum of the SBD roughly corresponds to the outer edge of the parent body disk and is followed by a very sharp brightness decrease, with a slope  ≤–10 between 75 and 95 AU, i.e. between brightnesses of 1 and 0.1, a range corresponding approximately to the dynamical range accessible to the available images. This is significantly steeper than the one that would be expected if no planet was present (–3.5 according to Thebault & Wu 2008) and is fully compatible with the observed sharp luminosity decrease.

Longwards 95 AU, the flux level is lower ( ≤ 0.1 ADU) and the SBD is flatter (slope  ≃ –3.8). We also note a plateau inside the parent body ring at a level of  ~0.2, due to both the inward drift of small grains because of the Poynting-Robertson effect and to the dynamical injection of particles after close encounters with the planet. Of course, not too much significance should be given to the SBD obtained inwards of the disk since our simulations (focused on the outer regions) do not consider any inner planet shaping the inner edge of the disk. Nevertheless, they show that, should “something” have truncated the disk at around 67 AU in the past, then the effect of one external planet on such a truncated disk could lead to an SBD compatible with observations in the inner regions. For the 5   M_Jup case, the fit of the observed SBD is slightly degraded, but mostly in the region beyond 90 AU where flux levels are close to the 0.1 threshold. For the 3   M_Jup case, however, the fit gets very poor for almost the whole outer region (keeping in mind that we are here showing the best radial cut).

We conclude that a 8   M_Jup planet located on a circular orbit at  ~25 AU from the main ring provides a satisfying fit (especially considering the non-negligeable uncertainties regarding flux values far from the main ring) to the observed SBD. According to the derived detection limits, such a massive planet would have been detected almost everywhere except in a very narrow region along the disk’s semi-minor axis; however, we remind the uncertainties inherent to the models used to link planet masses and luminosities as a function of the system’s age. In any case, even a less massive perturber of e.g. 5   M_Jup would still give an acceptable fit of the observed luminosity profile. The external planet scenario thus seems the most likely one for shaping the outer regions of the disk. Of course, these results are still preliminary and should be taken with caution. A more thorough numerical investigation should be carried out, exploring a much wider parameter space for planet masses and orbit, as well as deriving other outputs that can be compared to observations, such as 2-D synthetic images. Such a large scale numerical study exceeds the scope of the present work and will be the purpose of a forthcoming paper. Note also that the more general issue of how planets shape collisionally active debris disks will be thoroughly investigated in a forthcoming paper (Thebault, in prep.).

8. Summary and future prospects

In this paper, we have provided the first high-resolution images of the HR 4796A disk at L′ band. They allow us to see a narrow disk at almost all PA. As the technics used, Angular Differential Imaging is expected to impact the final disk shape and appeareance, we have developped simulations to investigate quantitatively the impact of the reduction procedures on the disk parameters. We conclude that the information on the inner part of the disk is significantly impacted, and that the procedure may in some cases (depending on the amplitude of rotation of the field of view), produce important artefacts. This is specially true for LOCI reduction, while classical ADI affects the data to a lower extent. We showed in particular that the streamers detected by Thalmann et al. (2011) at the outer edge of the disk are probably due to such artefacts.

Using both ADI and SAM data, we have derived unprecedented lower limits to the presence of planets/companions down to 25 mas from the star. The present data allowed then to put first interesting constrains on the location of the possible planet that could produce the inner edge of the disk. We showed that the planet responsible for the inner edge must be closer than 15 AU from the ring if located along or close to the semi-major axis. The forthcoming high dynamics instruments such as SPHERE on the VLT and GPI on GEMINI will allow to test this hypothesis with much more accuracy, and be able to actually detect this planet in most cases.

We have discussed several hypotheses to explain the sharp outer edge of the disk: gaseous disk, dynamically cold disk, planet on the outer edge. Using detailed simulations, we showed that a planet located outside the planetesimal ring could nicely reproduce the STIS data. Further simulations will help to better constrain the planet and parent bodies characteristics.

In any case, this work shows how disks characteristics can help constraining possible planet properties. A very important information can be brought by the dependance of the disk properties (ring width, SBD) as a function of wavelength. Resolved images in the future will be crucial to further understand this system.

Acknowledgments

We acknowledge financial support from the French Programme National de Planétologie (PNP, INSU). We also acknowledge support from the French National Research Agency (ANR) through project grant ANR10-BLANC0504-01 and ANR-2010 BLAN-0505-01 (EXOZODI). We also thank C. Marois, B. McIntosch and R. Galicher for discussing their L′ Keck data with us. We also thank the anonymous referee for his/her helpful comments.

References

All Tables

All Figures

Fig. 1 HR 4796A disk at L′ (linear scale). The pixel scale is 27 mas/pixel. From top to bottom, the same data reduced with nADI, cADI, cADI-disk, mcADI, rADI, LOCI, and mLOCI (see text). Note that the color codes are identical for all cADI reductions reductions on the one hand, and LOCI reductions on the other hand, to enable comparisons within a given method, but different cuts are used for cADI, rADI and LOCI reductions.

In the text

	Fig. 2 (Left) From top to bottom 1) simulated HR 4796SD-like disk (projected, no noise); 2) simulated cADI reduction of this disk once inserted into our data cube, and convolved by a theoretical PSF. We assumed a 23 degrees FoV rotation as Thalmann et al. (2011), 3) idem with LOCI reduction. (Right) Idem without noise.
In the text

	Fig. 3 Top: simulated disk (model HR 4796SD) inserted into the data: images (linear scale) reduced with cADI, mcADI, LOCI and mLOCI, for a rotation angle amplitude of 85 deg similar to that of our data. Bottom: idem for a disk model assuming αout =  −4 (model HR 4796blowoutSD). North is up and East is to the left. The real disk is NE-SW oriented; the simulated one is NW-SE oriented.
In the text

	Fig. 4 Top: cADI image of HR 4796 on April, 6th. Bottom: idem on April, 7th.
In the text

	Fig. 5 cADI, mcADI, rADI, LOCI and mLOCI images of simulations of fake disk + fake point-like source (indicated by a green arrow).
In the text

	Fig. 6 Best elliptical fit to the L′ cADI data. NE is to the left and SW to the right.
In the text

	Fig. 7 Observed SBD for the HR 4796A disk at L′ along the major axis (after a binning of 5 pixels perpendicular to the major axis): cADI and mcADI reductions (left); LOCI and mLOCI reductions (right). We note that the peaks of the SBD on the NE and SW do not coincide which is due to the shift of the ellipse center along the major axis.
In the text

	Fig. 8 Top: from left to right: simulated disk, simulated disk after convolution, after cADI, and after LOCI reductions. Bottom: corresponding SBDs, showing the evolution of the profile after different steps of reduction. See text.
In the text

Fig. 9 Left: simulated radial brightness distributions for the HR 4796ASD disk (green) and HR 4796ASBD disk with blowout (red) at L′ along the major axis (log scale) once inserted in the data cube and after cADI reduction has been applied. For comparison, observed SBD (black). Right: simulated radial brightness distributions for the HR 4796ASD disk (green) at L′ along the major axis (log scale) once inserted in the data cube and after LOCI reduction has been applied. For comparison, observed SBD (black).

In the text

	Fig. 10 Left: 2D map of 5-σ detection limit of point-like structures towards HR 4796A, using SAM data. Isocontours for Δmag = 3.5, 4.5, 5.5, and 6.5. Right: detection limits expressed in Jupiter masses using COND03 models; isocontours for masses of 20, 40, 60, 80 and 100 MJup. In both images, North is up and East to the left.
In the text

	Fig. 11 5-σ detection limit of point-like structures around HR 4796A, along the major axis in the NE and SW directions using SAM and ADI data.
In the text

	Fig. 12 2D map of 5-σ detection limit of point-like structures towards HR 4796A, using all available data (CADI). Left: expressed in contrasts (isocontours from 7 to 11 mag, steps of 1 unit). Right: translated in masses using COND03 models (isocontours from 2 to 10 MJup, steps of 1 MJup).
In the text

	Fig. 13 Fake planets inserted at 0.3, 0.5, 0.8, and 1′′ on both sides of the disk, with fluxes corresponding to the 5-σ detection levels, and reduced with cADI and LOCI.
In the text

	Fig. 14 2D map of 5-σ detection limit of point-like structures around HR 4796B (April 6th), expressed in contrast (left), with contour-levels of 8 to 11 mag (step = 1 mag); and in Mass (right), with contour-levels of 1 to 6 MJup (step = 1 MJup).
In the text

	Fig. 15 Possible remaining locations of a planet inner to the disk that could produce an inner sharp edge at 77 AU given the present detections limits (left: case of a circular orbit; right: case of an elliptical orbit with e = 0.1). The possible locations are restricted to the region between the yellow curve and the red ellipse that mimics the disk itself. North is up, East to the left.
In the text

Fig. 16 Synthetic surface brightness profiles obtained, using Thebault (2012)’s numerical model, for 3 different masses of a putative perturbing outer planet: 8   MJup, 5   MJup and 3   MJup. For each case, the planet has a circular orbit and is placed as close as possible to the main ring of large parent bodies in order to truncate it at about 75 AU without destroying it (see text for more details). The observed profile, derived from Schneider et al. (2009), is shown for comparison. For each planet mass, we show the radial cut (i.e., for one position angle along the disk) that provides the best fit to this observed profile. The horizontal line delineates approximately the part (above this line) of the SBD accessible to the observations assuming a dynamical range of 10.

In the text