Spectroflat: A generic spectrum and flat-field calibration library for spectro-polarimetric data

Context. Flat-fielding spectro-polarimetric data with one spatial and one spectral dimension is inherently difficult as the imprint of the spectral lines needs to be separated from other wavelength-dependent instrumental effect (e.g., fringes or prefilter profiles) and wavelength-independent effects (e.g., dust and sensor response). Current approaches for spectrometers are often based on moving the grating or they depend on optical models and/or on lab calibration data. They are also limited to small spectral regions and are instrument-specific. Approaches that would be suitable for polarimeters have not been reported yet. Aims. We present an approach that allows for flat-field calibration data to be to obtained for diffraction-grating-based, long-slit spec-trographs combined with temporally modulated polarimetry from high-resolution solar telescopes. This approach is based on nominal flat-fielding procedures performed during the instrument’s science operations. Methods. We performed a precise and field-dependent correction of the spectrographic distortion effect (resulting in curved spectral lines, typically denoted as a “smile” effect) to ensure the orthogonality of the spectral and spatial dimensions. We identified distortions by tracking the position of multiple spectral lines within the full spectral field of view. From the raw modulated flats, we then removed the solar line imprints and derived separate flat-fields for sensor and slit dust features. Optionally, wavelength calibration and continuum correction can be included in this process. Results. We have created generic Python libraries that can be plugged into existing Python-based data reduction pipelines or used as a standalone calibration tool. We show that for spectrographs covering many spectral lines, a correction of the smile distortion based on optical models alone is not sufficient. Our results demonstrate a suppression of fringes, sensor artifacts, and fixed-pattern imprints in demodulated data by one order of magnitude. For intensity images, the photon noise level can be closely attained after calibration. Our correction works across the full spectral range. The algorithm was tested for different wavelength regimes with emission (EUV range) or absorption (near-UV, VIS, IR range) spectra, on data acquired with ground-based (SST/TRIPPLE-SP, GREGOR/GRIS), balloon-borne (SUNRISE-III/SUSI), and space-based (SolO/SPICE) instruments. The data calibrated with our method offer robust and precise inversion results. Conclusions. We have extended existing spectroscopic flat-field techniques to modern instruments with large imaging sensors covering many spectral lines simultaneously, and with polarimetric capabilities, where methods described so far are not adequate. We believe that our method is applicable as a standard calibration approach for most modern high resolution large-FOV, long-slit spectrographs – both with and without polarimetric capabilities.


Introduction
The flat-field calibration process characterizes and corrects images for the non-homogeneous response of imaging sensors, such as those based on charge-coupled devices (CCD) and complementary metal-oxide-semiconductor (CMOS) detectors, across their light-sensitive area.The flat-field also typically includes the contribution of the optical components in the beam path, from the main telescope mirror to the imaging detector.In order to obtain images closer to the real object, all recorded data must be corrected for the effects accounted for by the flatfield (e.g., dust specs, vignetting, fringes, etc).The quality of this calibration process strongly influences the scientific utility of the observational data.Flat-field calibration is thus compulsory for any kind of astronomical observation and it affects ground-, balloon-and space-based observations alike.This paper considers the specific difficulties inherent in flat-fielding solar spectrographs (producing images with one spectral and one spatial dimension) combined with polarimetry.As the accumulation of dust particles is a recurrent process and the stability of other flat-field features depends strongly on the structural stability and configuration of the telescope and post-focus instrument, flat-field data are to be obtained close in time before and/or after the observation.In practice, multiple flat-fields are recorded during an observation day (see e.g., Chae et al. 2013;Kotrč et al. need methods to remove solar imprints in both dimensions from the calibration data. Typically, an imaging flat-field is retrieved from an extended, non-uniform source by shifting the image on the sensor and using the mean of multiple such displaced images (Tyson 1986).Other methods, such as proposed by Kuhn et al. (1991), derive the flat-field from a small series of displaced images via a least-squares estimate.Most commonly, the spatial uniformity is enforced by moving the telescope in a random-walk manner or following a (rotating) ellipsis or Lissajous-pattern in a quiet Sun region near disk center, while constantly collecting and averaging frames (Denker et al. 1999).For spectrographic observations this approach alone is not sufficient, since the solar line features will not be displaced enough on the spectrograph sensor by pointing the telescope to a slightly different location; at the same time, the telluric lines will not be displaced at all.Although averaging spectral images (collected along a path around the solar disk center) does produce uniform quiet-Sun-like line profiles, they still need to be removed in order to obtain the desired flat calibration data.
Current flat-fielding procedures for diffraction-gratingbased, long-slit1 spectrographs often include moving the grating (either to a nearby continuum section or constantly back and forth to displace line imprints), defocussing the image, and/or ignoring vignetting and other spectral features alltogether (examples in Keller et al. 2003;Chae et al. 2013;Wülser et al. 2018).Jones (2003) and Wang et al. (2019) pointed out that for spectroscopic observations, any change to the instrument configuration modifies the imaging of optical impurities by either shifting them or changing their imprints amplitude and/or shape on the sensor.Furthermore, Jones (2003) demonstrated that such configuration changes modify the fringe patterns as well.
Spectroscopic data are inherently affected by an optical distortion, which causes the spectral lines to appear bent on the detector, commonly known as line smile (Zhao 2003).The smile distortion has to be corrected before the desired flat-field information can be successfully extracted.Typical spectrographic requirements for a maximum residual smile distortion range between 10% and 20% of a pixel size (Kaiser et al. 2008;Reimers et al. 2015).The smile correction is currently done in various ways.One method estimates the expected curvature function using an optical model and use it to correct the data, as described in Qiu et al. (2022), for instance.Another method adjusts the optical design to meet the aforementioned requirements on residual smile distortion without any postcorrection (Wei et al. 2016).However, this is not always possible or desirable, as the additional lenses or mirrors required may lead to higher instrumental complexity and losses in efficiency and/or imaging quality.Techniques to determine the smile curvature from flat-field measurements exist; for instance fitting a loworder polynomial to the detected line core positions of a single line (see for instance Bommier & Rayrole 2002;Wöhl et al. 2002;Chae et al. 2013).Cai et al. (2017) first presented an approach to extract the smile characteristic from multiple lines.They used the median of all telluric lines in the wavelength window to correct the full frame.This makes their approach more precise than previous methods with a single line smile characteristic, but it does not allow for a differential smile correction of the full field of view (FOV).Further, several scientifically interesting spectral regions without any telluric lines exist.Data from space based or balloon borne instruments do also not show telluric lines for obvious reasons.Moreover, the spectrographic curvature effect can vary quite drastically for large spectral FOVs (see e.g., Sect.2.2), limiting the applicability of these smile correction methods to instruments that cover only narrow spectral regions of a few Å.
In the case where the mean images can be considered spatially uniform and "free from smile", other approaches for removing spectral lines and producing a flat-field exist: Jones (2003) removed the line imprints in the image by dividing the 1D spatial information for each spectral point by a least-squares fit to this spectral point along the spatial dimension (parallel to the slit).A more common approach is the mean profile method (see Bommier & Rayrole 2002;Wöhl et al. 2002), where every spectral profile of the frame (i.e., a cut perpendicular to the spatial dimension for a given spatial resolution element) is divided by the spatially averaged spectral profile.
In the case of spectropolarimetric data calibration, the correction of both spatial flat-field effects and smile becomes even more challenging.This is because high-precision polarimetry requires the detection of small differential signals and, thus, differential distortion effects between modulation states may have a significant influence on the resulting Stokes parameters.For high-precision polarimetry with accuracy requirements of 0.1% or less in the measured degree of polarization, the requirements for the maximum differential smile distortion after correction is at the level of a few percent of a pixel only (comparable to effects and scales of seeing introduced cross-talk; e.g., Judge et al. 2004 andIglesias et al. 2016).Also, inversion codes are sensitive to flat-fielding effects and especially fixed pattern noise after demodulation (see, e.g., Bellot Rubio & Collados 2003;Longcope & Parnell 2009) therefore, they are heavily dependent on properly calibrated data.Until now, to the best of our knowledge, no detailed procedure for spectropolarimetric data has been published so far.
With the availability of modern large-size detectors, instruments that allow the observation of wide spectral windowscovering several nanometers and therefore registering tens to hundreds of spectral lines simultaneously -have become more relevant and widespread.Riethmüller & Solanki (2019) investigated the spectropolarimetric Stokes inversions used in remote sensing of the Solar atmosphere.They compare the performance of the traditional approach of inverting only a few spectral lines registered in multiple wavelength regimes to the joint inversion of many lines covering a wide spectral range and observed simultaneously.They found more accurate inversion results and an extended height coverage in the "many lines" case.On the other hand, the successful application of numerical methods used to restore images subject to variable instrumental or ambient optical aberrations, such as the spectral restoration technique developed by van Noort (2017), also require an accurate smile correction over large (spectral) FOVs.
In this work, we introduce a novel algorithm, spectroflat, and we provide a publicly available generic library for the highprecision calibration of spectropolarimetic observation data.Our technique accounts for detection and correction of the smile distortion over larger spectral FOVs and generates flat-field images in an automated manner from calibration data captured close in time to the observation and in the operational instrument configuration.The rest of the paper is organized as follows.Section 2 explains the method in depth, including notation, input data requirements, smile correction, and the extraction of multiple flat-field images.It also summarizes the optimal data reduction approach with calibration data obtained through A22, page 2 of 14 Hölken, J., et al.: A&A, 687, A22 (2024) spectroflat.Section 3 validates the algorithm using solar data from four instruments including the infrared (IR), visible (VIS), and near-(NUV) and extreme-ultra violet (EUV) spectral regimes.These instruments are: (1) the spectropolarimetric extension of the TRI-Port Polarimetric Echelle-Littrow (TRIPPEL) spectrograph (Kiselman et al. 2011) attached to the Swedish Solar Telescope (SST/TRIPPLE-SP)2 .(2) the GRE-GOR Infrared Spectrograph (GREGOR/GRIS) (Collados et al. 2012) installed at the Observatorio del Teide, Tenerife; (3) the Sunrise Ultraviolet Spectropolarimeter and Imager (SUNRISE-III/SUSI) (Feller et al. 2020).We note that an overview of the SUNRISE I+II missions can be found in Solanki et al. (2010Solanki et al. ( , 2017)), while a description of the SUNRISE-III mission is in preparation; (4) the Spectral Imaging of the Coronal Environment instrument on board of Solar Orbiter (SolO/SPICE) (SPICE Consortium et al. 2020).In Sect.4, we summarize our main conclusions.

Method
The input data is assumed to be previously calibrated for the camera zero-input response (dark current and bias level) and non-linearity.Furthermore, we assumed the individual frames to be already sorted with regard to their modulation state (if any) and combined into mean images for each modulation state.The targeted flatness, for instance, in terms of the root-mean-square (RMS) variation of the mean flat-field frame, largely depends on the science case and targeted sensitivity.Let us define the spectropolarimetric input data cube as having dimensions of s × y × λ, where the symbols denote the modulation state, spatial location along the spectrograph slit, and wavelength, respectively.We refer to the individual mean modulation state data (y × λ) in such a cube as state images, the following method is applicable to every state image independently.The result of this approach are flat-field matrices for each modulation state.We denoted this set of flat-fields as a "modulated flat".Intensity-only spectrographic data is treated as data with only one modulation state, i.e. a cube with s-dimension equal to one.In addition, we assume the spectral dimension is extending along to the imaging sensor rows (x-axis) and the spatial dimension extends along the sensor columns (y-axis).Therefore, we refer to individual spectra simply as rows.Since transposing a matrix is a numerically conservative operation, this requirement is purely formal.
The image rotation with respect to the sensors x-axis is best measured with dedicated data from a suitable calibration target.If no distortion effects are apparent the measured rotation can be given as a configuration value, and will then be corrected by the library.Higher level distortions (e.g., keystone) are out of the scope of the current implementation and need to be corrected before feeding the data to spectroflat.The deviation along the y-axis is removed by the smile correction.We call the shift of a pixel in the spectral direction with respect to its reference position (i.e., its deviation from a parallel line to the y-axis), its offset.
Overall, spectroflat is capable of correcting the full FOV of the input or only a region of interest (ROI).As the functionality is the same in both cases, we can assume and showcase full FOV correction.The implementation makes heavy use of concurrent processing.Computing time depends on the number of cores and amount of RAM available but is typically in the order of minutes.spectroflat follows the basic idea of estimating separate flat-fields for different fixed pattern imprints sources, for instance, the sensor, slit, and illumination field, as presented by Wöhl et al. (2002).However, our smile detection is done for the full FOV and we extended the idea to also extract the flat-fields for each modulation state individually.Further, the flat-fields themselves are extracted using new algorithms.

Preparation
The first step is an adhoc removal of the dust imprints and any other spatial variation from the state images to increase the quality of the smile correction.The corrected image is obtained by fitting a high order (default 13) polynomial to the pixel values of each image column (compare Jones 2003).If the sensor presents a column to column variation, for instance, due to conversion gain, the algorithm can be configured to remove this from the corrected image.In this case, the average column to column variation of all columns is used to adjust the gains in all columns multiplicatively.If such a component exists, it has to be taken out in this step since (a) the stripy pattern might interfere with the line core detection of the smile detection (see below) and (b) after smile correction and rotation this pattern will not be parallel to the image y-axis anymore and removal would become unnecessarily complex.
An example of the results of this procedure is shown on the right hand side of Fig. 1.The example flat-field data is taken from the 2017 TRIPPEL-SP campaign on the Swedish Solar Telescope (SST).

Desmiling
The smile detection algorithm is executed on the corrected state images obtained in the preparation step.Where dust imprints and column to column sensor effects are temporarily removed.As correction reference, the mean spectrum of the central five rows is used to further reduce the influence of noise and row to row variations.From this 1D reference spectrum (R), a set of spectral lines is selected automatically based on configuration values (relative line depth, distance between line centers).For each selected line, a Lorentzian is fitted to detect the line center with sub-pixel resolution.For each row, the same set of lines is selected and their corresponding sub-pixel center positions are detected using the same fit routine.
The relative displacement between line centers of a given row (C) and the reference spectrum (R), is determined by fitting several polynomials (p, of degrees two to a configurable maximum) to the detected differences in the line center positions.For each fit polynomial, the new position of each pixel is obtained using a cubic spline (CS) interpolation, resulting in a smile-corrected row spectrum ( Cp ).In a brute force approach the polynomials and corrected spectra are computed for each polynomial-degree in the configured range.The polynomial that minimizes the error function, is selected.The offsets of the individual pixels in spectral direction are computed using the selected polynomial.This is done for every image row individually.
After the offsets of all rows have been detected, an offset and an error map are generated.After all modulation states have been processed, the default behavior is to average the individual offset maps and use the mean map to correct all modulation A22, page 3 of 14  states, in order to avoid differential polarization artifacts. 3.To correct the smile distortion of any applicable frame, each row is CS-interpolated according to the detected offsets. Figure 2 shows the global offset and error maps corresponding to Fig. 1.The cuts in panel a demonstrate that a global smile curvature determined from a model or a single line is not enough for instruments targeting multiple spectral lines.Panel c shows the pixel-wise error values E(p min , x) for the selected polynomial.Figure 3 shows spectral profiles of the image around the Fe I, 5250.6 Å line.In the lower panel, the line centers are shifted and differ by less than 10% of a pixel.
For all the instruments analyzed in this work, unless the instrument configuration does not change, the resulting smile distortion pattern is stable over time.It can therefore be applied to frames that are already temporally averaged, for instance, in the slit scanning direction.Not correcting the individual frames saves computation time during data reduction.

Sensor and slit flats
For preparation (see Sect. 2.1), we used an ad-hoc correction to remove dust from the image in order to improve the smile 3 It is possible to extract an offset map per modulation state.As none of our datasets suffer from modulation state-dependent smile effects introduced by the modulation package, we have not been able to test this mode rigorously until now.Simple simulations show that it might be possible to mitigate differential artifacts in some cases, but testing is incomplete at time of writing.We therefore mark this mode as experimental.detection.However, this is an imprecise method to extract the dust imprints from the flat images.To create a precise flatfield matrix, a 1D spectrum can be generated by averaging all rows of the smile-corrected image.This 1D spectrum is then repeated in spatial dimension to create another artificial image.
A22, page 4 of 14 This artificial image is then "re-smiled", namely, shifted according to the offset map to re-introduce smile.The original input image is divided by it to retrieve what we call a dust-flat.Optionally, this process can be repeated multiple times.A good metric to determine how many iterations are needed is to stop when the difference between consecutive dust-flats is small.We have achieved relative differences with a standard deviation in the 10 −4 -range after three to four iterations.A slit flat is generated from the dust-flat.For this, all columns are averaged to a 1D spatial representation, removing all features that vary in the spectral dimension.This column is again repeated to the full image dimensions to create the slit flat.Subtracting the slit-flat from the dust-flat yields the sensor flat.Figure 4 shows the final dust-flat and the separated sensor and slit flats of the example in Fig. 1.The separation of the slit flat from the sensor flat allows us to track and correct "drifting" slit features (e.g., due to changes in instrument temperature and/or mechanical configuration) independently of the dust imprints on the sensor in the science data.Furthermore, it can work vice versa to remove slit features independent of changing features in the sensor flat, such as moving dust grains.Since the slit and sensor dust flat(s) are extracted from the input flat images without smile correction they can also be applied to science frames without the need of desmiling and interpolation.

Illumination field
The illumination field represents the smooth low order intensity variation.With spectroflat, this information is mainly included in the dust flat.Accordingly, the variation of the final spectroflat illumination fields are always in the 10 −6 range.Therefore, we did not apply these.However, we kept this step for compatibility with the Wöhl et al. (2002) approach, and for border cases where it might be needed.
From the dust-flat-corrected and desmiled input image, we then removed the line imprints using the standard mean profile method.We found it common to be left with some undesired, sharp line residuals due to spatial gradients within the lines.Similarly to Wöhl et al. (2002), we removed them by smoothing.This was done by first linearly interpolating over these residuals in spectral direction and then applying a Gaussian filter to the full frame.The result is an illumination field on the sensor.Figure 5 shows the corrected input image corresponding to Fig. 1, before and after line removal, and after residuals cleaning.The time stability of the illumination field is comparable to the stability of the smile-distortion pattern, thus, it can also be applied to science frames that have already been temporally averaged.

Wavelength calibration and continuum correction
As an optional step, the detected smile offsets can be amended by a wavelength calibration.The desmiling and the wavelength calibration of the science data can then be performed in a single interpolation step.
The input data can be calibrated using several solar atlases.The Hamburg Fourier transform spectrograph (FTS) atlas (see Neckel 1999) and the Second Solar Spectrum (SS2) atlas of Stenflo & Gandorfer (2014) are currently supported.Other atlases can be provided as a plug-in to the atlas-fit routine, for instance, to support other spectral regimes.First, a spectral re-sampling and convolution with a Gaussian of the atlas is carried out to match the input data sampling and resolution.Then, the sub-pixel spectral difference between the line centers in the data and the atlas is fitted using the same procedure described in Sect.2.2.The resulting additional offset information is added to the previously generated smile offset map.If the selected atlas provides information on the continuum intensity, the continuum calibration of the data is done as follows (see Fig. 6).
First, the difference between the maximum intensity and the continuum value specified in the reference atlas is estimated.For this, the atlas spectrum is divided into a configurable number of equidistant sections and the maximum point in every section is selected.Next, a high-order polynomial (configurable, default: 14, should never exceed the number of sections) is fitted to the selected maximum points.Outliers of the maximum points are filtered out if they differ more than 20% of the fitted polynomial.Then, we fit a fifth-order spline through the remaining maximum points.Finally, the difference between the spline interpolation and the reference continuum is computed and used as reference spectral continuum variation.
In a second step, for each individual row in the input data spectrum, the same approach specified above is used to find a spline interpolation of the measured continuum intensity profile.The computed reference continuum variation is added to the solution found for the input data spectrum.The correction of the solution with the reference continuum variation from the atlas prevents the overcorrecting of strong and blended lines.The resulting continuum correction solution is a 2D map and can be used in addition to or instead of the spectroflat illumination field.As for other flat-fields, the correction is multiplicative and normalized to one.
To allow for better fits of the wavelength solution and the continuum correction, the algorithm performs a least-squares fit of the effective full width half maximum (FWHM) and the gray stray light component of the Atlas to the data.Lines of known variable depth (i.e., telluric lines) can be masked out for this purpose.As a first order approximation the spectrographic PSF can be assumed to be a global Gaussian.Hence, we convoluted A22, page 5 of 14 Fig. 6.Spectral and continuum calibration using the FTS quiet-Sun, disk-center atlas from Neckel (1999).The top panel shows the uncorrected and corrected spectrum, the used continuum correction curve, the atlas reference continuum level and the atlas (see legend).The middel panel presents the difference between the atlas and the calibrated data in units of the above panel.The lowest panel shows the fitted dispersion offset in pixels.
the Atlas with a Gaussian function with FWHM equal to the differential FWHM between instrument and atlas.The atlas FWHM is assumed to be known, while the data FWHM is a free parameter.The other free parameter in the fit is the intensity offset introduced by a stray-light component (Martinez Pillet 1992).Figure 6 shows an example calibration result for the TRIPPLE-SP flat-field dataset.

Data reduction process
Once all calibration data is obtained using the spectroflat algorithm the reduction of a batch of science frames {F t } t∈T is usually carried out according to the method described below.To describe the integration into a full calibration workflow, we also include steps with calibration data obtained via other methods that are not within the scope of this work.For instance, non-linearity (NL) correction, tracking moving dust parts on the slit and sensor, and obtaining demodulation matrices from polarimetric calibration.
1. Apply NL and zero-point correction (e.g.dark field) to each individual frame: (2) 2. Apply spectroflat dust flats.Identify t with a modulation state, and select the corresponding modulated flat for each frame.If needed shift slit and sensor flat for each frame such that it matches the dust patterns best: 3. Sort data by modulation state s and bin N frames per state in temporal (i.e., scanning) direction: 4. Apply spectroflat smile distortion correction to averaged frames: s,∆t := desmile(F s,∆t ).
If the illumination field and continuum correction are both applied, they can be combined into a single "soft flat" matrix which will reduce steps ( 5) and ( 6) to a single operation.Since multiplication is numerically less expensive than division, it is advised to save all flat-field matrices as gain tables, namely, as 1 FF * .

Results and discussion
The spectroflat algorithm yields flats for sensor and slit dust imprints for each polarimetric modulation state.It has been tested with data from several long-slit spectrographs combined with temporally modulated polarimetry installed at high resolution solar telescopes.It further characterizes the smile distortion pattern precisely and provides smile correction methods for science frames.It is able to do spectral and continuum level calibrations by comparing with a reference atlas.
We applied our method to several datasets obtained with the facility instrument GREGOR/GRIS, the SST/TRIPPEL-SP prototype instrument, with SUNRISE-III/SUSI during pre-flight validation tests, and with SolO/SPICE.The wavelength regimes cover infrared (IR), visible (VIS), near-ultraviolet (UV) and extreme ultraviolet (EUV) datasets.The IR, VIS, and UV datasets feature absorption lines, while the EUV presents an emission spectrum.The spectroflat algorithm and library are suitable for instruments using CCD and CMOS sensors alike.The test datasets span cases from pure spectrographic (no polarimetric modulation) observations to observations with up to 12 states per polarization modulation cycle.
The test dataset from SPICE is not a regular flat-field; rather it was constructed from a quiet Sun raster map with full spectral coverage taken from the SPICE 3.0 data release (SPICE Consortium et al. 2023)4 .It is therefore not sufficiently spatially flat to perform a full analysis.However, we were able to perform a full smile correction and showcase the general applicability of the correction approach.Figure 7 shows the constructed pseudo-flat measurement and extracted dust flat.The result of spectroflat smile correction, compared to level one and two data products of the SPICE pipeline, is shown in Fig. 8.In all other datasets, spectroflat produced reliable and stable results, efficiently removing fixed pattern imprints from the science data, with only minimal configuration adjustments per instrument.The nominal flat-fielding procedure for GRIS is not suitable for our new approach of modulated flats.We were able to generate a non-modulated flat-field from the data and demonstrate the applicability of our method.However, for our new approach of having an individual flat-field map per modulation state, only a quarter of the 50 frames can be used per modulation state.To ensure sufficient spatial and spectral flatness we combined two subsequent flat-field recordings.
After applying the sensor and slit flats the fixed pattern imprints and the artifacts in the demodulated data are effectively A22, page 7 of 14 reduced by up to an order of magnitude.Figure 9 shows the spectral profiles of the demodulated flat-field frame from Fig. 1 before and after correction.The strong global slopes in the linear polarization states Q/I and U/I are removed.The residual I to Q/I, U/I and V/I cross talk and V/I to Q/I and U/I cross-talk around the stronger lines are not tackled by the algorithm presented here and need to be corrected on a per instrument basis.The sensor used in the TRIPPLE-SP setup features a strong column-tocolumn response variation, which is almost completely removed by the spectroflat algorithm.Such a heavily biased dataset is considered to be a worst case and its results should be evaluated separately from the rest.Demodulating the averaged flat-fields allows us to evaluate the polarimetric performance of the procedure.In short, no polarization is expected and the resulting Stokes components should be a flat line at zero with added photon noise.Thus, the smaller the variation and residual peaks, the better the result.With FF := {FF s } s∈{1,...,n} we denote the per modulation state averaged flat-field calibration frames, namely, the input image.With S = {I, Q, U, V}, we denote the demodulated and averaged Stokes science images.To prepare our data we remove the instrumental polarization background (mean offset from zero) after demodulation.The demodulation residuals, measured as the mean peak to peak value computed for several continuum regions (i.e., with no spectral lines present) before and after applying spectroflat to all datasets are given in Table 1.Another estimation of the residuals is given in Table 2, which presents the root-mean-square (RMS) variation of the full spectral profiles of the central row including absorption line regions.Both metrics show a reduction of fixed pattern imprints by up to an order of magnitude5 .
For the SUSI dataset the detector conversion gain used is 1.8 DN/e − .Assuming a Poisson distribution for the photoelectrons generated in the detector, the relative photon noise can be estimated via pn(FF) := where F i denotes the flux in individual flat-field frames in units of DN, and N the number of frames averaged.Comparison with the relative flat-field RMS, given by yields an estimate on fixed pattern imprints still present after correction.Since there is basically no line-free continuum region in the SUSI near UV dataset recorded at 412 nm, we analyzed ROIs with fixed height of 600 px (spatial dimension) and different widths (1-14 px, spectral dimension) in a quasi-continuum section.For a ROI width up to 6 px width (3 600 px total) the relative RMS is on the same scale as the estimated photon noise.
For larger widths weaker spectral line imprints contribute to the RMS measurement.Thus, we conclude that no relevant fixed pattern imprint is left above the noise level, after correction.
Figure 10 shows the relative RMS of corrected and uncorrected data in comparison to the estimated photon noise level.
To highlight the benefits of spectroflat, we compare it with the Wöhl et al. (2002) method applied to the same dataset.Some implementation details are not specified in Wöhl et al. (2002); thus, we did our best to select values that produce an optimal correction.The original algorithm presented by Wöhl et al. (2002) does not account for strong column to column variations.As noted above, the TRIPPLE-SP data is heavily influenced by such a column comp pattern.We have therefore added the same pre-correction step as described in Sect.2.1 to our implementation of Wöhl et al. (2002) algorithm, to allow for a fairer comparison.We note that spectroflat treats the modulation states of the averaged flat-field frames independently, while Wöhl et al. (2002) does work with a single mean image composed of all modulation states.Moreover, Wöhl et al. (2002) performed the division by the mean spectrum after smoothing and interpolation.In spectroflat, we do not apply any smoothing prior to the extraction of the flat-field information.This helps avoid residuals that originate from artificially broadened spectral lines.
A key difference between the methods is that spectroflat employs multiple lines for the smile correction, while A22, page 8 of 14 Hölken, J., et al.: A&A, 687, A22 (2024)  Wöhl et al. ( 2002) uses a single one.In our implementation of Wöhl et al. (2002), we selected the Fe I line at 5250.6 Å for smile detection, which is strong and located near image center.Moreover, Wöhl et al. (2002) do not remove dust imprints prior to the smile detection.This may cause errors in the line center detection.Figure 11 shows the offset and error-maps of the smile correction for Wöhl et al. (2002).In comparison to Fig. 2, the multi-line approach of spectroflat shows a better performance (by two orders of magnitude) in terms of residual error (see Eq. ( 1)).
A widely used metric for flat-field performance is the residual RMS variation in a relatively uniform area of the corrected flat-field frames (see Wöhl et al. 2002, their Table 2).Table 3 shows the RMS as a percent of the mean signal, computed within the rectangular box (as shown in Fig. 12).This metric translates to the signal-to-noise ratio (S/N) defined as the mean intensity divided by the RMS in the selected region.On the flat-fielded TRIPPLE-SP flat images (FF s ) this yields values of 14.6 for the uncorrected data, 69.9 after applying the Wöhl et al. (2002) method and 118.3 after spectroflat correction.And similarly, comparing spectroflat alone against the full GRIS pipeline, we get S/N values of 5.7 for the uncorrected data, 94.6 after applying the GRIS pipeline (including dual beam recombination), and 92.1 after spectroflat correction (single beam) for the GREGOR/GRIS dataset recorded in the 10 820 Å window.
For further comparisons of the spectroflat and Wöhl et al. ( 2002) methods, we examined reduced SST/TRIPPLE-SP Table 3. RMS variation in percent of the mean count in a sub region of TRIPPLE-SP data chosen such that strong spectral lines and strong dust impurities are avoided.

Method
s FF s /n FF dust FF illum I science Uncorrected 7.17% --10.98%Wöhl et al. (2002) 1.04% 5.32% 0.27% 9.89% Spectroflat 0.85% 6.84% 0.00% 9.08% science data.Figure 13 shows the I, Q/I, U/I and V/I spectra of a five-second averaged quiet Sun measurement.The upper row presents data that was only corrected for the camera zero-response as a reference.The strong sensor response pattern imprint and slit-dust impurities are clearly visible in Stokes I.All Stokes parameters show diagonal fringes and gradients across the FOV.These effects are drastically reduced by the spectroflat correction shown in the middle row.In comparison, Wöhl et al. (2002) correction (bottom row) does not fully remove the fringes and intensity gradients from the Q/I, U/I and V/I images.Also, we observe a residual column to column response pattern in I. Figure 14 shows spectral profiles at row number 150 of the images in Fig. 13.There is a reduction in fixed pattern imprints in all Stokes parameters for both correction methods.However, the residual imprints from the column to column variation remain visible after the Wöhl et al. (2002) correction.In the other Stokes parameters, the differences between both correction methods are minimal after division by I.
To test the influence of the flat-field and spectral calibrations on the value of the atmospheric parameters retrieved by the inversion algorithms typically employed in remote sensing, we performed a comparison using the Milne-Eddington (ME) HeLIx + inversion code (see Lagg et al. 2004Lagg et al. , 2009)).We applied it to the measurements shown in Fig. 13.To determine the atmospheric parameters, HeLIx + maximizes the FITNESS metric, namely, the inverse of the difference between measured and synthetic Stoke parameters, using a genetic optimization algorithm.See Lagg et al. (2004) for details on the HeLIx + algorithm A22, page 9 of 14  and fitting procedure.The exact propagation of residual flatfield errors to the inverted atmospheric parameters depends on the inversion code utilized.Therefore, we use a simplistic MEmodel and a statistical approach to test for the influence on the inversion robustness, rather than on individual results.We invert some selected spectral profiles from each case shown in Fig. 13 in 100 independent HeLIx + runs with random start parameters (within the defined bounds).The genetic code will determine slightly different best fits for each run.Hence, the comparison of peaks and footprints of the result distribution for basically uncorrected and the two reduced datasets can be used as a sanity and robustness check respectively.After removing the V cross-talk the linear polarization signal present in this dataset is below the noise level (see Fig. 14).Hence, we selected weights to ignore Q/I and U/I to avoid fitting artifacts triggering offsets in the magnetic flux B. All inversion results shown rely solely on I and V/I, which prevents the deduction of azimuth and inclination angles of the magnetic field.The histograms of the distribution of the results are given in Fig. 15, in terms of the distribution of the inversions of the Fe I 5250.6 Å line in row y = 150.Likewise, Fig. 16 presents the results for the adjacent rows, which have similar Stokes profiles.For all runs the quality of the fit (FITNESS) of spectroflat is better compared with the dataset calibrated with Wöhl et al. (2002) method.However, given that the differences in the corrected spectra of both methods are minimal, they provide similar inversion results, also in general agreement with the noisy raw data.The most prominent difference is the line of sight (LOS) velocity.The wavelength calibration was executed on the central image row (y = 600) for all datasets.For the uncorrected data, the smile was not removed, thus at row y = 150 we have a several pixel offset from the calibrated wavelength.
To further investigate the influence of the smile correction, we also studied lines where the smile solution of Wöhl et al. (2002) and spectroflat no longer agree, and their differences are more pronounced.Figure 17 shows the distribution of 100 individual inversion runs for row y = 150 for the Fe I 5247.0Å and Cr I 5247.5 Å lines.As both lines are formed under roughly the same atmospheric conditions, simultaneous ME inversions can be executed without penalties.Figure 18 shows the result for the adjacent rows which have similar Stokes profiles.The most dominant differences are again for LOS velocity which can be traced down to smile correction and wavelength calibration: We show earlier in this work that the smile distortion is not uniform (see Fig. 2).For the Wöhl representative dataset the smile correction was performed on the Fe I line at 5250.6 Å.For the results on that line (see Fig. 16) we have a mean difference of less than 100 m s −1 .However, in the 5247 Å region the smile solutions already show relevant offsets (see Fig. 19) leading to a mean difference exceeding 500 m s −1 .The difference in LOS velocity in the inversion results showcases the impact of residual error in the single line smile correction for larger spectral FOVs.

Limitations
As for flat-fields for pure imaging instruments, the input data to this algorithm has to be sufficiently uniform in spatial direction.Any remaining (solar) structure will be regarded as a flat-field effect and therefore be "corrected" out of the science frames.This would change the science data in an undesired way.Moreover, if the data is too noise dominated, the noise will affect the smile detection error metric (Eq.( 1)).As the spectroflat algorithm cannot distinguish between actual flat-field artifacts originating from the instrument and residual solar structures, the observer and user of the spectroflat algorithm have to ensure that the recorded flat-field data are averaged along a path around the disk center and sufficiently flat, depending on the requirements of a given science case.Both issues can be mitigated by integrating enough and sufficiently displaced frames.
The smile detection and wavelength calibration are aimed at calibrating the full FOV by applying row-wise changes.In order to do so, several spectral lines have to be tracked simultaneously.Hence, it requires multiple absorption or emission lines in the correction area; in particular, lines near the left and right borders of the correction area improve the result.
Finally, the spectroflat desmile algorithm is focused on detecting and removing the spectrographic curvature effect.Thus, other geometrical distortions (especially in spatial direction) need to be tackled separately.Also instrument specific cross-talk effects are beyond the scope of this algorithm.

Conclusion
We have introduced two publicly available python libraries.spectroflat to generate spectropolarimetric flat-fields and atlas-fit to amend the calibration data generated by spectroflat with wavelength calibration, along with continuum correction information against a given solar atlas A22, page 10 of 14      While both methods agree on the 5250 Å lines, they already start to notably diverge on the 5247 Å lines (350 pixels apart).
A22, page 12 of 14 Hölken, J., et al.: A&A, 687, A22 (2024) (Hölken 2023b,a).Both can be installed from the generic python pip repository, while the sources can be inspected and downloaded from a public Gitlab repository.Both libraries are agnostic to instrument specifics and can be included into existing python-based data-reduction pipelines.Technical description is available in the cited references.Our algorithms allow to obtain precise flat-field maps from nominal flat-field measurements recorded close in time to the science observations and using a representative instrument configuration with Sun pointing instead of an artificial light source.Thus, we avoid the shortcomings mentioned in Sect. 1.When applying our method we see a significant reduction of fixed pattern noise and residual artifacts.Furthermore, smile distortion effects are precisely registered and corrected.We have shown its applicability on a wide range of instruments, wavelength regimes and sensor types.A variation of smile-distortion as a function of wavelength is expected.However, we have been able to show that the smile distortion may vary less predictably along the spectral dimension, exceeding second-order distortions.Higher-order distortion effects, caused by optical manufacturing and alignment tolerances, are typically not reproduced by an optical model.Therefore, the currently prevalent approach (obtaining this information around a single line or from an optical model) is not suitable for large sensors targeting multiple solar lines simultaneously.Since multiple spectral windows without any telluric lines exist and telluric lines are not present at all in datasets from balloon-borne or space-based instruments, an approach using these lines for smile correction, as presented by Cai et al. (2017), was not suitable for this generic library.However, it might be of value to explore such a mode in a later release.
To prove the general performance of spectroflat, we examined the residual artifacts and the noise level of demodulated flat-fields from different instruments.We found the fixed pattern noise, measured as RMS or peak-to-peak variation in Stokes Q, U, and V, to be reduced by up to an order of magnitude by our method.Further, for a given science observation, we compared the calibration result obtained with our method with the result obtained with the method by Wöhl et al. (2002): -The fixed pattern imprint, quantified as the S/N of the Stokes I data, is significantly better suppressed by our method; -For the full frame we were able to achieve a better fringe and gradient suppression in Q/I, U/I and V/I; -The differences in the 1D Q/I, U/I, and V/I profile correction results of both methods are minimal; -The inversion results after both calibration methods agree within inversion error expectations, indicating that our method does not introduce systematic errors; -Higher fitness values for the inversions of the data obtained with our method indicate a better compatibility of the Stokes profiles with the atmospheric model used by the inversion algorithm; -The results on LOS velocity benefit strongly from a full FOV smile solution.For an in-depth analysis of the influence of the calibration methods on inversion results, a comparison against synthetic data is needed.This is planned for a follow up study.
Furthermore, our algorithm does not depend on an illumination field and works with sensor and slit flat alone.The smile correction can thus be applied on a temporally averaged frame, which significantly reduces the numerical costs of the flat-field correction, as compared to the original Wöhl et al. (2002) method.When applying the generic spectroflat to a GREGOR/GRIS single beam flat-field, the resulting S/N value is almost as high as after running the full instrument-specific GRIS pipeline.A detailed study on possible improvements of the GRIS pipeline using spectroflat is in preparation.For the SUSI test data, we where able to show that the fixed pattern imprints are suppressed down to the photon noise level.
The few configuration values required by our algorithm were found to be robust.In all tests based on a given instrument we were able to use the same instrument-specific configuration on all corresponding flat-field datasets.Due to its broad applicability and its model independence with respect to specific instrument properties and wavelength regimes, we believe that our method is applicable as a standard calibration approach for most modern large FOV long slit spectrographs with or without polarimetric capabilities.We welcome the solar-instrumentation community to use the code as is or to contribute to the code for even broader applicability.
Because of its robustness and clear benefits, including its relative numerical inexpensiveness, an on-board implementation on future space missions (with the goal of performing flat-fielding on board) appears feasible.This is a promising outlook since downstreaming fully reduced science data helps in reducing telemetry bandwidth significantly.Such a standardized pipeline approach will be explored with the (post-flight) data calibration pipeline of the Sunrise Ultraviolet Spectropolarimeter and Imager (SUNRISE-III/SUSI) on the Sunrise re-flight planned for summer 2024.
Fig. 1.TRIPPEL-SP input state image of the first modulation state is shown in panel a.Panel b shows the corresponding ad-hoc corrected image.Both color bars denote relative intensities in ADUs.

Fig. 2 .
Fig. 2. Offset map profiles shown in panel a for the colored rows indicated in panel b.Map of detected offsets in panel b.Color indicates the amount of shifting needed in units of pixels for each pixel along the x-axis to remove the smile distortion.Panel c shows a map of the pixel-wise error E(p min , x), and panel d the result of the error function as of Eq. (1) for the selected polynomials.

Fig. 3 .
Fig. 3. Intensity profile of different rows around the 5250.6Å spectral line.The prepared input data is shown in the upper panel, while the lower shows the intensity profiles after correcting the smile distortion.
Fig. 4. TRIPPLE-SP Dust-flat (a), separated slit flat with the horizontal slit features (b) and separated sensor flat (c).All color bars show normalized values.Range in all plots adjusted to increase visibility of features.
Fig. 5. Steps of the illumination field extraction: (a) desmiled input image, (b) illumination field, i.e., the desmiled input image after removing the spectral lines, showing vertical line residuals and (c) clean illumination field after reducing the line residuals.Note: panels b and c have different color scales to improve the visibility of the residuals.

Fig. 7 .Fig. 8 .
Fig. 7. SolO/SPICE L1 test dataset created from multiple quiet Sun science observations in the upper panel and extracted pseudo-flat in the lower one.The spectroflat algorithm cannot distinguish between actual flat-field artifacts and residual solar structures.Thus, these residual horizontal structures are regarded as flat-field effects.The vertical features in the flat are not line residuals but traces of the burn-in effect on the sensor.

Fig. 9 .Fig. 10 .
Fig. 9. Spectral profiles of a demodulated TRIPPLE-SP flat-field without D(FF ′ ) in panel a and with D(FF (1) ) and spectroflat applied in panel b.Left axes denote normalized Stokes I intensity while axes on the right hand side denote the relative polarization signal for Q/I, U/I, and V/I.

Fig. 12 .
Fig. 12. Position of the box to compute the RMS variation (Table 3) in one of the flat-field state images (left panel) and Stokes I of a science frames (right panel).Color bars give relative intensity in ADUs.The box is chosen such that spectral lines and strong dust impurities are avoided.

Fig. 13 .
Fig. 13.ROI from a five-second (500 frames) average of quiet Sun Stokes spectra from the 2017 SST/TRIPPLE-SP campaign.Panels in the upper row present data with no correction applied, while the middle row shows the same after spectroflat and our wavelength calibration are applied.The lower row panels show the same ROI, after the Wöhl et al. (2002) method is applied.Note: any instrumental cross-talk was not corrected in any of these datasets.

Fig. 14 .
Fig. 14.Spectral profiles at row number 150 of the images shown in Fig. 13.Top and bottom x-axis scales indicate the pixel position and the wavelength in Å, respectively.Note: the spectrum is flipped in wavelength direction with respect to the standard convention.

Fig. 15 .
Fig. 15.Histogram of 100 inversions run on the Fe I 5250.6 Å line at row 150 of the datasets shown in Fig. 13.

Fig. 17 .
Fig. 17.Same as Fig. 15 except that here the Fe I 5247.0Å and Cr I 5247.5 Å lines are inverted.

Table 1 .
Residuals in demodulated flat-field frames per Stokes parameter as mean peak to valley variation within a continuum region after dark field correction and after applying the spectroflat flat-field corrections to the flat-field data themselves.

Table 2 .
Same as Table 1 but for RMS variation of the full central row.