Multi-frame image restoration method for Luojia 1-01 night-light remote sensing images

ABSTRACT To address blur and noise issues caused by airlight and atmospheric scattering in nighttime imaging environment, we proposed a multi-frame image restoration method. First, the Luojia 1–01 night-light image degradation model was derived. Thereafter, the APSF (Atmospheric Point Spread Function) for night-light images was estimated. Improved dark channel prior and sparse constraint models were used to eliminate effects ofairlight and atmospheric scattering. Finally, a multi-frame sparse constraint model was established to eliminate image noise. The results show the proposed method is effective as it can reduce the image blur phenomenon, suppress image noise, and improve image quality.


Introduction
With greater socio-economic development and increase in human activities at night, nighttime lighting equipment has become increasingly popular. Night-light remote sensing satellites that can obtain information about visible light sources on land and water on cloudless nights are receiving more attention . Night-light remote sensing has the unique ability to reflect human social activities, so it is widely used in many fields such as mapping of urbanisation processes (Zhu et al. 2019), estimating socio-economic parameters (Elvidge et al. 1997, Li et al. 2013a, monitoring disasters (Gillespie et al. 2014) and armed conflicts (Li et al. 2013b), researching epidemiology (Kloog et al. 2008), and evaluating light pollution (Kamrowski et al. 2012(Kamrowski et al. , 2014. As part of a new generation of night-light remote sensing satellites, Luojia 1-01 possesses characteristics such as high spatial resolution, high temporal resolution, high radiation resolution, and high signal-tonoise ratio. Therefore, the applications for Luojia 1-01 night-light remote sensing images are broad and may be useful in many fields, such as evaluating artificial light pollution (Jiang et al. 2018), mapping urban extent (Li et al. 2018), estimating economic parameters (Zhang et al. 2019), simulating population distribution (Gao et al. 2019), and change detection (Li et al. 2019a). However, as a result of the impact of clouds and moonlight, several limitations exist for Luojia 1-01 night-light images (Jiang et al. 2018). In the process of acquiring night-light images, the Luojia 1-01 satellite is affected by many factors, such as airlight (He et al. 2011), atmospheric scattering (Narasimhan and Nayar 2003), satellite sensor vibration (Sudey andSchulman. 1985, Morio andKenichi 2001), satellite observation angles (Li et al. 2019b), and atmospheric disturbance. These factors result in a blurred degradation phenomenon for images, which leads to inaccurate analysis results when using night-light images for urban GDP assessment, population statistics, urban development level analysis, and other fields. Thus, there is a significant need for a method that can improve the quality of Luojia 1-01 night-light images, eliminate the effect of airlight and atmospheric scattering, and restore true brightness information.
Image restoration is an effective method of improving image quality in digital image processing. According to a priori knowledge such as the mechanism and process of image degradation, the real image is restored in order to suppress image noise and improve image quality (Gonzalez et al. 2004). Image restoration technology is divided into two categories, single-frame and multi-frame image restoration. Single-frame image restoration is used when processing a single image, and usually deals with degradation caused by noise or blur in the image. Single-frame image restoration is a serious ill-posed problem essentially, and it is often difficult to effectively restore an image to its ideal quality . Multi-frame image restoration is used to process a sequence of blurred images of a static object or scene, so as to obtain a single deblurred image (Huang 1984). Compared to single-frame image restoration, multi-frame image restoration has an advantage in that the quality of the restoration results is better and contains more abundant detail information (Dong et al. 2012). Compared with other night-light satellites, the Luojia 1-01 satellite travels in space at a speed of about 7 km/s, and captures an image every 5s. This satellite can take photographs continuously for 300s and up to 60 frames at a time. It can also fly 35 km in 5s, and the overlap of two adjacent images along the course is 225 km, with a course overlap degree of 86.54%. The Luojia 1-01 satellite parameters are shown in Table 1, and the photography diagram is shown in Figure S1. Although different visiting time of the satellite may impact the image content and quality (Li et al. 2020), Luojia 1-01 satellite takes images every 5s, the content of multiple images in the same area is basically the same. But the difference of image details and noise will lead to the difference of image quality. There is abundant complementary information among these overlapping images, allowing them to form a sequence image group. Furthermore, the data condition of multi-frame image restoration using complementary information is satisfied. Therefore, according to the data conditions of the Luojia 1-01 satellite, the multiframe image restoration method may be used to improve night-light image quality and obtain restoration results with abundant detailed information.
Multi-frame image restoration has two main components: PSF estimation and the restoration model. At present, there are many methods for PSF estimation in various imaging environments. The multi-scattering APSF estimation method for daytime imaging environments was researched by Qiu et al. (2017). For nighttime imaging environments, the APSF estimation methods were researched taking into consideration different weather conditions, such as haze, mist, and rain (Metari andDeschênes 2007, He et al. 2013). The PSF estimation method for light source degraded images was researched by Narasimhan and Nayar (2003). Aimed at multi-frame image restoration methods, El-Khamy proposed an improved Wiener filter restoration algorithm for multi-frame images to eliminate image blur and noise (El-Khamy 2005). A multi-frame blind convolution restoration method based on sparse constraint was researched by Dong et al. (2012), while a multi-image de-blurring algorithm based on collaborative restoration was proposed to process multiple blurred images by . An improved multi-frame blind deconvolution restoration model was proposed to deal with the image oversaturation phenomenon by Ye et al. (2016). After eliminating airlight from the image using the dark channel prior, a multi-frame image restoration model with Bayesian posterior probability was used to restore the image (Wang et al. 2019). In low-illumination image restoration processing, the dark channel prior method is advantageous for eliminating the airlight effect. Meanwhile, the regularization method can reduce the sensitivity of the model to noise and make full use of the prior information of the image. Therefore, dark channel prior and regularization methods are both widely used in image processing. The majority of the aforementioned restoration models used L 1 norm as the regularization term, but solving the L 1 regular term in the process of model solving is often difficult and slow. The L 0.5 norm has a better constraint than the L 1 norm, and the L 0.5 /L 2 norm is the standardized form of the L 0.5 norm, which has scale invariance. Therefore, a restoration model of the regular term of the L 0.5 /L 2 norm is proposed, which can be more easily optimised. In summary, in response to the problems of blurred degradation and noise in nightlight images, along with the characteristics of Luojia 1-01 images, we propose a sparse constrained multi-frame restoration model based on scattering and reflection characteristics. The multi-frame image restoration method is used to enrich the image details, reduce the image noise, and finally improve the image quality.

Night-light image degradation and the degradation model
According to the working mode, remote sensing satellites can be divided into common satellites and night-light satellites. Common satellites are those that work during the daytime. Both common satellites and night-light satellites can perform long distance imaging. However, due to the unique nighttime imaging mode of Luojia 1-01, differences exist between common remote sensing images and night-light images in regards to the blurred degradation process. Common remote sensing satellites primarily take images of ground features in the daytime, such as urban roads, buildings, and cars. However, nightlight remote sensing satellites use the light brightness information of various visible lights on the ground in the nighttime. For common remote sensing images, blurred degradation is mainly caused by atmospheric absorption and airlight luminescence ( Figure S2). During the satellite imaging process, because of atmospheric absorption, the radiation intensity of the reflected light from the scene is weakened when it is transmitted from the scene point to the observation point (Wang et al. 2019). At the same time, a large number of suspended particles in the air act as active light sources, which transmits the ambient light of the scene, such as sunlight and surface reflection, to the imaging device, resulting in image degradation. These suspended particles, which essentially act as active light sources, are called airlight (He et al. 2011). The common remote sensing image degradation model is shown in Equation (1): where, IðxÞand JðxÞare the observed image and the original clear image respectively; tðxÞis the transmittance; A represents airlight; JðxÞtðxÞ is the direct attenuation term of the incident light; and finally, Að1 À tðxÞÞ is the term of airlight luminescence (Tan 2008). The transmittance tðxÞcan be expressed as shown in Equation (2): where, β represents the scattering coefficient; d is the scene depth; and tðxÞ is in the range of 0 to 1. As tðxÞ becomes larger, more light reflected from the scene surface reaches the observation point.
Night-light remote sensing images mainly show visible light information. The spatial resolution of the Luojia 1-01 satellite is 130 m. This means that each pixel in a Luojia 1-01 night-light image is the sum of light brightness information across 130 m�130 m. In an ideal situation, every light source on the ground is independent, and lights from different light sources do not interfere with each other. Actually, lights from each source are scattered by particles in the atmosphere at night, causing the lights to interact with each other. In the near ground, multiple scattering phenomena are produced, while in the far ground, single scattering phenomenon is produced. Glow phenomenon is caused by atmospheric scattering and leads to further blurred degradation of night-light images. Therefore, the blurred degradation factors of night-light images include atmospheric absorption, airlight interference, and atmospheric scattering. The degradation diagram for night-light images is shown in Figure 1.
In summary, in order to eliminate the effect of airlight, weaken atmospheric scattering on image quality, and obtain the true brightness information of the light source, a Luojia 1-01 night-light image degradation model is proposed in this paper, as shown by Equation (3): where, k is the atmospheric point spread function APSF, and n is the noise caused by the vibration of the satellite sensor. This model can effectively reflect the degradation of night-light satellites in the process of capturing images. It can also provide ideas for improving the quality of night-light images. The estimation of transmittance t(x) and airlight A in the model is introduced in Section 3.1, and the APSF estimation method is introduced in Section 3.2. After using the improved dark channel prior to estimate t(x) and A, Equation (3) can be rewritten as Equation (4): Let g(x) =ðIðxÞ À Að1 À tðxÞÞÞ=tðxÞ, then Equation (4) can be rewritten as Equation (5).
where, g(x) is the image without airlight interference. According to Equation (5), after eliminating the airlight effect, solving the clear image J(x) becomes an image restoration problem that can be solved using deconvolution, which weakens the atmospheric scattering k.

Improvement of dark channel prior algorithm to eliminate airlight effect
In the degradation model of common remote sensing images, the dark channel prior is often used to eliminate the effects of airlight from satellite images. The dark channel prior was proposed by Kaiming He (He et al. 2011) and was further developed by Li et al. (2017) with the aim of improving image processing affected by blurring. The change of detail (CoD) prior was proposed by Li et al. (2015) to better maintain image detail information.
For images without haze in most non-sky local areas, there is always at least one colour channel with a very low value for some pixels in most non-sky local areas, which refers to the dark channel prior. That is, the minimum light intensity in this region is a very small value (He et al. 2011). The dark channel prior is defined by Equation (6): where, c represents one of the three channels r, g, and b; J c is a colour channel for image J; and Ω x ð Þ is a local block centered on x. If J is an outdoor image without haze, except for the sky region, the intensity of J dark is low and approaches 0, which means that J dark is the dark channel of J (He et al. 2011). The dark channel prior is mainly used for RGB images. First, the minimum value of the grey value in three channels for each pixel in the image is taken to obtain a grey image. Then, the final dark channel image is obtained by minimum filtering of the grey image. Luojia 1-01 night-light remote sensing images are singlechannel grey images with only one channel each. The value of each pixel in the Luojia 1-01 night-light image is the minimum value for the corresponding position pixel. The dark channel prior was simplified in this study to meet the data condition of Luojia 1-01 night-light images (Equation (7)): In regards to airlight A (Equation (3)), the brightest 0.1% pixel at the top of the improved dark channel is selected, and the point with the largest pixel value is used for the value of airlight A. The calculation equation of t(x) in the Luojia 1-01 night-light image is shown in Equation (8): where, ω is the coefficient and the range is 0 to 1. The larger the value of ω, the better the airlight elimination effect. The obtained airlight A and transmittance t(x) are put into Equation (3) to eliminate the airlight effect in night-light images, and then g(x) is acquired.

APSF estimation method for night-light images
In Equation (5), the APSF of the image is usually unknown, but it can be estimated using the prior information in the image. Night-light satellites mainly photograph visible light sources on the ground. During the nighttime imaging process, the light from these active light sources results in significant scattering by atmospheric particles, resulting in the glow phenomenon (Bu et al. 2019). The glow phenomenon can effectively reflect the scattering characteristics of light from ground point sources in homogeneous atmospheric media at night and thus reveal the effect of the environment on the image. The APSF of the image can be estimated using the glow of independent point objects. By using independent point objects, the APSF value of night-light images under a variety of weather conditions was estimated (Bu et al. 2019). The APSF estimation method is shown in Equations (9)-(12): g m ðTÞ ¼I 0 e À β m TÀ α m logT (10) where, IðT; μÞ refers to APSF; g m (T) is the light attenuation in different weather environments; L m μ ð Þ is the m-order expansion of Legendre polynomials, which represent the light diffusion caused by multiple scattering; μ is the cosine value of the light scattering angle θ. I 0 is the intensity of the light source, and T refers to the atmospheric optical thickness. α m and β m refer to coefficients, and q is the forward scattering parameter, which ranges from 0 to 1. The value of q changes according to different weather conditions ( Table 2). The initial value of k can be obtained using the model. However, due to differences in imaging quality of independent point objects and randomness of selected point objects, there is still some deviation in the APSF value. Thus, it should be further updated and modified in the restoration model.

Multi-frame restoration model of night-light images
In Equation (5), k and g(x) can be used to solve the original clear image J(x). However, it is an inverse problem, and the solution is often not convergent due to the influence of noise n. There are many advantages to the regularization method, such as high convergence stability, better use of prior information, and improved ability to suppress image noise. Therefore, the regularization method is more suitable for night-light image restoration. In most of the regularized restoration models, the L 1 norm is used as the regularization term, and the image is constrained by minimising the norm value. However, the L 1 norm is a scale variant. When a blurred image is processed, L 1 norm will reduce the highfrequency information of the image, resulting in further blurring of the image. In the process of image solving, the L 1 norm regularization term has the additional problem of slow solution speed. Therefore, a sparse constrained multi-frame restoration model based on the L 0.5 /L 2 regularization term was proposed in this paper (Equation (13)). The L 0.5 /L 2 norm is scale invariant. It can keep the boundary information of light in the image, and it has a better constraint effect on the image and faster solution speed. The L 1 norm constraint was added for the APSF term to increase the contribution to the restoration results. Meanwhile, the L 1 norm is used to replace the L 0 norm to deal with the NP-hard (non-deterministic polynomial) problem of the L 0 norm while solving. As a result of the limitations of the single-frame image restoration method, there are some issues with the restoration results, such as less detailed information enhancement and decreased noise suppression effects. Because of the unique multi-frame data conditions of Luojia 1-01, we can use the complementary information between frames to obtain high quality images with more detailed information and less noise. The multi-frame restoration model of night-light images is as follows: where, i is the sequence number of multi-frame images; s is the total number of multiframe images; u and v i represent clear and degenerate images respectively; k is the APSF of the images; and finally, v i À k � u 2 2 stands for the fidelity term, which can guarantee minimum training error with the original data during the data calculation process. The second term is the L 0.5 /L 2 regularization term. It is a sparse metric constraint that promotes the scale invariance of u. The third term represents the APSF constraint term, which ensures that the test error of the model is small. γ and λ represent the weights of the fidelity term and APSF regularization term respectively. When γ becomes larger, the denoising effect is greater and the image is smoother; when λ becomes larger, the effect of noise suppression on APSF is better. Because there are two variables in this model, it cannot be solved directly. Therefore, the method of alternating minimization (Cho and Lee 2009) was adopted to solve the model. The optimal result is obtained by alternating updates between u and k. Thus, Equation (13) was divided into two independent problems (Equations (14) and (15)), which solve the clear image u and the optimal APSF values respectively. In the process of image solving, the iterative shrinkage threshold method (Wu and Luo 2013) is primarily used, whereas the iterative least square method (Rubin 2006) is used to solve the APSF.
The recovery process of Luojia 1-01 night-light remote sensing images is shown in Figure 2, and the detailed experimental steps are as follows: (1) Multi-frame night-light remote sensing images are input; (2) Image registration is used to make the size and position of the images identical; (3) Airlight A and transmittance t(x) for each image are estimated; (4) The improved dark channel prior is used to eliminate the effect of airlight on the images; (5) The light source intensity I 0 is obtained and the atmospheric optical thickness T is calculated according to the input image; then, the forward scattering parameter q is estimated according to weather conditions; (6) I 0 , T, and q are put into the atmospheric point spread function model to estimate the APSF value for each image; (7) The APSF and parameters are input into the multi-frame restoration image model; (8) The result is output and its quality is evaluated.
In regards to the quality evaluation of the experimental results, images with high definition, more detailed information, and less noise were better in vision. Tenengrad and average gradient were used in quantitative evaluation (Zhai et al. 2011). Tenengrad is an image evaluation index used to measure the sharpness and edge information of an image, while the average gradient value not only measures image sharpness, but shows the small detail contrast and texture transformation features. The larger the values of the two indicators, the better the image quality.

Experimental data
The experimental area should be sufficiently large, and the photographing conditions should be basically the same. Therefore, we selected six Luojia 1-01 night-light images in total: three of Tangshan, and three of Tianjin. The images were taken continuously in clear weather on 29 October 2018 in the experimental area (Figure 3), the characteristics of which are listed in Table 3. To further prove the effectiveness of the method, three additional night-light images of Tianjin were obtained; these were continuously captured in haze weather on 26 September. Generally, the forward scattering parameter q and atmospheric optical thickness T of APSF estimation in clear weather are 0.2 and 1.2 respectively. In haze weather, q and T are 0.75 and 4, respectively (Narasimhan and Nayar 2003). The 3D result of APSF estimation in clear weather is shown in Figure 4.

Results and analysis
Figures 5-6 show the experimental results for Tangshan, while Figures S3-S4 and Figures  S5-S6 show the clear weather and haze weather experimental results for Tianjin, respectively. Figure 5(a-c) show three blurred images after image registration. The improved dark channel described in this paper was used to eliminate airlight for Figure 5(a-c), with the results shown in Figure 5(d-f), respectively. The single-frame image restoration method (Bu et al. 2019) was used to recover the Figure 5f, and the result is shown in Figure 5(g). Figure 5(h) shows the average result of three recovery images. Figure 5(i) shows the result of the multi-frame image restoration method described in this paper, while Figure 6 shows the detailed results of the corresponding images in Figure 5.
In order to verify the validity of the method, a small area of Tianjing experimental results under haze weather was chosen and three-dimensional curves were set up for comparative analysis ( Figure S7). Meanwhile, the tenengrad function and average gradient were used to analyze the experimental results. The indicator analysis results are shown in Tables 4-6. Figures 5-6 and Figures S3-S6 show that compared to the original image, the overall image quality is significantly improved after elimination of airlight. The image degradation caused by airlight luminescence is reduced and the image becomes clearer, indicating that the airlight elimination method is valid. However, the image still contains effects of the glow phenomenon. After restoration, the image quality is further improved. The restored image is much clearer and brighter, and the glow phenomenon is reduced.  Additionally, the detailed information of the restored image is increased, and the edge contour is more distinct. Although the average result of the restored images can suppress image noise, due to differences in the quality of each image, the image quality of the average recovery result is still different from the single-frame restored result, so it cannot achieve a good processing effect. In terms of visual effects, the proposed method is the best as it not only provides more detailed image information, but also suppresses noise and provides better restoration results. At the same time, the changes in the experimental results in haze weather are more obvious than those in clear weather. It indicates the proposed method is effective. From Figure S7, we can see that although the three original images are relatively consistent overall, there are significant differences in some of their details. It indicates that a single image cannot provide comprehensive image information, which will affect the accuracy of later indicator estimation. Therefore, multi-frame image restoration is necessary. After eliminating airlight and weakening its influence, the image as a whole becomes darker. Compared with the restoration results, we find that the restored images become sharper and the brightness of the light increases significantly. It indicates that the glow phenomenon in the image is weakened, the edge information is enhanced, and the image is clearer. At this time, the light brightness value is closest to the true brightness value of the light source. The brightness information of the multi-frame restoration image is more than that of the single-frame restoration image, which indicates the multi-frame restoration image contains more abundant brightness information and detail information, and the quality of the multi-frame restoration image is better.
According to the evaluation results of the two indicators from Tables 4-6, the two indicators of the image after eliminating airlight are better than original images, which means that the airlight elimination method can improve the night-light image quality. At the same time, the two indicators of restoration results are improved, proving that the image quality is better following restoration. Compared with the three restoration results, the two indexes of our method are the best, as this shows that the proposed method has more advantages in enriching image information and suppressing image noise.
Therefore, from the perspective of the restoration effect and objective evaluation indicators, the result of our method is better than other images in visual evaluation and indicator evaluation, which indicates that the image quality is improved after restoration and the data processing method is feasible.

Conclusions
By analyzing the reasons for the blurred degradation of Luojia 1-01 night-light images, a multi-frame image restoration method based on Luojia 1-01 night-light images   Figure 5 degradation model was proposed. The experimental results show that the data processing method used in this paper is feasible. The effects of airlight and the glow phenomenon caused by atmospheric scattering are significantly reduced in the recovery image. Compared with the result of single-frame image restoration, the noise suppression effect is more obvious, detailed information of the image is more abundant, and the image quality is improved. It provides a more effective scientific method for the research of multi-frame restoration methods. The recovery image can provide a more accurate data basis in some applications, such as the estimation and analysis of socio-economic indicators and extraction of urban exent, and it is more conducive to data mining of night-light data.

Disclosure statement
No potential conflict of interest was reported by the author(s).