Snapshot high dynamic range imaging based on adjustable attenuation microarray mask

High Dynamic Range Imaging Based on Attenuation Microarray Mask has broad application prospects due to its good real-time performance and small size. But at the current level of craftsmanship, it is hard to fabricate a microattenuation array mask whose attenuation rate is adjustable. This leads to the fact that the imaging dynamic range cannot adapt to changes in scene brightness in most cases. To this end, this paper proposes a novel imaging system whose dynamic range can adaptively change according to the brightness of the scene. The core components are the micro polarization array mask mounted on the CMOS surface and the on-sensor rotatable linear polarizer in front of the lens. By controlling the rotation angle of the polarizer placed before the lens, the CMOS pixel exposure can be precisely controlled. Therefore, the imaging system dynamic range can be adjusted adaptively according to the scene brightness. Through horizontal comparison with Sony’s multi-quadrant polarization chip, we determined the optimal parameters of the multi-quadrant micro-polarization array mask for extending the dynamic range. The experimental results show that the imaging performance remains consistently good even when the dynamic range of the photographed scene is large, and the dynamic range of the device can be adapted to the dynamic range of the shooting scene, so that the image after shooting and processing can always show sufficient details. By rotating the polarizer in front of the lens to a specific angle, the high dynamic imaging of the scene can be significantly improved.


INTRODUCTION
High dynamic range imaging (HDR) [1,2] technology is currently one of the research hotspots in the field of computational imaging.The term dynamic range describes the brightness ratio between the brightest and darkest part of an image without losing image detail.The higher the dynamic range of the imaging device, the greater the range of scene radiances that can be captured, which in turn can present more target detail.Existing image detectors and display devices typically provide 8 bits of brightness data at each pixel and its dynamic range is only about 48dB, which is much smaller than the dynamic range that the human eye can perceive (about 120 -150 dB).This is the main reason why many conventional pictures tend to look less attractive than the real scene that the human eye sees.
Designing and developing detectors that can record a wide range of brightness variations is the ultimate solution for improving the dynamic range of imaging equipment.In this regard, Varnava C et al [3][4][5] have done a lot of excellent work.However, it will take some more time for this technical route to mature and practical as its high cost, low image resolution and effects.
In order to, the most obvious approach for improving the dynamic range is to capture images with different exposures for the same scene and sequential exposure [6,7] is the most commonly used method, which capture multiple images (usually more than 3) continuously at high speed in an instant.The exposure for each image is different.Short-exposure images focus on capturing detail in bright areas, while long-exposure images are mainly used to capture dark areas.Then the image sequences are fused with a tone mapping algorithm [8][9][10][11] to enhance the overall scene dynamic range.The method has a simple hardware system, so it has been widely used in camera apps for smartphones.But it requires the target to be stationary, otherwise not only is the process of image alignment complicated, but artifacts are inevitable in the fused picture.In addition, multiple exposures limit the frame rate of the output HDR image.So, its video is not smooth.
In order to overcome the weak real-time performance of the above sequential exposure method, multiple sensors are used to capture images with different exposures for continuous HDR video output.For instance, beam splitters are employed to generate multiple copies of the same scene with different luminance levels and each copy is detected by different sensor [12] .This method requires multiple CMOS and precision-aligned optics, which is difficult to achieve.
Another method that can capture multiple images with different exposure using only one CMOS has been proposed.In 2000, SK Nayar et al [13] proposed the method of dividing focal plane with attenuation film array, that is, the CMOS surface is covered with a microarray attenuation film to make the exposure of adjacent pixels different.However, due to the limitations of the fabrication level at that time, the optical crosstalk between pixels was difficult to solve, and only simulation results were given in the literature.Furthermore, in the 2009 work of Todor Georgiev et al [14,15] , HDR imaging of a single camera is achieved by aperture encoding with the structure of light field camera.However, due to the limitation of the light field structure, the resolution of the resulting image is low, and there is a significant mosaic effect.
In addition to the quest for real-time HDR, the actual HDR imaging results are also crucial.Because simply pursuing the imaging dynamic range does not make much sense [1,6] : a camera with a dynamic range of up to 150dB acquires images that are not always better than those of ordinary cameras.For example, to display the same scene, an image with higher dynamic range means that the difference in its pixel gray values is smaller.The image contrast may become lower and the details are not obvious.
The best results come from the perfect matching of the dynamic range of the device and the scene being filmed.Therefore, it is tempting to be able to adjust the dynamic range of the imaging system in real time to match the shooting scene.To achieve this, the main approach at present is using a liquid crystal space light modulator (SLM) [17] or DMD [18,19] to control the irradiance in different areas of the scene to acquire the best global exposure.however, the drawbacks of this method are also obvious: first, the whole system is sophisticated and expensive; Secondly, the closed-loop response speed of the system often cannot keep up with the brightness variation of scene, and the oscillation effect occurs; Finally, the output image resolution is limited by SLM and DMD.
In summary, a perfect HDR imaging protocol has not yet emerged.However, the key technical indicators pursued in this field are clear: real-time, low cost, small size, dynamic range can be adjusted according to the shooting scene.Work in this paper seeks to have breakthroughs and optimizations in these aspects: first, in order to solves the problem of real-time, low cost and small size, our dividing focal plane method employed the CMOS covered with a micro-attenuation array to acquire 4 images of different exposures with one snapshoot; second, The CMOS pixel exposure can be precisely controlled by placing a rotatable polarizer in front of the lens, allowing for adjustable dynamic range.

THEORY AND METHOD
It is difficult to fabricate an adjustable attenuation microarray mask with conventional optical coating processes.But the adjustable attenuation rate can be achieved indirectly by micro-polarization array mask [20] .The principle of an imaging system with adjustable dynamic range is shown in Figure 1 [21] .In Figure 1, a rotatable polarizer is placed in front of the lens, and the light emitted by the target becomes linear polarized after passing through the polarizer.That is to say, the polarization direction of the light emitted by the target can be controlled by adjusting the rotation angle of the polarizer.The camera's CMOS sensor is covered with a micro-polarization array which is comprised of micro-polarizers with a specific angle on each pixel.4 Adjacent micro-polarizers form a computational group and each of them has a different polarization direction.According to Malus law, the rotation of the polarizer in front of the lens changes the light intensity each pixel received, and the light intensity variation is jointly determined by the rotation angle of the polarizer and the polarization direction of the micro-polarizer before the pixel.For example, suppose that the polarization direction angle of the polarizer in front of the lens is θ, angle of the micro-polarizer before pixel is β, and the incident light intensity is I 0 , then the actual light intensity received by the pixel I is: Assume that the extinction ratio of the polarizer is 1000:1 and that the four polarization directions β are 0°, 71.6°, 84.3° and 90° respectively.When the polarization direction θ of the linear polarizer in front of the lens is 0°, the ratio of the incident light intensity of the four types of pixels is 1 : 0.1: 0.01: 0.001, at which point the dynamic range is enhanced by approximately 60 dB, a value that can be calculated using the following general formula [13] . = 20 × ( When the four polarization directions are 0°, 71.6°, 84.3° and 90°, the incident light intensity of the four types of pixels are balanced so that the image quality of targets in different brightness conditions in the scene can be taken into account.Assuming a dynamic range of 60 dB for the camera itself, the final dynamic range can be up to 120 dB. It is assumed that the micro-polarization array consists of four types of micro-polarizers with different angles of β.When the polarization direction θ of the linear polarizer in front of the lens is rotated from -180° to 180°, the incident light intensity received by the pixels behind each of the four types of polarizers changes according to the above equation, resulting in a change in the maximum detectable light intensity, as shown in Figure 2, where the vertical coordinates are the light intensity transmission rates received by the four types of pixels.As a result, the dynamic imaging range changes.Since the increase in imaging dynamic range is equal to the ratio of the maximum to minimum transmission of light intensity received by the four types of pixels, the exact increase will vary with the rotation of the polarizer in front of the lens.Assuming that the polarizer in front of the lens is rotated from -180° to +180°, again using the four β values above, we have calculated a curve of the increase in dynamic range with the angle of the polarizer and the results are shown in Figure 3.The curve shows that the upper limit of the dynamic range increase is 60dB, which is determined by the extinction ratio of the polariser (1000:1) and has nothing to do with the exact value of θ.However, it is worth noting that the curve in Figure 3 changes more dramatically when the linear polarizer at the front of the lens is rotated around 0°, due to the large variation in the ratio of the fluxes of the four types of pixels at this point, as can also be observed in Figure 2.This means that , when θ is rotated around 0°, a more balanced distribution of the different pixel attenuation amounts is obtained than when it is rotated around -90°, and the quality of the output high dynamic image is better.Therefore, in order to make the adjustment of the dynamic range of the imaging system more sensitive and effective, the polarizer in front of the lens can be restricted to oscillate within a small angular range centered at 0°, e.g.[-40° 40°].The next step is to determine in real time to which angle the polarizer in front of the lens needs to be adjusted to obtain the best image quality.The conventional strategy is to evaluate the image quality of the four images received simultaneously and to guide the rotation of the polarizer by evaluating its exposure intensity.As image evaluation algorithms tend to be time consuming, this method can be slow and prone to poor imaging due to inaccurate polarizer angle adjustments.
Therefore, we abandon the closed-loop feedback scheme.We adopt a strategy where the camera continuously acquires images at a high frame rate (e.g., 70 fps) and the polarizer oscillates back and forth over a specific angular range (e.g.[-40° 40°]), and then extracts the best frames from the acquired image sequence.In this way, the complexity of the algorithm can be significantly reduced.
The specific method for determining whether the imaging quality of the four images is optimal is as follows.
First, set the upper and lower limits of the appropriate exposure grey value, within which the pixels are considered to be properly exposed.
Then, the number of suitably exposed pixels in the 4 images with different polarization modulation directions is calculated, i.e. if any of the 4 pixels in the same coordinates in the 4 images are suitably exposed, we consider the pixel value to be valid, otherwise, if all 4 pixels are not within the threshold of suitable exposure, then the pixel point is invalidly exposed.The number of all exposed invalid pixel points in the image is counted.
Finally, the image with the fewest exposure invalid points among a group of adjacent frames is found and is considered the best exposed image.Only the selected best-exposed image is subjected to HDR fusion.Due to the difficulty of manufacturing a camera as shown in Figure 1, for the time being we have considered using a Sony polarization camera for this experiment.Sony's polarization chip has four polarization directions of 0°, 45°, 90° and 135°.Therefore, when the polarization direction of the linear polarizer in front of the lens is rotated from -180° to 180°, the variation curve of the light intensity transmission rate received by the four types of pixels is shown in Figure 4. Comparing Figure 2, it can be found that: due to the uniform spacing of the four polarization direction angles of the Sony polarization chip, the corresponding periodic variation in transmittance is more pronounced.
Figure 5 shows the curve of the increase in dynamic range of the system with one rotation of the polarizer in front of the lens (i.e., from -180° to 180°) when using Sony polarization chip.
Figure 5.The increased dynamic range of the imaging system varies with the angle of rotation of the polarizer with Sony polarization chip A comparison of Figures 3 and 5 reveals the following two points: firstly, the dynamic range is more monotonous, due to the four polarizations directions on the chip, which make the ratio between light intensity transmittances less flexible.Secondly, although the upper limit of dynamic range improvement in Figure 5 is 60dB, the lower limit is higher compared to Figure 3.The lower limit in Figure 3 is approximately 6dB, while the lower limit in Figure 5 is 16.7dB.This indicates that the CMOS pixels have a greater range of dynamic range improvement when adopting the four polarization directions in Figure 1, which makes the polarization directions in Figure 1 more reasonable and more adaptable to the brightness variations of the actual scene.

EXPERIMENTAL RESULTS AND DISCUSSION
Our experimental system is shown in Figure 6.The camera model is BFS-U3-51S5PC-C from Flir, and its sensor chip is a color polarizing chip from Sony, model number: IMX250MYR-C.The motorized rotary stage is a Zaber product with a through-hole in the middle, model number: X-RST120AK-E03.The polarizer is an Edmund Optics product with the polarization extinction of 1000:1 in the visible band.The polarizer is mounted in the through-hole in the middle of the stage and rotates when the it is started.The camera lens is positioned at the same height as the center of the through-hole in the center of the stage and images the distant target through the through-hole.The polarizer at the through-hole of the stage is thus positioned just in front of the lens.To ensure smooth rotation of the polarizer, a small gap is left between the rotary stage and the lens.Figure 6.Picture of the experimental system prototype Firstly, the system was used to capture a static scene to test its high dynamic imaging performance.The camera captured images at the highest frame rate (71Hz) while the polarizer in front of the lens rotated at a uniform rate, and a series of raw images were obtained.From each raw frame, four images can be extracted (each image corresponds to one polarization direction.The brightness of each image is different due to the polarizer in front of the lens).In a series of images, three typical sets of images were selected for HDR fusion processing.The results of the fusion are shown in Figure 7, where it can be seen that the amount of information in the fused images differs when the angle of the polarizer in front of the lens is changed.Note the difference in cloud detail in the sky in Figure 7 between the three image species.As HDR fusion algorithms are not the focus of this paper, we used the well-established commercial software Photomatrix Pro to perform the fusion process, the interface of which is shown in Figure 8.In order to ensure the consistency of the image fusion algorithm, we have adopted the "Realistic" style and kept the parameter settings exactly the same, as shown in Figure 8.A more detailed result is given in Figure 9.When the polarizer in front of the lens is oriented at 0°, the brightness of the image in the 0° polarization direction is the greatest among the four sub-images obtained by the polarization camera, while the brightness of the image corresponding to the 90° polarization direction is the least, when the light transmission rate is about 1‰ and almost all detail information is lost.45° and 135° polarization directions correspond to the same brightness of the image.As can be seen from the image fusion results: some of the detail information in the sky is missing, as the sky areas are either underexposed (for the images corresponding to the 90° polarization direction) or overexposed (for the remaining three frames) in these four images.In the rightmost black and white image, the white area indicates the part of the image fusion result with reasonable grey values, while the black area is the part of invalid pixels due to poor exposure (the image information within this area is lost).As the polarizer in front of the lens is rotated to -2° and -4° respectively, the brightness of all four subimages changes accordingly, most notably in the 90° polarization direction, which gradually improves the detail in the sky part of the scene, especially in the cloud cover.The black and white image on the far right shows a significant increase in the area of reasonably exposed areas: when the polarizer in front of the lens is rotated to -2°, only a small part of the sky is overexposed and the windows of the building are underexposed; when the polarizer in front of the lens is rotated to -4°, there is no loss of detail in either the sky area or the windows of the building, and the grey values are in a reasonable range throughout the image.Figure 9. HDR image fusion results when the polarizer in front of the lens is rotated to different angles.It can be seen that the brightness of the four sub-images is related to the angle of the polarizer, which leads to differences in the fusion effect.The black and white binary image shows the areas of the corresponding fused image that are suitably exposed or not.The white areas represent the well exposed parts and the black areas represent the parts where information is lost due to overexposure or underexposure.

CONCLUSION AND DISCUSSION
The above experiments verify that this system can achieve real-time imaging with adjustable dynamic range in hardware.The dynamic range of the image can be changed by controlling the rotation of the polariser in front of the lens, and the well exposed pixel area of the image can be determined and selected autonomously.It can be seen from the fused images that the system is able to provide complete detail in both bright and dark areas without artefacts or glare, and that the light intensity range of the fused images is evenly distributed throughout the light intensity range.
However, the temporal response of the method in this paper still needs to be improved.As a next step, we are considering the use of a self-made polarization imaging chip to enable a more rational setting of the polarization direction and thus achieve better high dynamic imaging results.

Figure 1 .
Figure 1.Schematic diagram of the principle of an imaging system with adjustable dynamic range

Figure 2 .
Figure 2. The variation curve of the light intensity transmission received by the four types of pixels with the change of the polarizer rotation angle θ

Figure 3 .
Figure 3.The increased dynamic range of the imaging system varies with the angle of rotation of the polarizer

Figure 4 .
Figure 4. Variation of light intensity transmission with rotation angle of the polarizer due to different orientations of the polarization mask in Sony polarization chip

Figure 7 .
Figure 7. Results of the fused image of a static scene obtained with the polarizer in front of the lens at different angles.It can be seen that the amount of information in the fusion result varies depending on the angle of the polarizer.Note the difference in the detail of the clouds in the sky presented in the three images.

Figure 8 .
Figure 8. Photomatrix pro software was used to achieve HDR image fusion, with "Realistic" selected as the fusion style.The parameters are shown in the figure.