Noncontact imaging photoplethysmography to effectively access pulse rate variability

Abstract. Noncontact imaging photoplethysmography (PPG) can provide physiological assessment at various anatomical locations with no discomfort to the patient. However, most previous imaging PPG (iPPG) systems have been limited by a low sample frequency, which restricts their use clinically, for instance, in the assessment of pulse rate variability (PRV). In the present study, plethysmographic signals are remotely captured via an iPPG system at a rate of 200 fps. The physiological parameters (i.e., heart and respiration rate and PRV) derived from the iPPG datasets yield statistically comparable results to those acquired using a contact PPG sensor, the gold standard. More importantly, we present evidence that the negative influence of initial low sample frequency could be compensated via interpolation to improve the time domain resolution. We thereby provide further strong support for the low-cost webcam-based iPPG technique and, importantly, open up a new avenue for effective noncontact assessment of multiple physiological parameters, with potential applications in the evaluation of cardiac autonomic activity and remote sensing of vital physiological signs.


Introduction
Fluctuations in the interval between heartbeats, attributed to the continuous changes in the sympathetic-parasympathetic balance of the autonomic nervous system (ANS), are a sign of healthy cardiac function and a valuable tool to investigate the neural control of the heart. The quantification of these fluctuationsheart rate variability (HRV)-has proven its effectiveness in various research and clinical studies pertaining to cardiovascular disease, diabetic autonomic dysfunction, hypertension, and psychiatric and psychological disorders. 1,2 HRV has historically been obtained from the electrocardiograph (ECG). However, to obtain reliable recordings, multiple-lead ECG measurements are needed, requiring several electrodes to be carefully attached to the skin by a trained operator. Such a procedure could preclude frequent assessments of HRV for general use. Noncontact HRV measurements present an even greater challenge, and few attempts have been made. 3,4 Photoplethysmography (PPG) is an optical technique that noninvasively measures arterial pulsations in the microvascular tissue bed. 5 The pulsatile component of the PPG signal oscillates with the heart cycle period. The ease of use, low cost, and convenience make PPG an attractive area of study in the biomedical community and has stimulated research into alternative methods for accessing HRV with the aim of providing a simple and reliable evaluation of ANS activity. A number of recent studies have shown that the functional characteristics of pulse rate variability (PRV) are comparable to those of HRV, [6][7][8] where the former can be easily acquired from PPG.
The need to apply a PPG sensor directly to the skin constrains its practicability in situations such as skin damage (burn/ulcer/trauma) or when free mobility is required. Several studies have been performed using spring-loaded contact PPG probes to investigate the effect of contact force on the amplitude and timing of the PPG signal. 9,10 It was shown that this force must be carefully controlled to obtain high-quality, clinically useful data from the signals. One potential solution to this kind of problem is to use noncontact imaging PPG (iPPG) to assess physiological variables remotely, i.e., by means of peripheral blood perfusion measurements. The use of an imaging sensor could provide additional functionality with respect to a point sensor, particularly motion compensation and region of interest selection, both of which enable a robust and flexible noncontact PPG system. The past decades have witnessed rapid growth in the literature pertaining to iPPG techniques. [11][12][13][14][15][16][17][18][19][20] Among these applications, Jonathan and Leahy have demonstrated the feasibility of a cellular phone-based iPPG technique in heart rate assessment, stimulating its application for personal healthcare, which might influence a broad population of users. 19,20 Additionally, a growing list of personal physiological monitoring apps that utilize the camera modules in cellular phones have attracted significant public interest, even though they were not intended for clinical or diagnostic use. 21 However successful iPPG has been in acquiring physiological parameters, including heart rate and respiration rate, few attempts have been made to assess ANS using noncontact PPG. Recently, Poh and colleagues have introduced a remote PRV assessment technique using a low-cost webcam with ambient light as the illumination source, which shows significant potential for advancing personal care and telemedicine. 13 Nevertheless, a number of key questions remain concerning this pioneering feasibility study, the most important of which is the sample rate of the webcam (15 to 30 fps), which is much lower than the standard sample frequency in HRV analysis 1 (normally taken to be > 100 Hz). Although it is possible to increase the effective sample rate via interpolation in signal postprocessing, the influence of initial low sample frequency on the PRV analysis is still far from fully understood. A low sampling rate is likely to produce a jitter in the estimation of the pulse-pulse interval (PPI) sequences, which could alter their frequency spectrum considerably.
To further prove the practicability and feasibility of imaging PPG in remote PRV measurement, as well as to investigate the influence of sample rate on the PRV analysis, a high speed CMOS camera-based imaging system for capturing the iPPG signals at a sample frequency up to 200 fps is introduced and described here. The immediate objectives of this study were, first, to assess the practicability and the feasibility of the iPPG system in assessing PRV and, second, to investigate and quantify the influence of different sample rates on PRV analysis. The investigation conformed to the principles outlined in the Declaration of Helsinki (1989) of the World Medical Association, and it was approved by the local Research Ethics Committee. The nature of the research was explained to the subjects before the recordings, and their informed consent was obtained. All subjects were asked to refrain from consuming caffeine or alcohol, and they were asked not to smoke or undertake strenuous exercise for the 2 h preceding the study.

Instrument Setup
The iPPG system is schematically presented in Fig. 1(a). A monochrome CMOS camera (model: EoSens, MC 1363-63, Mikrotron GmbH, Unterschleissheim, Germany) with a spectral range of 500 to 1000 nm was focused on the palm of the participant's hand using a standard F-mount lens (model: Nikkor 20 mm, f∕2.8D, Nikon, Japan). Two commercial infrared (λ ¼ 880 nm, Δλ ¼ 20 nm, 10 W) light sources (model: ABUS TV 6818, ABUS Security-Centre GmbH, German) were mounted on either side of the camera to provide a uniform illumination of the target area: the palm of the left hand. Images were captured for 4 min at a rate of 200 fps and an exposure time of 4 ms, producing a raw image size of 256 × 384 pixels. The pixels were encoded in 10-bit gray scale, which allowed the camera to detect the weak pulsations of the microvascular tissue bed. During the image recording, a reference contact PPG signal was obtained simultaneously from a reference LED-PPG circuit, as shown in Fig. 1(b). Specifically, the reference PPG signal was initially obtained by means of a standard commercial contact PPG sensor (model: SA30014C, Shanghai Berry Electronic Tech. Co. Ltd., China) placed on the middle finger of the left hand and a DISCO4 data acquisition system (Dialog Devices, UK). This plethysmographic signal was then transferred to a custom-written LabVIEW (National Instruments Co., USA) software platform and served as the power intensity signal to trigger a red LED (λ ¼ 650 nm, Δλ ¼ 15 nm, RS Components, UK) through a DAQ (model: USB-6008, National Instruments Co., USA). The red LED probe was attached to the target area with double-sided adhesive tape during image acquisition, providing a clear and reliable reference signal for easy comparison with iPPG signals during postprocessing.

Experimental Protocol
As mentioned above, the purpose of this study was to assess the practicability and the variability of the iPPG in assessing PRV and to investigate and quantify the influence of different sample rates on PRV analysis. To accomplish these objectives, 10 sets of recordings were taken from 10 healthy subjects enrolled during four successive days. The image sequences were taken with the subject at rest to minimize physiological variations and motion. A soft cushion was placed under the target hand to further minimize the motion. The palm of each subject was exposed to the infrared illumination source, and the distance between the camera lens and the skin was about 400 mm. All measurements were taken in a temperature-controlled (20°C to 22°C) darkroom by a trained operator. The activity of the ANS governing the PRV clearly reveals circadian rhythms. 22 Hence, the recordings for this study were obtained within the period of 9:30 to 17:30 (median ¼ 12∶30) for consistency.

Image Processing
Motion artifacts were minimized by recording images under resting conditions. Hence, a simple yet efficient spatial averaging approach 15 was adopted and was found to be adequate to attenuate relatively small motion artifacts. 11,14 Once a set of recordings was acquired, the raw images were divided into discrete subwindows to produce a new set of reduced frames, as shown in Fig. 2(b). The value of each pixel in the reduced frame was set as the average of all the pixel values within each subwindow. Though it compromises the spatial resolution, such a procedure has been shown to significantly improve the signal-to-noise ratio. 15 Subwindow size was set at 8 × 8 pixels, resulting in a reduced frame size of 36 × 48 pixels. PPG signals were then derived from each pixel position across a sequence of frames. A fifth-order Butterworth filter with cutoff frequencies set at [0.05, 4] Hz was employed to further attenuate noise.

Heart/Respiration Rate Extraction from iPPG
In our previous studies, a time-frequency analysis [timefrequency representation (TFR)] was used to assess the timevarying heart rate (HR). 12 The HR was first obtained through averaging the fundamental HR frequency from the TFR trace calculated from the reference contact PPG signal, and this HR was then treated as a reference. Successive calculations were then performed on the reduced frames, where if the difference between the current HR estimation and the reference value exceeded the threshold of 9 bpm, the algorithm isolated these regions as corrupt and rejected the invalid HR. The HR iPPG could then be obtained by averaging HRs within all the valid subwindows. For each set of recordings, a 1-min moving window (12,000 frames) was employed for this HR calculation with 0.5-min overlap (6,000 frames). This yielded 7 HR readings for each subject. A similar approach was employed to obtain the respiration rate (RR iPPG ).

Trough Detection
To analyze the PRV, the first step is to detect the pulse-pulse intervals. A revised technique based on wavelet transforms was used for detecting trough positions, where the PPG signal was flat and contained little disturbance, thus facilitating PRV analysis. After bandpass filtering, the wavelet transform was performed on the obtained PPG signals, where each peak and trough pair in the PPG signals corresponds to a positive maximum and negative minimum pair with two zero-crossing points indicating the peak (maximum value) and trough (minimum value) positions. The locations of these zero-crossing points (n), where the minimum values were revealed in the original which the time scale has been expanded, show the plethysmographic waveform in which RR and HR can be clearly seen. The upper TFR trace is from the cPPG, and the lower one is from iPPG, with a color bar indicating the power intensity. The position at which the iPPG signal was obtained is shown by the black box (1 × 1 pixel) and an arrow in (b), while the region where the cPPG reference signal was obtained is indicated with a larger black box (3 × 3 pixel). The signal is from Subject #1 (male, age ¼ 28 years).
PPG signals, served as the raw trough positions. A semiautomated procedure was then employed to further eliminate the false positive and negative trough detections and to improve the accuracy. A manual validation was initially conducted to verify n 1 as the first trough position. HR was then calculated within a moving window W t1;t2 (t2 − t1 ¼ 10 s), and then 0.9 to 1.1 of HR −1 was adopted as the PPI threshold. If 0.9 HR −1 < jn i1 − n i2 j < 1.1 HR −1 , where t1 ≤ n i1 , n i2 ≤ t2, the algorithm considers n i2 a valid trough position. If no valid trough position was found, a search back approach was performed to find the minimum value within the threshold range, and a warning index was saved for postvalidation.

Pulse Rate Variability Analysis
A series of straightforward and efficient time domain and frequency domain measurements was employed to extract the PRV information. The mean values of pulse-pulse intervals (MPP) and standard deviation of the pulse-pulse intervals (SDPP) were employed in the time domain analysis. As a discrete event series, PPI is an unevenly sampled time series, hence an interpolation and resample approach was conducted before the frequency domain analysis. In practice, it is recommended that the duration of the recording be about 10 times the wavelength of the lowest-frequency band of the spectral component being investigated. 1 Therefore, only the low frequency (LF: 0.04 to 0.15 Hz), the high frequency (HF: 0.15 to 0.40 Hz), and the LF/HF ratios were calculated in this work. The HF component, which has a peak at respiratory frequency, corresponds to respiratory sinus arrhythmia (RSA) and reflects parasympathetic influence on the heart through efferent vagal activity. The LF component, consisting of fluctuations below 0.15 Hz and usually centered at about 0.1 Hz, is mediated by both cardiac vagal and sympathetic nerves. 23,24 Hence, the ratio of the LF and HF represents the sympatho-vagal interaction. LF and HF were also expressed in normalized units to account for interindividual differences, denoted as LF n ¼ LF∕½Total power − VLFðvery low frequency; 0.003 − 0.04 HzÞ and HF n ¼ HF∕ ðTotal power − VLFÞ, respectively. 1 Postprocessing and analysis were performed with custom software in Matlab 2008a (MathWorks, Natick, Massachusetts, USA).

Influence of Frame Rate on PRV Analysis
A down-sampling approach has been adopted (Fig. 3) to investigate the influence of sample rates on the PRV analysis. Once a set of reduced frames was acquired, three new frame sequences with sample frequencies of 100, 50, and 20 fps were obtained by down-sampling the original reduced frames. These new frame sequences used the iPPG recordings with the same image configuration yet different sample frequencies, which could therefore be treated independently. The PPG signal extraction and preprocessing procedures were repeated. To refine the trough fiducial point, especially for the low-sample-rate iPPG recordings, the PPG signals were interpolated with a spline function 13 to reach an effective frequency of 200 Hz. These interpolated PPG signals were then ready for the trough detection and PRV analysis as described above.

Statistical Analysis
To evaluate the reliability of the iPPG system for remote physiological assessments, Bland-Altman analysis 25 was performed to compare iPPG and contact PPG (cPPG). The difference between iPPG and cPPG was plotted against their average, as were the mean and the standard deviation (S.D.) of the differences and 95% limits of agreement (AE1.96 S:D:). The Pearson's correlation coefficients and the corresponding p-value were also calculated to assess the physiological variables (HR, RR, and PRV) from both systems. Differences between the PRV measurements at different sample rates were assessed with one-way analysis of variance (ANOVA, factor ¼ sample rate) to demonstrate the influence of sample frequency on the PRV analysis. The group differences were assessed by post hoc analysis using Fisher's least significant difference (LSD), where groups represent the different sample frequency. All analyses were performed with α (Type I error) set at 0.05 using the statistical software program SPSS for Windows, version 17.0 (IBM, Armonk, New York). Figure 2 shows an example of the PPG signals and their corresponding TFR traces. The reference cPPG signal was obtained by calculating the mean intensity value within the region of the reference LED. The variations in the signal due to heartbeat and respiration are clearly visible for both PPG signals in the inserts of Fig. 2(c) and 2(d). It can be seen in the TFR traces that the oscillations of RR, HR, and the first harmonic component derived from the iPPG signals were in agreement with those obtained from the commercial pulse oximeter sensor readings. Bland-Altman analysis was used to evaluate the agreement of HR and RR between the imaging method and the contact readings. The results of the Bland Altman analysis are shown in Fig. 4. A close agreement between these two techniques was revealed. Specifically, in no case did the mean difference of the physiological measurements (HR/RR) between the two methods differ significantly from zero. The mean bias of RR is 0.03 breaths∕ min with 95% limits of agreement −1.29 to þ1.35 breaths∕ min; while the mean bias for the heart rate is −0.04 bpm with 95% confidence interval −3.00 to þ2.93 bpm. Significant correlations between both techniques were also revealed (Pearson's correlation for RR and HR, r 2 > 0.95, p < 0.001).

Pulse Rate Variability Measurement
A typical example of the imaging PPG signals is presented in Fig. 5(a) and 5(b) along with the contact PPG signals recorded with the reference LED-PPG system. The boxes in Fig. 5(a) and 5(b) show data at a magnified time scale (60 to 80 s), where clear plethysmographic waveforms with detected trough positions are shown in both PPG signals [cPPG (black), iPPG (green)]. The obtained pulse-pulse intervals are shown in the right panel of Fig. 5(a) and 5(b). It is evident that the two PPI sequences are in close agreement. The resulting power spectral density maps are shown in Fig. 5(c), where both spectra exhibit comparable functional characteristics and a dominant LF component. Table 1 summarizes the time and frequency domain results of the PRV analysis [values are mean AE standard error of the mean (SEM)]. Significant correlations between cPPG and iPPG were found in all estimated results (Pearson's correlation, r > 0.85, p < 0.001).
To further evaluate the performance of the imaging PPG in assessing PRV information, Bland-Altman analysis was employed to assess the agreement between contact and imaging readings of the estimated PRV results. Figure 6 shows that the mean biases for MPP and SDPP are −1.00 and −0.49 ms, respectively. The corresponding 95% limits of agreement are from −9.94 to 7.94 ms and −11.54 to 10.56 ms, which shows that they do not differ significantly from zero. Similar statistically comparable results are revealed in the frequency analysis. The mean biases between cPPG and iPPG for the measurement of LF n , HF n , and LF/HF are −0.008, 0.005, and 0.046. The corresponding 95% confidence intervals are −0.093 to 0.078, −0.076 to 0.086, and −0.381 to 0.473. Again, this shows that none of these biases differs significantly from zero.

Effects of Frame Rate on PRV Analysis
Three additional down-sampled iPPG sequences were obtained as described previously (i.e., 100, 50, and 20 fps). Table 2 summarizes the PRV results for different sample frequencies.
Further statistical analysis showed no significant influence of the sample rate on any of the PRV measurements: MPP Treating the PRV measurements calculated from cPPG readings as the reference, significant correlations between cPPG and the three down-sampled iPPG sequences were also uncovered (Pearson's correlation, r > 0.81, p < 0.001). We note that MPP increases monotonically with increasing sample frequency (MPP 20 fps < MPP 50 fps < MPP 100 fps < MPP 200 fps ), whereas no such consistent trend is seen with the other related variables. The variations of PRV measurements in the down-sampled sequences indicate a negligible loss of data for reductions in sample frequency down to 20 fps.

Discussion
This study considers the remote accessing of physiological information (i.e., RR, HR, and PRV) at a fast sample rate (200 fps) by means of iPPG and compares the results to those obtained from a commercial pulse oximeter sensor. The strong correlation and good agreement between these two methods for capturing PPG signals proves the value of an iPPG system in remotely measuring PRV. On the basis of these results, we have demonstrated the feasibility of using an iPPG system to appraise cardiac autonomic activity remotely. The influence of initial sample frequency on the PRV analysis has also been revealed. Statistical results showed nonsignificant difference among various sample frequencies. This finding thereby provides further support for a previous study that reported on a webcam-based system for measuring multiple physiological parameters. 13 Since Huelsbusch and Blazek 18 introduced a novel experimental charge coupled device (CCD) camera-based iPPG system in 2002, a steadily increasing number of iPPG systems have been described for a variety of noncontact physiological assessments. [11][12][13][14][15][16][17][18] Although they have been successful in remote physiological measurements of heart/respiration rate, 11,12,14,15 perfusion mapping, and blood oxygen saturation, 15,18 the maximum sample rate of these pioneering iPPG studies is <50 Hz, which is much less than the standard minimum sample rate 1 for reliable heart rate variability measurements (100 Hz). In the iPPG setup described here, the camera was configured to capture the images at a sample frequency of 200 fps, offering relatively accurate time domain resolution and ensuring reliable PRV analysis. As can be seen in Fig. 2, a clear imaging PPG signal has been obtained even from a small region (1 × 1 pixel in the reduced frame, corresponding to 8 × 8 in the raw image). Furthermore, time-frequency analysis shows close agreement in respiration rate, heart rate, and the first harmonic components of the signals. Further statistical analysis of the measured  physiological variables (RR and HR) shows that the iPPG system exhibits functional characteristics comparable to conventional contact PPG sensors, i.e., the maximum difference is about 3 bpm for the HR and 1.3 breaths∕ min for RR, providing support for the derived PRV assessments. From the clinical point of view, the word "comparable" means that measurements by different instruments/techniques should be sufficiently close so as not to change the actions and decisions regarding the patient's treatment. 16 For instance, in the Emergency Severity Index triage, an adult patient with a HR > 100 bpm is considered tachycardic. 26 In this scenario, a difference of ∼3 bpm (3% error) obtained by our remote iPPG system would most likely be acceptable, given the variability in this quantity however it is measured.
In contrast to various well-established QRS detection techniques in HRVanalysis, 27 there are fewer publications pertaining to the characterization of PRV signals. In order to obtain the PPI sequences, a wavelet transform-based semi-automated trough detection algorithm has been introduced in this study. By adopting the short-window moving HR as the threshold, the present method achieved an accuracy of 95% in all subjects before the manual verification approach, which resulted in 100% accuracy. Statistical analysis of the time/frequency domain results calculated from the obtained PPI sequence showed a strong correlation (r > 0.85, p < 0.001) and close agreement between the imaging PPG and the standard contact PPG sensor. The results of this study thereby provide further and reliable support for the idea that an iPPG system can successfully obtain clinically useful information about the ANS. The last two decades have witnessed the recognition of a significant relationship between the ANS and cardiovascular mortality, including sudden cardiac death. One recent study of 763 elderly subjects has reported the time-and frequency-domain measures of HRV and the relationship of these measures to all-cause mortality during a four-year follow-up period. 28 However, the normal HRV standards for various age and gender subsets are  still missing, and larger prospective population studies with longitudinal follow-up are needed. 29 Such studies might be significantly facilitated by employing imaging PPG technology. One recent iPPG study has described a technique for the assessment of multiple physiological parameters using a basic webcam and provided evidence of its feasibility in appraising the autonomic nervous system. 13 However, the low sample frequency (i.e., 15 fps) of the webcam used might influence the PRV analysis. Through down-sampling the iPPG recordings, we simulated three additional iPPG sequences with lower sample rates and calculated the PRV. An interpolation approach was then adopted before the PRV analysis to refine the troughs in the down-sampled iPPG signals and improve the timing estimation of pulse-to-pulse intervals. Statistical results presented no significant difference among the various sample rates, which was in keeping with the independent relationship between the variations of PRV measurements and sample frequency. These findings agree well with recent work by Poh, McDuff, and Picard 13 and emphasize the necessity of interpolation in PRV assessments, especially when a low sample frequency is used, as in webcam-based iPPG.
One main source of error arises from the motion artifact that could occur because the sensor has no contact with the skin, consequently leading to corrupted signals and presenting problems for the subsequent signal processing. The primary aim of this study was to test the feasibility of the iPPG technique in assessing the ANS. The images were recorded from a palm that was supported with a soft cushion. We found that a simple spatial averaging approach was sufficient to minimize the minor motion artifacts observed under these experimental conditions. Nevertheless, the use of sophisticated and more effective motion artifact compensation methodologies is encouraged, and more generalized imaging registration 12,30,31 and blind source separation 12-14 methods should be used in the future. Originating from the pulse wave and governed by the cardiac cycle, PRV has been shown to be a good surrogate for HRV. A further study is under consideration to assess the performance and to appraise the accuracy of the iPPG-based PRV using the gold standard (ECG-based HRV) with more subjects. In addition, the recording time for this study was relatively short, and future work needs to extend the monitoring time to enable long-term, continuous, and comprehensive [e.g., VLF component (0.003 to 0.04 Hz)] measurements.

Conclusion
This study provides an important insight into the effective and comprehensive interpretation of remote physiological assessment. Imaging PPG technology was employed to investigate the feasibility of noncontact assessment of PRV, an index of ANS activity. By comparing the iPPG results with those from a standard contact sensor, we have demonstrated that the 200-fps iPPG system presented here provided measurements of HR, RR, and PRV that were closely comparable to those acquired from contact PPG. In addition, we found that the negative influence of a low initial sample rate could be compensated by interpolating the signal to improve the time domain resolution and PRV measurements. The results thereby offer further support for the applicability of the iPPG technique and constitute a further step toward effective, remote, and multiple physiological assessments, providing a promising alternative to conventional contact PPG techniques, with clear applications in triage, low cost home-based monitoring systems, and telemedicine.