Statistical Analysis of Non Linear Least Squares Estimation for Harmonic Signals in Multiplicative and Additive Noise

In this paper we consider the problem of parameter estimation for the multicomponent harmonic signals in multiplicative and additive noise. The nonlinear least squares (NLLS) estimators, NLLS1 and NLLS2 proposed by Ghogho et al. (1999b) to estimate the coherent model parameters for single-component harmonic signal, are generalized to the multicomponent harmonic signals for the cases of nonzero- and zero-mean multiplicative noise, respectively. By statistical analysis, some asymptotic results of the NLLS estimators are derived, including the strong consistency, the strong convergence rate and the asymptotic normality. Furthermore, the NLLS1- and NLLS2- based estimators are proposed to estimate the noncoherent model parameters for the cases of nonzero- and zero-mean multiplicative noise, respectively, meanwhile the strong consistency and the asymptotic normality of the NLLS-based estimators are also derived. Finally some numerical experiments are performed to see how the asymptotic results work for finite sample sizes.


Introduction
In this paper, we consider the following model of the multicomponent harmonic signals in multiplicative and additive noise: where j = √ −1, M, ω 0 k 's, φ 0 k 's, s k (t)'s, and n(t) are the number of signals, the frequencies, the phases, the multiplicative noise and the additive noise, respectively. In addition, we make the following assumptions: Assumption 1. (i) The frequencies ω 0 k 's are distinct in (0, π) and the phases φ 0 k 's are deterministic constants in [0, π); (ii) The multiplicative noise s k (t) is a sequence of i.i.d. real-valued random variables with mean μ 0 sk ≥ 0 and variance σ 2 sk ; (iii) The additive noise n(t) is a sequence of i.i.d. complex-valued random variables with mean zero, and both the real and imaginary parts have variance σ 2 0 /2 and they are independent; (iv) The noise s k (t)'s and n(t) are mutually independent.
However, multiplicative noise may also occur in a variety of applications. For example, in Doppler-radar processing , knowledge of the frequency from a pulse train reflected from a moving object yields the target's velocity, and it is more appropriate to model the harmonic signals as having random rather than constant amplitude if the point-target assumption is no longer valid, or, when the target scintillates; in underwater acoustic applications , the multiplicative noise describes the effects on acoustic waves due to the fluctuations caused by the medium (diffraction, internal waves, or microstructures) changing orientation and interference from scatterers of the target. Several methods have been suggested to estimate the frequencies of harmonic signals in multiplication and additive noise under the assumption that the number of signal is known, such as second-order statistics (Besson and Castanie, 1993), higher-order statistics (Swami, 1994), cyclic statistics Zhou and Giannakis, 1995;Li and Cheng, 1998;Ghogho et al., 1999a), and three-step iterative approach (Bian et al., 2009 and2011). However, in practice, the number of signals can be unknown and estimating the number of signals will be the first step. Without multiplicative noise, there are some methods to determine the number of signals, e.g., Akaike information criterion (AIC) (Akaike, 1974) and minimum description length (MDL) (Schwarz, 1978). But the performance of AIC and MDL will degrade in the presence of unknown multiplicative noise. For this purpose, a few methods have been proposed to estimate the number of signals of harmonics in multiplicative and additive noise. For example, Li and Cheng (1998) proposed cyclic statistics based on the statistical properties of sample cyclic-moments, and obtained the strong consistency and strong convergence rate of the proposed estimator; Yang and Li (2007b) proposed enhanced matrix based on the eigenvalue properties of covariance matrix of the constructed enhanced matrix.
It is well known that the NLLS estimation has been seen as the most intuitive method and it has played a very important role in harmonic parameter estimation. As early as 1970s, the statisticians Walker (1971Walker ( , 1973 and Hannan (1973) preliminarily studied the NLLS method in harmonic parameter estimation. Later, much more statisticians like Rao and Zhao (1993), Giannakis and Zhou (1995), Stoica et al. (1997), Kundu and Mitra (1999), Besson and Stoica (1999), Ghogho et al. (1999b), Cohen and Francos (2002) did a lot of works and moved forward this field further. For example, Kundu and Mitra (1999) considered the multicomponent harmonics model in additive noise and derived some fantastic statistical results for the NLLS estimators including the strong consistency, the strong convergence rate and the asymptotic normality and so on; Ghogho et al. (1999b) considered the singlecomponent harmonics model in multiplicative and additive noise and proposed two specific NLLS estimators, i.e. NLLS 1 and NLLS 2 that consist of matching the data and the squared data respectively, moreover the expressions for the asymptotic covariances of the two NLLS estimators were derived. In this paper, we generalize the NLLS estimators proposed by Ghogho et al. (1999b) to the case of multi-component harmonics in multiplicative and additive noise. We separately address the cases of nonzero-and zero-mean multiplicative noise by using NLLS 1 and NLLS 2 , respectively, to estimate the coherent model parameters. By following the proof techniques proposed by Kundu and Mitra (1999), the asymptotic covariances of the two NLLS estimators in the case of multicomponent multiplicative noise are derived, which are consistent with the corresponding results (see Ghogho et al., 1999b) if it is reduced to the single-component model, moreover the strong consistency and the strong convergence rate are also derived. Furthermore, the NLLS-based estimators are proposed to estimate the noncoherent model parameters, and the strong consistency, the strong convergence rate and the asymptotic normality for the proposed estimators are also derived. Finally, the simulation results verify the efficiency of all the estimators.
The rest of the paper is organized as follows. In Sec. 2, we describe the parameter estimation based on NLLS 1 and NLLS 2 for multicomponent model (1) in the cases of nonzeroand zero-mean multiplicative noise, respectively. The statistical analysis and the theoretical results for the estimators are provided in Sec. 3. In Sec. 4, we present some numerical experiments, and finally we conclude the paper in Sec. 5. All proofs are provided in the Appendices.

Nonzero-mean Multiplicative Noise
In this subsection, we consider μ 0 sk = 0, for k = 1, 2, . . . , M. First, the data-based NLLS 1 proposed by Ghogho et al. (1999b) is employed to estimate the coherent model parameters including the frequencies, the phases, and the multiplicative noise means in this case.
Then, the NLLS 1 -based estimators are proposed to estimate the noncoherent model parameters, i.e., the noise parameters including the multiplicative noise variances, the total noise variance, and the additive noise variance.
In this case, the coherent and noncoherent components in data x(t) = y(t) + ε(t) are given, respectively, by Notice that the noncoherent component ε(t) is zero-mean and can be regarded as additive noise. Hence, the coherent model parameters including the frequencies, the phases, and the multiplicative noise means in this case can be estimated by least squares estimation directly based on data x(t) referred as NLLS 1 (see Ghogho et al., 1999b). We denote the parameter vector θ = (θ 1 , θ 2 , . . . , θ M ), where θ k = (ω k , φ k , μ sk ) for k = 1, 2, . . . , M. Similarly, θ 0 and θ 0 k 's are also defined. Now the NLLS 1 of true parameter vector θ 0 , i.e.,θ , can be obtained byθ where Based on the NLLS 1 defined above, the following estimators are proposed to estimate the multiplicative noise variance σ 2 sk , for k = 1, 2, . . . , M, and the total noise variance σ 2 = σ 2 0 + M k=1 σ 2 sk , respectively, where the symbol "Re" denotes the real part of a complex value. Finally, combining the estimations of the multiplicative noise variances and the total noise variance, the additive noise variance σ 2 0 can be estimated intuitively bŷ

Zero-mean Multiplicative Noise
In this subsection, we consider μ 0 sk = 0, for k = 1, 2, . . . , M. First, the squared-data-based NLLS 2 proposed by Ghogho et al. (1999b) is employed to estimate the coherent model parameters, including the frequencies, the phases, and the multiplicative noise variances in this case. Then, the NLLS 2 -based estimators are proposed to estimate the noncoherent model parameters, i.e., the remaining noise parameters including the total noise variance and the additive noise variance.
In this case, the coherent harmonic signals in x(t) vanish. Fortunately, spectral analysis of the squared data allows one to recover the harmonic signals (see Ghogho et al., 1999b;Giannakis and Zhou, 1995). The coherent and noncoherent components in squared data x 2 (t) = y(t) + ε(t) are given, respectively, by where p 0 k = (σ 0 sk ) 2 , for k = 1, 2, . . . , M. Notice that the noncoherent component ε(t) is zero-mean and can be regarded as additive noise. Hence, the coherent model parameters including the frequencies, the phases, and the multiplicative noise variances in this case can be estimated by least squares estimation based on the squared data x 2 (t) referred as NLLS 2 (see Ghogho et al., 1999b). We denote the parameter vector Similarly, θ 0 and θ 0 k 's are also defined. Now the NLLS 2 of true parameter vector θ 0 , i.e.,θ , can be obtained bŷ where Notice that x(t) is a sequence of independent complex-valued random variables with mean zero under Assumption 1. Hence, the total noise variance σ 2 = σ 2 0 + M k=1 σ 2 sk , i.e., the variance of x(t), can be estimated directly bŷ Finally, combining the estimations of the multiplicative noise variances and the total noise variance, the additive noise variance σ 2 0 can be estimated intuitively bŷ

Statistical Analysis
In the case of multi component harmonics in additive noise, the asymptotic performance of the NLLS estimators was studied by Kundu and Mitra (1999). In this section, by following the proof techniques proposed by Kundu and Mitra (1999), we address the case of multicomponent harmonics in multiplicative and additive noise and derive some statistical results including the strong consistency, the strong convergence rate, and the asymptotic normality for the estimators described in Sec. 2 corresponding to the cases of nonzero-and zero-mean multiplicative noise.

Nonzero-mean Multiplicative Noise
We first state the statistical results of the NLLS 1 for the coherent model parameters including the frequencies, the phases, and the multiplicative noise means as follows.
Theorem 3. Under Assumption 1, (θ − θ 0 )D converges in distribution to a 3Mvariate normal distribution with mean vector zero and the covariance matrix = diag{ 1 , 2 , . . . , M }, which is a M-dimensional block diagonal matrix whose block element is given by, for k = 1, 2, . . . , p, Corollary 1. Under the same assumption of Theorem 3, the NLLS 1 's, i.e.,ω k ,φ k andμ sk , for k = 1, 2, . . . , M, are asymptotically consistent and normal, with asymptotic variances given by The expressions of the variances in Corollary 1 follow from the asymptotic covariance matrix (ACM) in Theorem 3. By comparing the ACM in Theorem 3 with the corresponding result of NLLS 1 obtained by Ghogho et al. (1999b) corresponding to the case of singlecomponent harmonics in multiplicative and additive noise, it shows the consistency if the multicomponent model in (1) is reduced to the single-component case. Note that Theorem 1 is a very strong result, it means that the NLLS 1 's will be closer and closer to the true parameter values as the sample size increases. Therefore the biases and the mean squared errors (MSEs) tend to zero as sample size increases. Meanwhile, Theorem 2 gives the strong convergence rate of the frequency estimator that is also an important issue in statistics. Furthermore, the obtained asymptotic distributions in Theorem 3 will help us to obtain the confidence bounds for these coherent parameters. From the expression of ACM in Theorem 3, it is immediate that the corresponding NLLS 1 's are asymptotically uncorrelated for distinct harmonic components, the NLLS 1 's of the multiplicative noise mean are asymptotically uncorrelated with both of the NLLS 1 's of the frequency and the phase for the same harmonic component, and the ACM does not depend on the frequencies and the phases but only depends on the noise. Note that Theorem 3 also gives us the idea of the weak convergence rate of the NLLS 1 's, such as it is O P (T −3/2 ) for the frequencies but O P (T −1/2 ) for all of the phases and the multiplicative noise means, which indicates that the estimation accuracy for the nonlinear parameters is better than the one for the linear parameters.
Now we state the statistical results of the NLLS 1 -based estimators for the noncoherent model parameters, i.e., the noise variances including the multiplicative noise variances, the total noise variance, and the additive noise variance.
Theorem 4. Under Assumptions 1 and 2,σ 2 sk 's are strongly consistent estimators of σ 2 sk 's, and it also can be obtained that √ T (σ 2 sk − σ 2 sk ) converges in distribution to a normal distribution with mean zero and variance σ * sk , which is given by Theorem 5. Under Assumptions 1 and 2,σ 2 is a strongly consistent estimator of σ 2 , and it also can be obtained that √ T (σ 2 − σ 2 ) converges in distribution to a normal distribution with mean zero and variance σ * , which is given by Theorem 6. Under Assumptions 1 and 2,σ 2 0 is a strongly consistent estimator of σ 2 0 , and it also can be obtained that √ T (σ 2 0 − σ 2 0 ) converges in distribution to a normal distribution with mean zero and variance σ * 0 , which is given by Corollary 2. If we further assume that all the noise distributions are Gaussian, then the expressions of the asymptotic variances in Theorems 4-6 can be reduced to Corollary 3. Under Assumptions 1 and 2, the estimators of all the noise variances, i.e., σ 2 sk 's,σ 2 , andσ 2 0 , are asymptotically consistent and normal, with asymptotic variances given by where σ * sk 's, σ * , and σ * 0 are defined in Theorems 4-6, respectively.
The simple formulas in Corollary 2 follow from the complicated ones in Theorems 4-6 under the additional Gaussian assumption. Theorems 4-6 imply that, under Assumptions 1 and 2, it is possible to estimate the parameters quite accurately not only for the frequencies, the phases and the multiplicative noise means but also for the multiplicative noise variances, the total noise variance and the additive noise variance when the sample size is large enough. Similarly as Theorem 3, Theorems 4-6 also tell us that the weak convergence rate of all the noise variances is O P (T −1/2 ). Moreover the obtained asymptotic distributions will help us to obtain the confidence bounds for the unknown noise variances.

Zero-mean Multiplicative Noise
In this subsection, we state the results for the case of zero-mean multiplicative noise along the same line as the statements for the case of nonzero-mean multiplicative noise. For the purpose of simplification or compactness of the paper, we simplify some statements by comparison with the case of nonzero-mean multiplicative noise, and we also omit some notations on the results, the readers can easily get the similarities and the differences between the cases of nonzero-and zero-mean multiplicative noise by simple comparison.
We first state the statistical results of the NLLS 2 for the coherent model parameters including the frequencies, the phases, and the multiplicative noise variances as follows.

Theorem 8. The same result (Theorem 2) holds.
Theorem 9. The similar result (Theorem 3) holds with the only difference lied in the covariance matrix whose block element is given by, for k = 1, 2, . . . , M, Corollary 4. If we further assume that all the noise distributions are Gaussian, then the expressions of A k and B k in Theorem 9 can be reduced to Corollary 5. Under the same assumption of Theorem 9, the LSEs, i.e.,ω k ,φ k , andσ 2 sk , for k = 1, 2, . . . , M, are asymptotically consistent and normal, with asymptotic variances given by Note that, by comparing the ACM in Theorem 9 with the corresponding result of NLLS 2 obtained by Ghogho et al. (1999b) corresponding to the case of single-component harmonics in multiplicative and additive noise, it shows the consistency if the multicomponent model in (1) is reduced to the single-component case. Now we state the statistical results of the NLLS 2 -based estimators for the noncoherent model parameters, i.e., the remaining noise variances including the total noise variance and the additive noise variance.

Numerical Results
In this section, we present some numerical experiment results to see how the NLLS's or the NLLS's-based estimators work for finite sample sizes, and whether the asymptotic results can be used for small sample sizes. We consider the following common model: In this case, both the multiplicative noise s 1 (t) and s 2 (t) are i.i.d. real Gaussian random variables whose variances are 1.0 and 2.0, respectively. The additive noise ε(t) is i.i.d. complex Gaussian random variables with mean zero, and both of the real and imaginary parts have variance 0.5. All the noises are mutually independent.
In the case of nonzero-mean multiplicative noise, the multiplicative noise means in model (13) are taken as 1.0 and 2.0, respectively. We consider six different sample sizes, namely T = 50, 100, 150, 200, 250, and 300. For each data set generated from the above model, the coherent model parameters including the frequencies (ω 0 1 , ω 0 2 ) = (1.0, 2.0), the phases (φ 0 1 , φ 0 2 ) = (2.0, 3.0) and the multiplicative noise means (μ 0 s1 , μ 0 s2 ) = (1.0, 2.0), can be estimated by NLLS 1 proposed in Eq. (4), after that the noncoherent model parameters including the multiplicative noise variances (σ 2 s 1 , σ 2 s 2 ) = (1.0, 2.0), the total noise variance σ 2 = 4.0 and the additive noise variance σ 2 0 = 1.0 can be estimated by the NLLS 1 -based estimators proposed in Eqs (5-7) respectively. We replicate the process 500 times and calculate the average estimates (AE) and the MSEs for all the parameters. The corresponding asymptotic variances (AVAR) are also reported for the purpose of comparison. We also calculate the approximate 90% confidence limits for all the parameters and obtain the expected confidence interval length (length) using the true parameter values (see Kundu and Mitra, 1999). The coverage percentages (coverage) are also obtained over 500 replications. It worth noting that, due to the highly nonlinearity of the objective function of the NLLS 1 , we employed a hybrid stochastic searching algorithm to solve this nonlinear programming problem. The genetic algorithm was first used as a global method to search the entire parameter space, then followed by the Nelder-Mead Simplex algorithm to search for the local minimum in the vicinity of output from the genetic algorithm. The corresponding optimization procedures were implemented using the MATLAB gatool and fminsearch functions, respectively. Multiple settings for the genetic algorithm were tested, and the following were selected, Generations = 500; PopulationSize = 120 (20 × number of parameters); EliteCount = 8; CrossoverFraction = 0.5; MutationFcn = {@mutationuniform, 0.5}. All the computations are performed in MATLAB 7.5 (R2007b). All the numerical results are presented in Tables 1-4 corresponding to the frequencies, the phases, the multiplicative noise means, and the noise variances, respectively.
The following observations are very clear from the numerical experiments. First, it can be observed that the AEs of all the parameters are very close to the true parameter values in all the considered cases, meanwhile the MSEs of all the parameters gradually decrease and approach the AVARs as the sample size increases, which verifies the consistency of the proposed estimators and also shows the validity of the asymptotic results even for moderate sample sizes. Next, it is also clear that in all cases the Lengths decrease as sample size increases, and the coverage percentages are nearly 90%, which means that the asymptotic results can be used to obtain the confidence bounds of the unknown parameters even for moderate sample sizes. Then, it can be observed as expected that the estimations of the frequencies are more accurate than other parameters for almost all of the considered sample sizes. Finally, it can be seen from Table 4 that the NLLS 1 -based estimators proposed for the estimations of noise variances also work well.
In the case of zero-mean multiplicative noise in model (13), after squaring the data, the coherent model parameters including the frequencies, the phases, and the multiplicative noise variances, can be estimated by NLLS 2 proposed in Eq. (10), after that the noncoherent model parameters including the total noise variance and the additive noise variance can be estimated by the NLLS 2 -based estimators proposed in Eqs (11) and (12), respectively. The procedures of implementation are the same as the case of non-zero mean multiplicative   noise mentioned above. Due to the space limitations, the simulation results are just provided in Tables S1-S3 in the supplementary corresponding to the frequencies, the phases, and the noise variances, respectively. It can be observed from Tables S1-S3 that the similar results as the case of nonzero mean multiplicative noise mentioned above hold, which verify the efficiency of the proposed estimators in the case of zero mean multiplicative noise.
In addition, we carry out the following simulations to evaluate the performance of the proposed hybrid stochastic searching algorithm. The MSE is employed for the performance measure. For comparison, the MSE, the AVAR, and the asymptotic Cramer-Rao bound (ACRB) are included. Notice that the Fisher information matrix is reduced to zero in the case of zero-mean multiplicative noise, hence the ACRB cannot be obtained in this case (see Ghogho et al., 1999b). Thus, we restrict our attention to the case of nonzero-mean multiplicative noise, in which the ACRB has been provided by Francos and Friedlander (1995) and Mao and Bao (1996). It is well known that the signal-to-noise ratio (SNR) is usually valuable for the performance evaluation, and it has been reported from Ghogho et al. (1999b) that the intrinsic SNR referred as ISNR is meaningful for the harmonics in multiplicative and additive noise, hence the ISNR is included in these simulations. In this case, the ISNR of the first harmonic component referred as ISNR1 is defined as I SNR1 = 10log10((μ 0 s 1 ) 2 /σ 2 s 1 ) (dB), and the ISNR2 is similarly defined. We carry out the simulation by the same procedures from the model (13) with the same parameters except varying σ 2 s 1 , and then the MSE, AVAR, and ACRB versus ISNR1 for different coherent parameters can be evaluated. All the results in terms of ISNR1 are shown in Figs. 1-3 corresponding to frequency, phase, and multiplicative noise mean, respectively. The following observations are very clear from the numerical experiments. First, it can be observed that the MSEs are close to the corresponding AVARs as the ISNR1 increases for all the cases. Next, it is interesting to observe that for the cases of both frequency and phase, i.e., nonlinear parameters, the AVARs are almost always less than the corresponding ACRBs, and only one of two frequencies or phases are close to the corresponding ACRB, which also hold for MSEs and ACRBs. Finally, for the case of multiplicative noise mean, i.e., linear parameter, one of two can close to the corresponding ACRB as the ISNR1 increases. On the other hand, we have also carried out the similar simulation in terms of ISNR2, and we do not show the data due to the space limitations, from which the similar results hold.

Conclusions
This study focused on the parameter estimation of multicomponent harmonic signals in multiplicative and additive noise, in which the nonzero-and zero-mean multiplicative noise cases were addressed separately based on NLLS. First, the NLLS 1 and NLLS 2 proposed by Ghogho et al. (1999b) were generalized to estimate the coherent model parameters for the multiple harmonics corresponding to the cases of nonzero-and zero-mean multiplicative noise, respectively. Then, the NLLS 1 -and NLLS 2 -based estimators were proposed to estimate the noncoherent model parameters. Next, by following the techniques of performance analysis proposed by Kundu and Mitra (1999) corresponding to the harmonics only in additive noise, some statistical results for the proposed estimators were theoretically proved, including the strong consistency, the strong convergence rate, and the asymptotic normality. Finally, the numerical results suggested that the asymptotic results can be used even for moderate sample sizes, which practically shows the efficiency of the proposed estimators.
If for any δ > 0 and for some 0 < H < ∞, then the LSEs,θ, is a strongly consistent estimator of θ 0 .
Proof. The proof can be obtained by contradiction along the same line as Lemma 1 of Wu (1981).
Proof of Theorem 4. To obtain the strong consistency ofσ 2 sk defined in (5), we firstly transform it into the following form, where the expression of R T (θ k , θ 0 ) is provided in the supplementary information due to space limitations. We observe that s 2 k (t) − (μ 0 sk ) 2 is a sequence of i.i.d. real random variables with mean σ 2 sk , which implies that 1 T T t=1 s 2 k (t) − (μ 0 sk ) 2 → σ 2 sk , a.s. because of Kolmogorov strong law of large numbers (see Rao, 1973, p. 115). It can be shown that (μ 0 sk ) 2 −μ 2 sk → 0, a.s. because of the strong convergence ofμ sk . Expanding R MN (θ k , θ 0 ) around the true value θ 0 by Taylor series expansion and combining Theorem 1, Theorem 2, and Lemma 2, it also can be shown that R T (θ k , θ 0 ) → 0, a.s. Therefore, combining (A.13) and the results mentioned above, it can be obtained thatσ 2 sk → σ 2 sk , a.s., which proves the result of strong consistency in Theorem 4.
To obtain the asymptotic distribution ofσ 2 sk , we firstly transform (A.13) into the following form, We observe that s 2 k (t) − (μ 0 sk ) 2 is a sequence of i.i.d. real random variables with mean σ 2 sk and finite variance Var[s 2 k (1)] under Assumptions 1 and 2, which implies that the first term on the right side of (A.14) converges in distribution to a normal distribution with mean zero and variance Var[s 2 k (1)], because of Lindberg-Levy central limit theorem (see Rao, 1973, p. 127). Combining the strong consistency and asymptotic normality ofμ sk and the limit properties of sequence of random variables (see Rao, 1973, p. 122), it can be shown that the second term converges in distribution to a normal distribution with mean zero and variance 2(σ 2 + σ 2 sk )(μ 0 sk ) 2 . After some calculations, it also can be shown by using Lemma 3 that the third term converges to zero in distribution. Therefore, combining (A.14) and the results mentioned above, it can be obtained that √ T (σ 2 sk − σ 2 sk ) converges in distribution to a normal distribution with mean zero and variance σ * sk = Var[s 2 k (1)] + 2(σ 2 + σ 2 sk )(μ 0 sk ) 2 , which proves the result of asymptotic normality in Theorem 4.