A Conjugate Gradient Method with Strong Wolfe-Powell Line Search for Unconstrained Optimization

In this paper, a modified conjugate gradient method is presented for solving large-scale unconstrained optimization problems, which possesses the sufficient descent property with Strong Wolfe-Powell line search. A global convergence result was proved when the (SWP) line search was used under some conditions. Computational results for a set consisting of 138 unconstrained optimization test problems showed that this new conjugate gradient algorithm seems to converge more stable and is superior to other similar methods in many situations.


Introduction
Nonlinear conjugate gradient methods are well suited for large-scale problems due to the simplicity of their iteration and their very low memory requirements, that is, they are designed to solve the following unconstrained optimization problem: (1) where R R f n  : is a smooth, nonlinear function, and its gradient is denoted by The iterative formula of the conjugate gradient methods is given by ,...
In the already-existing convergence analysis and implementations of the conjugate gradient method, the weak Wolfe-Powell (WWP) line search conditions are as follows: The strong Wolfe-Powell conditions consist of (4) and, Furthermore, the sufficient descent property, namely, Where c is, a positive constant, is crucial to ensure the global convergence of the nonlinear conjugate gradient method with the inexact line search techniques [12,13].
Recently, Wei et al. [31] gave a variant of the PRP method which is called the WYL method. Zhang studied and improved based on WYL a new conjugate gradient method, NPRP, and he proved that the NPRP method satisfied descent condition under strong Wolfe line search. Moreover, Zhang et al. proposed another modified method known as the MPRP method, where Dai and Wen [32] proposed a modified NPRP method known as the DPRP method. In this paper, enlightened by the above ideas, a modified PRP conjugate gradient method was proposed as follows: where, HRM denotes Hamoda, Rivaie, and Mamat. According to the results obtained by [30], the value of the parameter u can be set to 1 0   u , but in this paper, we will test our new method with an arbitrary value 4 . 0  u Therefore, we first provide the following algorithm:
Step 5: Set The following assumptions are often used in previous studies of the conjugate gradient methods:

Assumption B
In some neighborhood N of  , the objective function is continuously differentiable, and its gradient is Lipschitz continuous, that is, there exists a constant In 1992, Gilbert and Nocedal introduced the property (*) which plays an important role in the studies of CG methods. This property means that the next research direction approaches the steepest direction automatically when a small step-size is generated, and the step-sizes are not produced successively [33].

Property (*)
Consider a CG method of the form (2) and (3). Suppose that, for all 0 where  and  are two positive constants. We say that the method has property (*), if there exist constants The following lemma shows that the new method HRM k  has the property (*).

Lemma 2.1
Consider the method of form (2) and (3), Suppose that Assumptions A and B hold, then, the method HRM k  has property (*). Proof (8) and (10) The proof is finished.

The global convergence properties
The following theorem shows that the formula HRM with SWP line search possess the sufficient descent condition Hence, we obtain Using (6) and (11) By repeating this process and the fact (16) can be written as (17) By making the restriction (17) turns out to be This implies that (7) holds. The proof is complete.
The following condition known as Zoutendijk condition was used to prove the global convergence of nonlinear CG methods [15,34].
The proof had been given in [35,36]. In [10], Gilbert and Nocedal introduced the following important theorem:

Numerical Experiments
In the present numerical experiments, we selected thirty-two different functions which had been earlier considered in [37][38][39] for both small-scale and large-scale optimization problems. Each of these functions was tested with different variables which lie in the range from 2 to 10,000. We tested a set of 138 problems with strong Wolfe-Powell line search. The algorithm was implemented using MATLAB R2011b (7.13.0.564), applying the strong Wolfe-Powell line search. All of the numerical experiments were run on the same PC with an Intel (R) CoreTM i3-M350 (2.27GHz) CPU, 4GB of RAM, and the Windows 7 operating system. In order to assess the reliability of the new proposed method, HRM, we tested this method against the well-known classical and modified methods of the FR, PRP, MPRP, and DPRP methods using the same problems, and assumed that the best method should require fewer iterations and less CPU time. All of these algorithms terminated when  Table 1. In some cases, the computation stopped due to the failure of the line search to find the positive step size, and thus it was considered as a failure. In addition, we considered the search to have failed if the number of iterations exceeded 1,000 or CPU execution time exceeded 500 seconds. Numerical results were relatively compared with the CPU time and number of iterations. The performance results are shown in Figures 1  and 2 respectively, using a performance profile introduced by Dolan and More [40]. Under a strong Wolfe-Powell line search, the performance profile of all methods measured by the number of iterations required is shown in Figure 1, and the performance profile based on the CPU time used is in Figure 2. The shapes of the profile plots in both Figures 1 and 2 are almost alike. A thorough inspection of the left side of both figures indicates that the lowest curve represents the FR method. Therefore, this method possesses the lowest performance. The top left side curve indicates that the PRP method is the best performer. The curves for methods DPRP, MPRP, and HRM fall in between the two extreme curves. Thus, the performance of this set of methods is in the middle of the sets based the number of iterations and CPU time.
From results shown in Figures 1 and 2, it is evident that the FR method achieved a success rate of only 0.65, while the PRP method scored 0.79, and the DPRP method recorded 0.88. Furthermore, the MPRP method achieved 0.94, and our new method, HRM achieved 1, that is, our new method scored a 100% success rate. Such result indicates that HRM method is the best among the four methods in the perspectives of the number of iterations and the CPU time. Hence, our new method successfully solved all the test problems, and it is competitive with the well-known conjugate gradient methods for unconstrained optimization.

Conclusion
In this paper, we proposed a new conjugate gradient method for unconstrained optimization. Results showed that it could satisfy the sufficient descent condition and converge globally if the strong Wolfe-Powell line search was used. Numerical results show that the HRM method is efficient for the addressed problems.