Optimal soot blowing and repair plan for boiler based on HJB equation

The optimization of boiler soot blowing strategy and repair plan is investigated based on Hamilton–Jacobi–Bellman (HJB) equation in this paper. From the perspective of soot, we build a Markov process with three modes to describe the boiler running process. For the sake of applying HJB equation, a cost function is constructed according to the built Markov model, and HJB equation is derived by using optimality principle. The optimal soot blowing strategy and repair plan can be obtained by solving HJB equation. Particularly, the properties of value function, which are regarded as the optimality conditions, can ensure the existence and uniqueness of HJB equation solution. On this basis, Kushner method is used to overcome the difficulty of obtaining analytical solution of HJB equation. Moreover, numerical simulations are performed to verify the effectiveness of the proposed method and analyse the robustness of the obtained soot blowing strategy and repair plan when the system parameters vary.


Introduction
At present, there are many power generation methods of using clean energy, such as wind power, tidal power, and photo-voltaic power generation etc. But the thermal power generation is still the main power generation method [1], in which coal-fired power plants produce 41% of global electricity [2]. Although the locations of thermal power plants are flexible, the initial investment costs are low, and the operations are stable and reliable, the resource utilization of thermal power generation is low due to the deposited soot, which includes slagging [3] and fouling [4].
Fortunately, the soot-blowers can clean the deposited soot by using water, air or steam to improve the boiler efficiency. In fact, when more soot blowing operations are performed, the soot thickness deposited in the boiler can keep thinner CONTACT Jie Wen wenjie015@gmail.com to obtain higher efficiency, but the cost of soot blowing will increase. On the contrary, if less soot blowing operations are performed, the cost of soot blowing can decrease, but the thickness of deposited soot will be too thick, which leads to low efficiency. Thereby, the cost of soot blowing and boiler efficiency are in conflict. In general, the soot blowing operations are performed at regular intervals, which is the simplest soot blowing strategy, and the soot blowing intervals are determined based on the experience. Due to the fixity of soot blowing intervals and the inflexibility of soot blowing operations, the contradiction between cost and efficiency cannot be handled by the fixed interval soot blowing strategy very well. Thus, it is necessary to find a more reasonable or even optimal boiler soot blowing strategy to balance the soot blowing cost and boiler efficiency, i.e. determining the time interval of soot blowing or when to blow and how long to blow. Moreover, the large steam flow of soot blowing not only takes a lot of cost but also may damage the equipment, so the steam flow needs to be considered in the soot blowing strategy.
Taking into account the significance of soot blowing operations, lots of researchers have paid attention to the studies of boiler soot blowing long ago [5,6]. In earlier studies, researchers analysed the characterization of soot depositing processes to control soot blower [7][8][9][10], and optimizing soot blowing systems were also developed based on the heat-flux transfer monitoring system [11][12][13]. On this basis, a number of advanced control methods were applied into fouling assessment and soot blowing design of boiler systems, e.g. expert system [14], fuzzy control [15], support vector machine [16,17], relevance vector machine [18]. In particular, C. Cortés et al. used Artificial Neural Networks (ANN) to optimize boiler soot blowing [19], based on which the neural network model was proposed by Teruel et al. to predict fouling in a real boiler [20]. ANN was also combined with expert systems [21] or thermodynamic model [22] to optimize fouling control and soot blower operations. Recently, Kumari et al. proposed an optimized soot blowing strategy based on a dynamical nonlinear regression model to determine the critical cleanliness factor and the duration of soot blowing cycle [23].
The optimization of boiler soot blowing is a typical optimal control problem, so the most natural tool of handling this problem is the optimal control theory, which is not considered and not used from the related literature review. As the central to optimal control theory [24], HJB equation, which is a partial differential equation, is a necessary and sufficient condition for an optimum, and has been widely used in macro systems [25][26][27] and micro systems [28], thus we apply HJB equation to optimize boiler soot blowing strategy in this paper. As the solution of HJB equation, value function can give the minimum cost for a given dynamical system with an associated cost function. Thus, we build the dynamical model of boiler soot blowing first, which is a Markov process with three modes, and construct a suitable cost function, involving soot blowing strategy and repair plan. According to the dynamical model and cost function, HJB equation can be derived based on the optimality principle. On this basis, the optimal soot blowing strategy and repair plan can be obtained via solving HJB equation, and the properties of value function provide the optimality conditions to ensure the existence and uniqueness of HJB equations solution. However, it is difficult to solve HJB equation analytically, so numerical algorithms need using to obtain the numerical solution. Though there are many numerical algorithms that can solve HJB equation [29][30][31][32], Kushner method [33], which is one of classical Markov chain approximation methods, is selected to solve the HJB equation because that the model of boiler soot blowing built in this paper is a Markov process. Moreover, numerical simulations are performed to verify the effectiveness of proposed method. The results of HJB method optimizing soot blowing for the boiler model with two operation modes were presented in our previous works [34,35]. Compared with [34,35], the main contributions of this paper are reflected in two ways: (1) We extend the two operation modes to three operation modes, i.e. considering the repair mode; (2) Both of soot depositing rate and soot blowing rate have the exponential form instead of linear form, which is more consistent with the actual situation.
The rest of this paper is organized as follows. In Section 2, the dynamical model of boiler soot blowing is built and the cost function is constructed, based on which HJB equation is derived. Section 3 presents the optimality conditions and Kushner method. Numerical simulations are used to verify the effectiveness of HJB method and analyse robustness in Section 4. Section 5 concludes this paper.

Dynamical model of boiler soot blowing
For power plant, the coal feeders feed the pulverized coal into the boilers, and the throttles are used to adjust the air-fuel ratio. Through absorbing the radiant energy, water will become high temperature and high pressure steam, and then the steam turbines transfer the thermal energy to mechanical energy, which is used by generators to become electricity finally. During the operation of power plant boilers, the burning of pulverized coal produces soot. When the soot is deposited to a certain thickness, soot blowing operations with the moderate steam flow should be carried out to mitigate the effects of deposited soot on boiler efficiency. However, the soot blowing operations cannot always clean the soot completely each time. As time goes by, there will be more and more deposited soot that is not cleaned. At this time, the special soot blowing operations are required to perform, such as using steam flow exceeding normal level. Therefore, the boiler running process can be divided into three modes from the perspective of soot, including soot depositing mode, soot blowing mode and repair mode, which are denoted by 1, 2 and 3, respectively. The operation mode set is denoted by B, i.e. B = {1, 2, 3}, and then the boiler soot blowing system can be described by a Markov process mathematically. Let λ αβ represent the transition rate from mode α to mode β for α, β ∈ B, which satisfies the following conditions [35][36][37][38] and then all transition rates can compose the transition rate matrix where λ 12 = ω d , whose reciprocal represents the soot depositing time, describing 'when to blow'; λ 21 = ω b , whose reciprocal represents the soot blowing time, describing 'how long to blow'; λ 13 = ω r , whose reciprocal represents the boiler repair time, describing 'when to clean up soot thoroughly'; The reciprocal of λ 31 describes 'how long to finish the operation of cleaning up soot thoroughly', which is set as constant in this paper and denoted by C rt . The diagram of boiler operation modes transfer is shown in Figure 1.If the boiler operation mode is denoted as , the transition probability matrix P can be written as where where P symbolizes the conditional probability operator, and lim δt→0 o(δt) δt = 0. On this basis, the boiler soot thickness x(t) satisfies the ordinary differential equationsẋ where d(t) and b(t) are the soot depositing rate and soot blowing rate, respectively. According to (4a), the boiler soot thickness is determined by not only soot depositing rate but also soot blowing rate, which matches reality and intuition. Taking into account the exponential form of thermal efficiency curve and the relationship between soot thickness and thermal efficiency [39], the soot depositing rate d(t) is written as where ξ and μ are positive constant coefficients. Now, we consider the soot blowing rate in (4a). When the soot is thick, the soot is blown off more easily; on the other hand, the larger the steam flow, the faster the soot blowing rate. So the soot blowing rate b(t) is described as where r(t) is the time varying steam flow and can be adjusted on demand; ψ(x(t), r(t)) = c bc · r(t) · (e x(t) − 1) with positive coefficient c bc , which indicates that the soot blowing rate b(t) is positively correlated with the soot thickness x(t) and steam flow r(t). The curves of d(t) and b(t) with ξ , μ and c bc , r as the variables are shown in Figure 2, from which the curve changes are similar to the intuition and actual conditions. From (4b), one can see that the soot thickness is not continuous, and the discontinuity occurs the transfer from the repair mode to the soot depositing mode. Therefore, Equations (1)-(6) describe the dynamical model of boiler soot blowing. The construction idea of boiler soot blowing model in this subsection is from the manufacturing models in [36][37][38]. Relative to the manufacturing system, the soot depositing mode, soot blowing mode and repair mode of boiler can be regarded as the operational mode, maintenance mode and repair mode of manufacturing models, respectively. Thus, we can consider the optimization of boiler soot blowing from the standpoint of equipment health management in this paper.

HJB equation
In order to optimize soot blowing operation and repair plan by applying HJB method, HJB equation should be derived first. Considering the characteristic of HJB equation, we construct a cost function based on the boiler soot blowing model built in subsection 2.1 as follows where E{· | x, α} symbolizes the conditional expectation operator; ρ is the discount rate; in which c d is the cost per unit time of decreased effectiveness caused by deposited soot; c b = Kr is the cost per unit time of soot blowing with a constant coefficient K; c r is the cost of a single repair operation; The indicator function Ind(·) is defined as Now, we analyse how the cost function (7) is constructed. In cost function (7), consists of three items as shown in (8), which correspond to the cost in soot depositing mode, soot blowing mode and repair mode, respectively. In soot depositing mode, the cost is main caused by efficiency losses due to the deposited soot, and the cost is proportional to the soot thickness, so we use c d x(t) to calculate the cost in soot depositing mode, which is the first item of right-hand side in (8). In soot blowing mode, the cost contains the steam cost apart from the cost caused by deposited soot, and the steam cost is proportional to the steam flow, so we use c b r(t) + c d x(t) to define the cost in soot blowing mode, which corresponds to the second item of right-hand side in (8). In repair mode, the cost is mainly begotten due to repair operation, and we assume that the cost of single repair operation is fixed, so c r is used to describe the cost of repair mode, which is the third item of G(α, x, ω d , ω b , r, ω r ). Besides, given the time value of money, the discounted cost function is adopted in this paper. The discount is usually associated with a discount rate [40,41], so the discount rate ρ is involved in the cost function (7). Considering the boiler soot blowing model is a Markov process, we also use the conditional expectation operator to obtain average cost in (7). Integrating all factors, we construct the cost function J(α, x, ω d , ω b , r, ω r ) as shown in (7) with G(α, x, ω d , ω b , r, ω r ) in (8). Compared with the cost functions constructed in [34] and [35], the cost function proposed in this paper consists of three items, while the cost functions in [34,35] only consist of two items. The added item is the cost of a single repair operation, which is one of the main contributions of this paper. According to [42], we can rewrite the cost function (7) as (9) and the associated value function is where R + is the set of positive real numbers; (α) is the set of admissible control policies and defined as in which the 'min' and 'max' in the superscript of variables ω d , ω b , r and ω r represent the minimum and maximum of corresponding variables, respectively. According to the optimality principle, HJB equation can be written as From (10) and (12), one can see that the solution of HJB equation is the value function, which provides the minimum cost for the boiler soot blowing system (1)-(6) with the associated cost function (9). Thus, the optimization of soot blowing in this paper is described as follows: For the initial soot thickness x and initial boiler operation mode α, to seek the soot blowing strategy (α, x, ω d , ω b , r) and repair plan ω r in (α) so as to minimize J (α, x, ω d , ω b , r, ω r ) subject to (1)-(6), i.e.
It is obvious that one can obtain the optimal soot blowing strategy ω d , ω b , r and repair plan ω r by solving the HJB equation (12), so the next major work is to solve the HJB equation.

The solution of HJB equation
In this section, the properties of value function, which are regarded as the optimality conditions, are presented to ensure the existence and uniqueness of HJB equation's solution first, and Kushner method is applied to obtain the numerical solution in order to overcome the difficulty of getting analytical solution. The properties of value function are given by Lemmas 3.1 and 3.2.

Lemma 3.1: The value function v(α, x) is convex.
Proof: The lemma can be proved according to the definition of convex function in [43]. First, one should prove the cost function J(α, x, ω d , ω b , r, ω r ) is convex by applying the linearity of c d x(t)(Ind(α(t) = 1) + Ind(α(t) = 2)) and c b r(t)Ind(α(t) = 2) + c r Ind(α(t) = 3), i.e. proving the inequality holds for arbitrary soot thicknesses x 1 and x 2 , in which u 1 and u 2 are denoted as the arbitrary two admissible control strategies in (α). Then, the convexity of value function can be proved based on the convex function's properties described in [43]. The detailed derivation can be found in our previous works [34,35].
Based on Lemmas 3.1 and 3.2, the following theorem ensures the existence and uniqueness of HJB equation's solution.

Theorem 3.3: The value function v(α, x) is the unique viscosity solution of HJB equation (12).
Proof: The proof is divided into two parts: (1) The value function v(α, x) is a viscosity solution of HJB equation; (2) HJB equation has unique viscosity solution. In order to prove the first part, one should apply the continuity condition of v, and prove |v(α, x)| ≤ C(1 + |x| k ) as shown in [35], based on which one can prove that v is both a viscosity subsolution and a viscosity supersolution. On this basis, the second part can be proved by applying using reduction to absurdity with the help of Lemma 3.2 and the inequality Now we show that the inequality (14) holds. For G(α, x, ω d , ω b , r, ω r ), we have When α = 1, with k ≥ 1 and C ≥ c d . When α = 2, if c b r max ≥ c d , with k = 1 and C ≥ c b r max ; if c b r max < c d , with k = 1 and C ≥ c d . When α = 3, with C ≥ c r . According to the inequalities (17), (18) and (19), the inequality (14) holds if C ≥ max(c d , c b r max , c r ) and k ≥ 1. Due to the boundedness of soot depositing rate and soot blowing rate, we can replace a = ρ sup u |u−z| with a = ρ sup b(t),d(t) |b(t)−d(t)| in the proof of Theorem G.1 in [44], and the rest proof of second part is similar to the previous work [35], so we ignore the details of the proof here. Theorem 3.3 ensures the solvability of soot blowing optimization problem considered in this paper. Nevertheless, it is difficult to obtain the analytical solution of HJB equation, so we use numerical methods to solve the HJB equation in this paper. Although there are many numerical algorithms for solving HJB equation [29][30][31][32][33], Kushner method is applied because we built a Markov process model as the dynamical model of boiler soot blowing. If the finite difference interval of soot thickness is denoted as h x , then the first-order derivative of value function v(α, x) can be approximated as where (20) into HJB equation (12), Particularly, the theorem in [45] shows In order to obtain v h x (α, x), we rewrite the parameters in (21) for different boiler operation modes. More concretely, h x (α, ω d , ω b , r, ω r ) in (22) for α ∈ B = {1, 2, 3} can be written as (23) while p β (α, ω d , ω b , r, ω r ) for α, β ∈ B = {1, 2, 3} can be written as and then Equation (21) can be solved by the algorithm as shown in Algorithm 1, whose main idea from [33]. For the purpose of determining u n , the policy improvement technique, i.e. the step sizes of corresponding variables are halved for each additional iteration, is adopted in Step 3 of the algorithm. In order to improve algorithm efficiency, the boundary conditions can be ameliorated as presented in [35].
Step 2: Step 3: Adopt policy improvement technique to determine u n so that Step 4: Calculate variables c min := ρ 1−ρc , c max := ρ 1−ρ c, wherec := min Besides, for a Markov process with n modes, the limiting probabilities satisfy where Q(·) and π(·) are the transition matrix and limiting probabilities of the Markov process, respectively. The dynamical model of boiler soot blowing built in this paper is a Markov process with three modes, one can get that according to (30), from which we can obtain the limiting probabilities of three operation modes as follows where A = λ 12 λ 31 + λ 13 λ 21 + λ 21 λ 31 .

Numerical simulations
In this section, we will perform numerical simulations to verify the effectiveness of the proposed method and analyse the robustness of the obtained soot blowing strategy when the system parameters vary by means of sensitivity analyses. The parameters setting of numerical simulations are shown in Table 1, in which h ωr , h ωd , h ωb and h r represent the initial difference interval length of subscript variables, respectively. In order to verify the effectiveness of HJB method in the optimization of boiler soot blowing, numerical simulations are performed by using the parameters in Table 1. The simulation results are shown in Figure 3, in which Figure 3(a) is the curve of |c max − c min | with iterations, Figure 3(b,c) shows the optimal soot blowing strategy and repair plan obtained by solving HJB equation numerically, while Figure 3(d) presents the value functions in all operation modes.
From Figure 3(a), one can see that |c max − c min | decreases gradually as the increase of iterations, which indicates that the policy improvement technique used in this paper is effective. In Figure 3(b,c), the optimal soot blowing strategy and repair plan all have the form of Bang-Bang (BB) control, and the two boundary values of optimal strategy and repair time are the extreme values of corresponding variables, which is similar to the results in our previous works [34,35]. The value functions in Figure 3(d) show different trends as the increase of soot thickness for different operation modes: the value functions in mode 1 and mode 3 increase, while the value function in mode 2 increases first and then decreases.
In order to analyse this phenomenon, optimal soot blowing time, value function, optimal steam flow and the absolute value of value function gradient in mode 2 are put in a figure as shown in Figure 4, from which one can see that the change of value function in mode 2 is divided into three stages.
(1) When steam flow and soot bowing time all take minimum values, the value function is bigger and bigger when the soot thickness increases; (2) When steam flow takes maximum value and soot bowing time takes minimum value, the value function decreases at a lower rate as the increase of soot thickness; (3) When steam flow and soot bowing time all take maximum values, the value function decreases at a higher rate first, then the decline rate becomes smaller gradually and decreases at a lower rate.
The reasons are listed as follows: (1) In stage I, the cost caused by the low efficiency due to deposited soot is dominant in the overall cost. When the soot thickness becomes thicker and thicker, the boiler efficiency will decrease, thereby the more cost is spent, which leads to the increase of value function; (2) In stage II, the efficiency is increased because of the use of maximum steam flow, so the cost is reduced, and the value function decreases; (3) In stage III, due to the use of maximum soot blowing time and maximum steam flow rate, the soot thickness is reduced faster, and the efficiency is improved greatly, so the cost is reduced obviously, which leads to the rapid  decline of value function. However, as the soot thickness increases further, the improvement of efficiency is slow because that the soot thickness is too thick even the maximum steam flow and soot blowing time are used, thus the cost reduces with less rate, which leads to the gentle decline of value function.
On the other hand, we have all information about the value functions and probability limits from (21) and (32), so we can represent the average cost, which is shown in Figure 5, instead of the value function of each mode as shown in Figure 3(d). From Figure 5, we can see that the average cost is growing almost monotonically as the increase of soot thickness, i.e. the average cost is usual lower when the soot is thinner. However, there are two inflection points that need special attention: when the soot thickness is 7.86, the average cost is lowest if the soot thickness is more than 6, which means that it is a better choice to perform soot blowing operation or repair boiler; when the soot thickness is more than 9.88, the average cost will increase rapidly, which means the operations of soot blowing and boiler repair should be performed before the soot thickness reaches 9.88. Moreover, we extend the sensitivity analyses to illustrate the robustness of the soot blowing strategy and repair plan obtained by the proposed method when the system parameters vary. From the system model descriptions in (5) and (6), the system parameters contain ξ , μ and c bc . In order to analyse the robustness, take the percentage change of parameters as ±5%, ±10%, ±15%, ±20%, respectively, and the optimal switch soot thickness for each decision in soot blowing strategy and repair plan are shown in Figure 6, from which one can see that the robustness of ω b and r are better than that of ω d and ω r , i.e. ω d and ω r are more susceptible to the variety or uncertainty of parameters.

Conclusions
The optimal soot blowing strategy and repair plan are established based on HJB equation and Kushner method in this paper. A Markov process with three modes is built as the dynamical model to describe the boiler soot blowing. As a typical optimal control problem, HJB equation, which belongs to the optimal control theory, is applied to obtain optimal soot blowing strategy and repair plan. For deriving HJB equation, we construct a cost function according to the built boiler soot blowing model, based on which the properties of value function can ensure the existence and uniqueness of HJB equation solution. Considering the difficulty of solving analytical solution of HJB equation, Kushner method is applied to obtain the numerical solution. Particularly, the numerical simulation results also indicate that the HJB method is effective for optimizing soot blowing and repair plan, and helps understand the effects and impact of soot blowing strategy. The further research includes: (1) The repair is assumed to be perfect repair in the boiler soot blowing model of this paper, but the repair for the boiler is mostly imperfect repair in reality, which makes sense to consider the case of imperfect repair; (2) Take into account the randomness of soot depositing process by modifying the model of boiler soot blowing; (3) Consider the impact of boiler age and the repair number on boiler soot blowing model and cost function.

Disclosure statement
No potential conflict of interest was reported by the author(s).