COMPARATIVE PERFORMANCE ANALYSIS OF SOME ACCELERATED AND HYBRID ACCELERATED GRADIENT MODELS

We analyze a performance profile of several accelerated and hybrid accelerated methods. All comparative methods are at least linearly convergent and have satisfied numerical characteristics regarding tested metrics: number of iterations, CPU time and number of function evaluations. Among the chosen set of methods we numerically show which one is the most efficient and the most effective. Therewith, we derived a conclusion about what type of method is more preferable to use considering analyzed metrics.


ACCELERATED FACTOR IN ACCELERATED GRADI-ENT MODELS
We are analyzing accelerated gradient descent iterations for solving unconstrained optimization problems, mathematically described as: where f : R n → R is an objective function which we want to minimize.For function f we assume that it is uniformly convex and twice continuously differentiable function.Instead of usual iterative optimization schemes, expressed by: we focus on accelerated gradient iterations given by the following expression In ( 2) and (3) x k+1 stays for the next iterative function value, x k is the current iterative function value, t k is the iterative step size value and d k is the search direction vector.In (3) scalar γ k presents an iterative approximation parameter.Many authors confirmed, mostly numerically, that this parameter upgrades performance profile of posed optimization method.Nevertheless, in Stanimirović & Miladinović (2010) a class of methods containing acceleration factor is denoted as the class of accelerated gradient methods.From the analysis presented in Petrović & Kontrec (2017) we can conclude that one of the most efficient way for calculating acceleration parameter is through the features of the second order Taylor's series taken on the objective accelerated iteration.Although there are some alternative modes for deriving the acceleration parameter, we mention here several highly efficient accelerated models with acceleration parameter obtained by the Taylor's development: Andrei (2006Andrei ( , 2008)); Petrović & Stanimirović (2014); Petrović (2015); Stanimirović et al. (2015); Stanimirović & Miladinović (2010).In this regard we display the ex-pressions for acceleration parameters of some above mentioned methods: Regarding the theory of unconstrained optimization methods, we have a unique opinion that there are two crucial elements which defined a relevant iterative optimization scheme.The first one is the vector direction, d k , which directs the minima search.It is usually required to fulfil the descending condition: The second, equally important, is the value of the iterative step length, t k .This element is obtained trough the exact or inexact line search procedure.In practical purpose, the inexact algorithms are certainly more preferable choice for obtaining the optimal step size iterative value.With this regard, from all above exposed, we can rightly conclude that besides these two listed elements the value of acceleration parameter, γ k , is also important and crucial factor for one optimization method.
This paper is organized in the following way.In the second section we give an overview of some important hybrid models and hybridization process applied on accelerated gradient schemes.In the main section 3, we display obtained numerical results of four chosen models, conduct a comparative analysis and bring up a conclusion.

HYBRID MODELS
Some authors investigate a hybrid iterative systems for solving optimization problems.One of the first in this field was Picard.In his work Picard (1890) presented the following set of two iterations for solving optimization problems: Later on, Mann exposed his set of expressions and called it mean value methods in iterations Further on, Ishikawa presented a three-term model as next: In the above displayed schemes v k , z k and y k present the sequences defined by related iterations, parameters {α k }, {β k } ∈ (0, 1) and T : C → C is a mapping defined on nonempty convex subset C of a normed space E.
In a recent research Khan (2013), introduced the following set of relations In the same paper the author shows the advantages of posed process and confirms that so defined model outperforms previous three mentioned methods.
Taking good sides of the iteration set (8) the authors in Petrović et al. (2017) applied accelerated gradient descent SM method, presented in Stanimirović & Miladinović (2010), on this threeterm relation.As a result a hybrid accelerated scheme is developed.We call this iteration the HSM method and it is defined by the expression: where parameter α ∈ (1, 2) and γ k ≡ γ HS M k is iterative acceleration parameter which is computed using the second order Taylor's series of the HSM iteration In Petrović et al. (2017) the authors proved that the HSM method is at least linearly convergent on the set of uniformly convex and strictly convex quadratic functions.Numerical tests confirm significant benefits when the HSM scheme is used instead of its forerunner, the SM iteration.All these advantages indicate that this new hybridization concept can be applied on some other accelerated model and upgrade its features.
In Panić et al. ( 2018) some initial improvement is taken on the HSM iteration and the modified version of the HSM scheme is introduced.This model is denoted as the MHSM method.The improvement regarding the HSM iteration consists in reducing the initial step length value of the Backtracking line search algorithm.Numerical experiments show some betterment compared to the starting HSM method.

NUMERICAL COMPUTATIONS AND CONCLUSIONS DRAWN
In this section we expose comparative analysis of performance profile of four chosen methods.First comparative models is the accelerated gradient SM method presented in Stanimirović & Miladinović (2010).Second one is the hybrid accelerated method HSM from Petrović et al. (2017) which presents a hybridization of the SM method.Third model is the MHSM introduced in Panić et al. ( 2018) and it presents modified version of the HSM, where an initial improvement of starting value in Backtracking line search procedure was taken.Final comparative method is the accelerated gradient TADSS method which is revealed in Stanimirović et al. (2015).We now display algorithms of all listed methods: Algorithm 0.1 SM-method Stanimirović & Miladinović (2010) Require: Objective function f (x) and chosen initial point x 0 ∈ dom( f ).1: Set k = 0 and compute f (x 0 ), g 0 = ∇ f (x 0 ) and take γ 0 = 1.2: If test criteria are fulfilled then stop the iteration; otherwise, go to the next step.3: (Backtracking) Find the step size t k ∈ (0, 1] using Backtracking procedure with ) and g k+1 = ∇ f (x k+1 ).5: Determine the scalar approximation γ k+1 of the Hessian of f at the point x k+1 using γ S M k+1 representation.6: If γ k+1 < 0, then take γ k+1 = 1.7: Set k := k + 1, go to the step 2. 8: Return x k+1 and f (x k+1 ).
1: Set k = 0, compute f (x 0 ), g 0 and take γ 0 = 1.2: If g k < , then go to Step 9, else continue by the next step.
We conducted numerical tests, for each comparative model, on 12 functions from Andrei (2008) for ten different number of variables: 100, 500, 1000, 1500, 2000, 3000, 5000, 7000, 8000, 10000.In the next six tables we reveal achieved results of all tested models.In order to simplify the table representations we paired the results of the SM and the MHSM algorithms and there with the results of the HSM and the TADSS.In the first three tables i.e.Tables (1) (2) and (3) we illustrate the results of obtained number of iterations, CPU time and the number of function evaluations, respectively, for the first pair of methods (SM and MHSM).In the Tables (4), ( 5) and (6) we display the numerical outcomes for the second pair of methods (TADSS and HSM).For all tests the usual exit condition was taken: From last six tables we can count that the TADSS method outperform all others regarding all three analyzed metrics.Considering number of iterations in case of 8 test function TADSS is gives the best results, follows the MHSM with 2 test functions and the SM with 1 test functions.For 1 test function all methods show the same number of iterations, the same number of function evaluations and the same CPU time.Regarding the number of function evaluations, the TADSS upgrades the rest of comparative mathods in 10 test functions, while the SM gives the best results for 1 test function.Considering the CPU time metric, the TADSS shows convincingly best outcomes.
More clearer comparative view can be seen from the next Table (7) where the average values with respect to all three measured characteristics, regarding all four comparative methods, are included.From the previous Table ( 7) we can see that the TADSS in a large degree outperforms the other three schemes regarding all three measured metrics.More precise, considering the number of iterations the TADSS shows almost three times better results than the HSM, approximately 1.7 times better than the MHSM and even 6.5 times better results than the SM method.Regrading the number of function evaluations the TADSS obtains about 7 times lower outcomes compered to the HSM method, more then 4 times lower values compared to the MHSM and about 11.5 times lower number compared to the SM method.When the CPU time is is considered, the TADSS shows nearly 133.5, 159, 0.33 and 450 times faster execution time when compared to the HSM, the MHSM and the SM respectively.This fabulous numerical outcomes in favor to the TADSS scheme are not so surprising since in Stanimirović et al. (2015) the advantage of this method in comparatione to the SM scheme is numerically confirmed.Since in Petrović et al. (2017);Panić et al. (2018) betterment of the hybrid and modified hybrid version of the SM method in comparatione to the SM iteration is detected, it is not surprising that the HSM and MHSM in Table (7) give better outcomes than the SM model.
The previous analysis lead us to an interesting question: if we make a hybrid model of the TADSS method would it be more efficient than the very TADSS iteration?The answer on this question will be revealed trough some further investigations.
From presented facts we can conclude that the hybridization process presented in Khan (2013) is a good way for improving a performance profile of the certain optimization method.

Table 1 .
Numerical results of SM and MHSM methods tested on 12 large scale test functions regarding number of iterations metric.

Table 2 .
Numerical results of SM and MHSM methods tested on 12 large scale test functions regarding CPU time metric.

Table 3 .
Numerical results of SM and MHSM methods tested on 12 large scale test functions regarding number of function evaluations metric.

Table 4 .
Numerical results of TADSS and HSM methods tested on 12 large scale test functions regarding number of iterations metric.

Table 5 .
Numerical results of TADSS and HSM methods tested on 12 large scale test functions regarding CPU time metric.

Table 6 .
Numerical results of TADSS and HSM methods tested on 12 large scale test functions regarding number of function evaluations metric.

Table ( 4
) contains the number of iterations data, Table(5) contains CPU time data and Table (6) contains the number of function evaluations data, respectively, for both models.

Table 7 .
Average numerical outcomes for 12 test functions tried out on 10 numerical experiments in each iteration.