Deterministic_and_Stochastic_Schemes_for_Unconstrained_Optimization.pdf

Optimization problems play a crucial role in countless applications, ranging from finance and engineering to machine learning. This thesis explores the principal techniques for both deterministic and stochastic unconstrained optimization, presenting their key mathematical foundations. These foundations encompass convex optimization in a deterministic context and general methods used in stochastic optimization. While convex optimization assumes a convex cost function, stochastic optimization algorithms, such as stochastic gradient descent (SGD), do not generally require the cost function to be convex. Nevertheless, convexity assumptions often feature in the analysis of these algorithms to ensure convergence guarantees and the existence of a unique minimum.

This thesis provides an overview of traditional deterministic gradient descent, including its variations such as fixed step size and backtracking line search. It then focuses on stochastic gradient descent (SGD) and its applications. Moreover, derivative-free optimization methods, including simulated annealing (SA) and particle swarm optimization (PSO), are discussed for handling multi-dimensional non-convex functions. A comparison is made of the performance of these algorithms across different scenarios, and the importance of convexity assumptions in the analysis of various optimization algorithms, including SGD, is examined. The impact of hyper-parameter tuning on the convergence of these algorithms is also evaluated. The results of this analysis offer valuable insights into the selection and application of optimization algorithms, essential for practitioners and researchers in fields including, but not limited to, machine learning, deep learning, finance, engineering, and operations research.