Minimizers#

zfit supplies wrappers for different minimizers from multiple libraries. Most of the are local minimizers (such as Minuit, Ipyopt or ScipyLBFGSB are) while there are also a few global ones such as the NLoptISRES or NLoptStoGO.

While the former are usually faster and preferred, they depend more on the initial values than the latter. Especially in higher dimensions, a global search of the parameters can increase the minimization time drastically and is often infeasible. It is also possible to couple the minimizers by first doing an approximate global minimization and then polish the minimum found with a local minimizer.

All minimizers support similar arguments, most notably tol which denotes the termination value. This is reached if the value of the convergence criterion, which defaults to EDM, the same that is also used in Minuit.

Other than that, there are a also a few minimizer specific arguments that differ from each minimizer.

They all have the exact same minimization method minimize() which takes a loss, parameters and (optionally) a FitResult from which it can take information to have a better start into the minimization.

The following visualization shows how different minimizers perform on a complex version of the Rosenbrock function. Each minimizer starts from 5 different initial points (marked with symbols) and follows a path to the minimum. The contour plot represents the function values, with darker colors indicating lower values. For each minimizer, the plot shows the number of function evaluations, gradient calculations, and execution time.

Minuit#

Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit. The iminuit package is a fast, time-proven minimizer based on the Minuit2 C++ library. It is an especially robust minimizer that finds the local minimum quiet reliably for complicated fits. For large fits with hundreds of parameters, other alternatives are generally more performant. It is however, like all local minimizers, still rather dependent on close enough initial values.

Minuit minimizer paths on a complex Rosenbrock function Minuit minimizer paths on a complex Rosenbrock function

zfit.minimize.Minuit([tol, mode, gradient, ...])

Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit.

Levenberg-Marquardt#

Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.

LM minimizes a function by iteratively solving a locally linearized version of the problem. Using the gradient (g) and the Hessian (H) of the loss function, the algorithm determines a step (h) that minimizes the loss function by solving \(Hh = g\). This works perfectly in one step for linear problems, however for non-linear problems it may be unstable far from the minimum. Thus a scalar damping parameter (L) is introduced and the Hessian is modified based on this damping.

LevenbergMarquardt minimizer paths on a complex Rosenbrock function LevenbergMarquardt minimizer paths on a complex Rosenbrock function

zfit.minimize.LevenbergMarquardt([tol, ...])

Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.

Ipyopt#

Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.

This implemenation uses the IPyOpt wrapper

Ipopt (Interior Point Optimizer, pronounced “Eye-Pea-Opt”) is an open source software package for large-scale nonlinear optimization. It can be used to solve general nonlinear programming problems It is written in Fortran and C and is released under the EPL (formerly CPL). IPOPT implements a primal-dual interior point method, and uses line searches based on Filter methods (Fletcher and Leyffer).

Ipyopt minimizer paths on a complex Rosenbrock function Ipyopt minimizer paths on a complex Rosenbrock function

zfit.minimize.Ipyopt([tol, maxcor, ...])

Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.

Scipy#

The following visualizations show how different Scipy minimizers perform on the complex Rosenbrock function.

BFGS#

Local, gradient based quasi-Newton algorithm using the BFGS algorithm.

BFGS, named after Broyden, Fletcher, Goldfarb, and Shanno, is a quasi-Newton method that approximates the Hessian matrix of the loss function using the gradients of the loss function. It stores an approximation of the inverse Hessian matrix and updates it at each iteration. For a limited memory version, which doesn’t store the full matrix, see L-BFGS-B.

ScipyBFGS minimizer paths on a complex Rosenbrock function ScipyBFGS minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyBFGS([tol, c1, c2, ...])

Local, gradient based quasi-Newton algorithm using the BFGS algorithm.

LBFGSB#

Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.

Limited-memory BFGS is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS) using a limited amount of memory (or gradients, controlled by maxcor).

L-BFGS borrows ideas from the trust region methods while keeping the L-BFGS update of the Hessian and line search algorithms.

ScipyLBFGSB minimizer paths on a complex Rosenbrock function ScipyLBFGSB minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyLBFGSB([tol, maxcor, ...])

Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.

TrustConstr#

ScipyTrustConstr minimizer paths on a complex Rosenbrock function ScipyTrustConstr minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyTrustConstr([tol, ...])

Trust-region based local minimizer.

Powell#

Local minimizer using the modified Powell algorithm.

ScipyPowell minimizer paths on a complex Rosenbrock function ScipyPowell minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyPowell([tol, verbosity, ...])

Local minimizer using the modified Powell algorithm.

SLSQP#

ScipySLSQP minimizer paths on a complex Rosenbrock function ScipySLSQP minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipySLSQP([tol, gradient, ...])

Local, gradient-based minimizer using tho Sequential Least Squares Programming algorithm.name.

TruncNC#

ScipyTruncNC minimizer paths on a complex Rosenbrock function ScipyTruncNC minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyTruncNC([tol, maxcg, ...])

Local, gradient based minimization algorithm using a truncated Newton method.

COBYLA#

UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.

COBYLA constructs successive linear approximations of the objective function and constraints via a simplex of n+1 points (in n dimensions), and optimizes these approximations in a trust region at each step.

ScipyCOBYLA minimizer paths on a complex Rosenbrock function ScipyCOBYLA minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyCOBYLA([tol, verbosity, ...])

UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.

TrustNCG#

ScipyTrustNCG minimizer paths on a complex Rosenbrock function ScipyTrustNCG minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyTrustNCG([tol, ...])

PERFORMS POORLY! Local Newton conjugate gradient trust-region algorithm.

Dogleg#

This minimizer requires the hessian and gradient to be provided by the loss itself.

ScipyDogleg minimizer paths on a complex Rosenbrock function ScipyDogleg minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyDogleg([tol, ...])

This minimizer requires the hessian and gradient to be provided by the loss itself.

ScipyTrustKrylov#

ScipyTrustKrylov minimizer paths on a complex Rosenbrock function ScipyTrustKrylov minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyTrustKrylov([tol, ...])

PERFORMS POORLY! Local, gradient based (nearly) exact trust-region algorithm using matrix vector products with the hessian.

NewtonCG#

ScipyNewtonCG minimizer paths on a complex Rosenbrock function ScipyNewtonCG minimizer paths on a complex Rosenbrock function

zfit.minimize.ScipyNewtonCG([tol, gradient, ...])

WARNING! This algorithm seems unstable and may does not perform well!

NLopt#

The following visualizations show how different NLopt minimizers perform on the complex Rosenbrock function:

LBFGS#

Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.

This is most probably the most popular algorithm for gradient based local minimum searches and also the underlying algorithm in the Minuit minimizer that is also available as Minuit.

NLoptLBFGS minimizer paths on a complex Rosenbrock function NLoptLBFGS minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptLBFGS([tol, maxcor, ...])

Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.

Truncated Newton#

NLoptTruncNewton minimizer paths on a complex Rosenbrock function NLoptTruncNewton minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptTruncNewton([tol, ...])

Local, gradient-based truncated Newton minimizer using an inexact algorithm.

SLSQP#

NLoptSLSQP minimizer paths on a complex Rosenbrock function NLoptSLSQP minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptSLSQP([tol, verbosity, ...])

Local gradient based minimizer using a sequential quadratic programming.

MMA#

NLoptMMA minimizer paths on a complex Rosenbrock function NLoptMMA minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptMMA([tol, verbosity, ...])

Method-of-moving-asymptotes for gradient-based local minimization.

CCSAQ#

NLoptCCSAQ minimizer paths on a complex Rosenbrock function NLoptCCSAQ minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptCCSAQ([tol, verbosity, ...])

MMA-like minimizer with simpler, quadratic local approximations.

Subplex#

NLoptSubplex minimizer paths on a complex Rosenbrock function NLoptSubplex minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptSubplex([tol, verbosity, ...])

Local derivative free minimizer which improves on the Nealder-Mead algorithm.

COBYLA#

Derivative free simplex minimizer using a linear approximation with trust region steps.

COBYLA (Constrained Optimization BY Linear Approximations) constructs successive linear approximations of the objective function and constraints via a simplex of n+1 points (in n dimensions), and optimizes these approximations in a trust region at each step.

NLoptCOBYLA minimizer paths on a complex Rosenbrock function NLoptCOBYLA minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptCOBYLA([tol, verbosity, ...])

Derivative free simplex minimizer using a linear approximation with trust region steps.

MLSL#

Global minimizer using local optimization by randomly selecting points.

“Multi-Level Single-Linkage” (MLSL) is an algorithm for global optimization by a sequence of local optimizations from random starting points. MLSL is distinguished by a “clustering” heuristic that helps it to avoid repeated searches of the same local optima, and has some theoretical guarantees of finding all local optima in a finite number of local minimizations.

NLoptMLSL minimizer paths on a complex Rosenbrock function NLoptMLSL minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptMLSL([tol, population, ...])

Global minimizer using local optimization by randomly selecting points.

StoGO#

NLoptStoGO minimizer paths on a complex Rosenbrock function NLoptStoGO minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptStoGO([tol, randomized, ...])

Global minimizer which divides the space into smaller rectangles and uses a local BFGS variant inside.

BOBYQA#

Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.

This is an algorithm derived from the BOBYQA subroutine of M. J. D. Powell, converted to C and modified for the NLopt stopping criteria. BOBYQA performs derivative-free bound-constrained optimization using an iteratively constructed quadratic approximation for the objective function.

NLoptBOBYQA minimizer paths on a complex Rosenbrock function NLoptBOBYQA minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptBOBYQA([tol, verbosity, ...])

Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.

ISRES#

Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.

The evolution strategy is based on a combination of a mutation rule (with a log-normal step-size update and exponential smoothing) and differential variation (a Nelder-Mead-like update rule). The fitness ranking is simply via the objective function for problems without nonlinear constraints, but when nonlinear constraints are included the stochastic ranking proposed by Runarsson and Yao is employed.

NLoptISRES minimizer paths on a complex Rosenbrock function NLoptISRES minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptISRES([tol, population, ...])

Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.

ESCH#

Global minimizer using an evolutionary algorithm.

This is a modified Evolutionary Algorithm for global optimization, developed by Carlos Henrique da Silva Santos’s and described in the following paper and Ph.D thesis.

NLoptESCH minimizer paths on a complex Rosenbrock function NLoptESCH minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptESCH([tol, verbosity, ...])

Global minimizer using an evolutionary algorithm.

ShiftVar#

NLoptShiftVar minimizer paths on a complex Rosenbrock function NLoptShiftVar minimizer paths on a complex Rosenbrock function

zfit.minimize.NLoptShiftVar([tol, maxcor, ...])

Local, gradient-based minimizer using a shifted limited-memory variable-metric.

All minimizers#

zfit.minimize.Minuit([tol, mode, gradient, ...])

Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit.

zfit.minimize.LevenbergMarquardt([tol, ...])

Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.

zfit.minimize.Ipyopt([tol, maxcor, ...])

Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.

zfit.minimize.ScipyBFGS([tol, c1, c2, ...])

Local, gradient based quasi-Newton algorithm using the BFGS algorithm.

zfit.minimize.ScipyLBFGSB([tol, maxcor, ...])

Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.

zfit.minimize.ScipyTrustConstr([tol, ...])

Trust-region based local minimizer.

zfit.minimize.ScipyPowell([tol, verbosity, ...])

Local minimizer using the modified Powell algorithm.

zfit.minimize.ScipySLSQP([tol, gradient, ...])

Local, gradient-based minimizer using tho Sequential Least Squares Programming algorithm.name.

zfit.minimize.ScipyTruncNC([tol, maxcg, ...])

Local, gradient based minimization algorithm using a truncated Newton method.

zfit.minimize.ScipyCOBYLA([tol, verbosity, ...])

UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.

zfit.minimize.ScipyTrustNCG([tol, ...])

PERFORMS POORLY! Local Newton conjugate gradient trust-region algorithm.

zfit.minimize.ScipyDogleg([tol, ...])

This minimizer requires the hessian and gradient to be provided by the loss itself.

zfit.minimize.ScipyTrustKrylov([tol, ...])

PERFORMS POORLY! Local, gradient based (nearly) exact trust-region algorithm using matrix vector products with the hessian.

zfit.minimize.ScipyNewtonCG([tol, gradient, ...])

WARNING! This algorithm seems unstable and may does not perform well!

zfit.minimize.NLoptLBFGS([tol, maxcor, ...])

Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.

zfit.minimize.NLoptTruncNewton([tol, ...])

Local, gradient-based truncated Newton minimizer using an inexact algorithm.

zfit.minimize.NLoptSLSQP([tol, verbosity, ...])

Local gradient based minimizer using a sequential quadratic programming.

zfit.minimize.NLoptMMA([tol, verbosity, ...])

Method-of-moving-asymptotes for gradient-based local minimization.

zfit.minimize.NLoptCCSAQ([tol, verbosity, ...])

MMA-like minimizer with simpler, quadratic local approximations.

zfit.minimize.NLoptSubplex([tol, verbosity, ...])

Local derivative free minimizer which improves on the Nealder-Mead algorithm.

zfit.minimize.NLoptCOBYLA([tol, verbosity, ...])

Derivative free simplex minimizer using a linear approximation with trust region steps.

zfit.minimize.NLoptMLSL([tol, population, ...])

Global minimizer using local optimization by randomly selecting points.

zfit.minimize.NLoptStoGO([tol, randomized, ...])

Global minimizer which divides the space into smaller rectangles and uses a local BFGS variant inside.

zfit.minimize.NLoptBOBYQA([tol, verbosity, ...])

Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.

zfit.minimize.NLoptISRES([tol, population, ...])

Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.

zfit.minimize.NLoptESCH([tol, verbosity, ...])

Global minimizer using an evolutionary algorithm.

zfit.minimize.NLoptShiftVar([tol, maxcor, ...])

Local, gradient-based minimizer using a shifted limited-memory variable-metric.