Minimizers

zfit supplies wrappers for different minimizers from multiple libraries. Most of the are local minimizers (such as Minuit, Ipyopt or ScipyLBFGSB are) while there are also a few global ones such as the NLoptISRES or NLoptStoGO.

While the former are usually faster and preferred, they depend more on the initial values than the latter. Especially in higher dimensions, a global search of the parameters can increase the minimization time drastically and is often infeasible. It is also possible to couple the minimizers by first doing an approximate global minimization and then polish the minimum found with a local minimizer.

All minimizers support similar arguments, most notably tol which denotes the termination value. This is reached if the value of the convergence criterion, which defaults to EDM, the same that is also used in Minuit.

Other than that, there are a also a few minimizer specific arguments that differ from each minimizer.

They all have the exact same minimization method minimize() which takes a loss, parameters and (optionally) a FitResult from which it can take information to have a better start into the minimization.

The following visualization shows how different minimizers perform on a complex version of the Rosenbrock function. Each minimizer starts from 5 different initial points (marked with symbols) and follows a path to the minimum. The contour plot represents the function values, with darker colors indicating lower values. For each minimizer, the plot shows the number of function evaluations, gradient calculations, and execution time.

Minuit

Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit. The iminuit package is a fast, time-proven minimizer based on the Minuit2 C++ library. It is an especially robust minimizer that finds the local minimum quiet reliably for complicated fits. For large fits with hundreds of parameters, other alternatives are generally more performant. It is however, like all local minimizers, still rather dependent on close enough initial values.

zfit.minimize.Minuit([tol, mode, gradient, ...])

Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit.

Levenberg-Marquardt

Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.

LM minimizes a function by iteratively solving a locally linearized version of the problem. Using the gradient (g) and the Hessian (H) of the loss function, the algorithm determines a step (h) that minimizes the loss function by solving \(Hh = g\). This works perfectly in one step for linear problems, however for non-linear problems it may be unstable far from the minimum. Thus a scalar damping parameter (L) is introduced and the Hessian is modified based on this damping.

zfit.minimize.LevenbergMarquardt([tol, ...])

Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.

Ipyopt

Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.

This implemenation uses the IPyOpt wrapper

Ipopt (Interior Point Optimizer, pronounced “Eye-Pea-Opt”) is an open source software package for large-scale nonlinear optimization. It can be used to solve general nonlinear programming problems It is written in Fortran and C and is released under the EPL (formerly CPL). IPOPT implements a primal-dual interior point method, and uses line searches based on Filter methods (Fletcher and Leyffer).

zfit.minimize.Ipyopt([tol, maxcor, ...])

Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.

Scipy

The following visualizations show how different Scipy minimizers perform on the complex Rosenbrock function.

BFGS

Local, gradient based quasi-Newton algorithm using the BFGS algorithm.

BFGS, named after Broyden, Fletcher, Goldfarb, and Shanno, is a quasi-Newton method that approximates the Hessian matrix of the loss function using the gradients of the loss function. It stores an approximation of the inverse Hessian matrix and updates it at each iteration. For a limited memory version, which doesn’t store the full matrix, see L-BFGS-B.

zfit.minimize.ScipyBFGS([tol, c1, c2, ...])

Local, gradient based quasi-Newton algorithm using the BFGS algorithm.

LBFGSB

Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.

Limited-memory BFGS is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS) using a limited amount of memory (or gradients, controlled by maxcor).

L-BFGS borrows ideas from the trust region methods while keeping the L-BFGS update of the Hessian and line search algorithms.

zfit.minimize.ScipyLBFGSB([tol, maxcor, ...])

Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.

TrustConstr

zfit.minimize.ScipyTrustConstr([tol, ...])

Trust-region based local minimizer.

Powell

Local minimizer using the modified Powell algorithm.

zfit.minimize.ScipyPowell([tol, verbosity, ...])

Local minimizer using the modified Powell algorithm.

SLSQP

zfit.minimize.ScipySLSQP([tol, gradient, ...])

Local, gradient-based minimizer using tho Sequential Least Squares Programming algorithm.name.

TruncNC

zfit.minimize.ScipyTruncNC([tol, maxcg, ...])

Local, gradient based minimization algorithm using a truncated Newton method.

COBYLA

UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.

COBYLA constructs successive linear approximations of the objective function and constraints via a simplex of n+1 points (in n dimensions), and optimizes these approximations in a trust region at each step.

zfit.minimize.ScipyCOBYLA([tol, verbosity, ...])

UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.

TrustNCG

zfit.minimize.ScipyTrustNCG([tol, ...])

PERFORMS POORLY! Local Newton conjugate gradient trust-region algorithm.

Dogleg

This minimizer requires the hessian and gradient to be provided by the loss itself.

zfit.minimize.ScipyDogleg([tol, ...])

This minimizer requires the hessian and gradient to be provided by the loss itself.

ScipyTrustKrylov

zfit.minimize.ScipyTrustKrylov([tol, ...])

PERFORMS POORLY! Local, gradient based (nearly) exact trust-region algorithm using matrix vector products with the hessian.

NewtonCG

zfit.minimize.ScipyNewtonCG([tol, gradient, ...])

WARNING! This algorithm seems unstable and may does not perform well!

NLopt

The following visualizations show how different NLopt minimizers perform on the complex Rosenbrock function:

LBFGS

Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.

This is most probably the most popular algorithm for gradient based local minimum searches and also the underlying algorithm in the Minuit minimizer that is also available as Minuit.

zfit.minimize.NLoptLBFGS([tol, maxcor, ...])

Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.

Truncated Newton

zfit.minimize.NLoptTruncNewton([tol, ...])

Local, gradient-based truncated Newton minimizer using an inexact algorithm.

SLSQP

zfit.minimize.NLoptSLSQP([tol, verbosity, ...])

Local gradient based minimizer using a sequential quadratic programming.

MMA

zfit.minimize.NLoptMMA([tol, verbosity, ...])

Method-of-moving-asymptotes for gradient-based local minimization.

CCSAQ

zfit.minimize.NLoptCCSAQ([tol, verbosity, ...])

MMA-like minimizer with simpler, quadratic local approximations.

Subplex

zfit.minimize.NLoptSubplex([tol, verbosity, ...])

Local derivative free minimizer which improves on the Nealder-Mead algorithm.

COBYLA

Derivative free simplex minimizer using a linear approximation with trust region steps.

COBYLA (Constrained Optimization BY Linear Approximations) constructs successive linear approximations of the objective function and constraints via a simplex of n+1 points (in n dimensions), and optimizes these approximations in a trust region at each step.

zfit.minimize.NLoptCOBYLA([tol, verbosity, ...])

Derivative free simplex minimizer using a linear approximation with trust region steps.

MLSL

Global minimizer using local optimization by randomly selecting points.

“Multi-Level Single-Linkage” (MLSL) is an algorithm for global optimization by a sequence of local optimizations from random starting points. MLSL is distinguished by a “clustering” heuristic that helps it to avoid repeated searches of the same local optima, and has some theoretical guarantees of finding all local optima in a finite number of local minimizations.

zfit.minimize.NLoptMLSL([tol, population, ...])

Global minimizer using local optimization by randomly selecting points.

StoGO

zfit.minimize.NLoptStoGO([tol, randomized, ...])

Global minimizer which divides the space into smaller rectangles and uses a local BFGS variant inside.

BOBYQA

Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.

This is an algorithm derived from the BOBYQA subroutine of M. J. D. Powell, converted to C and modified for the NLopt stopping criteria. BOBYQA performs derivative-free bound-constrained optimization using an iteratively constructed quadratic approximation for the objective function.

zfit.minimize.NLoptBOBYQA([tol, verbosity, ...])

Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.

ISRES

Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.

The evolution strategy is based on a combination of a mutation rule (with a log-normal step-size update and exponential smoothing) and differential variation (a Nelder-Mead-like update rule). The fitness ranking is simply via the objective function for problems without nonlinear constraints, but when nonlinear constraints are included the stochastic ranking proposed by Runarsson and Yao is employed.

zfit.minimize.NLoptISRES([tol, population, ...])

Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.

ESCH

Global minimizer using an evolutionary algorithm.

This is a modified Evolutionary Algorithm for global optimization, developed by Carlos Henrique da Silva Santos’s and described in the following paper and Ph.D thesis.

zfit.minimize.NLoptESCH([tol, verbosity, ...])

Global minimizer using an evolutionary algorithm.

ShiftVar

zfit.minimize.NLoptShiftVar([tol, maxcor, ...])

Local, gradient-based minimizer using a shifted limited-memory variable-metric.

All minimizers

zfit.minimize.Minuit([tol, mode, gradient, ...])	Minuit is a longstanding and well proven algorithm of the BFG class implemented in iminuit.
zfit.minimize.LevenbergMarquardt([tol, ...])	Levenberg-Marquardt minimizer for general non-linear minimization by interpolating between Gauss-Newton and Gradient descent optimization.
zfit.minimize.Ipyopt([tol, maxcor, ...])	Ipopt is a gradient-based minimizer that performs large scale nonlinear optimization of continuous systems.
zfit.minimize.ScipyBFGS([tol, c1, c2, ...])	Local, gradient based quasi-Newton algorithm using the BFGS algorithm.
zfit.minimize.ScipyLBFGSB([tol, maxcor, ...])	Local, gradient based quasi-Newton algorithm using the limited-memory BFGS approximation.
zfit.minimize.ScipyTrustConstr([tol, ...])	Trust-region based local minimizer.
zfit.minimize.ScipyPowell([tol, verbosity, ...])	Local minimizer using the modified Powell algorithm.
zfit.minimize.ScipySLSQP([tol, gradient, ...])	Local, gradient-based minimizer using tho Sequential Least Squares Programming algorithm.name.
zfit.minimize.ScipyTruncNC([tol, maxcg, ...])	Local, gradient based minimization algorithm using a truncated Newton method.
zfit.minimize.ScipyCOBYLA([tol, verbosity, ...])	UNSTABLE! Local gradient-free dowhhill simplex-like method with an implicit linear approximation.
zfit.minimize.ScipyTrustNCG([tol, ...])	PERFORMS POORLY! Local Newton conjugate gradient trust-region algorithm.
zfit.minimize.ScipyDogleg([tol, ...])	This minimizer requires the hessian and gradient to be provided by the loss itself.
zfit.minimize.ScipyTrustKrylov([tol, ...])	PERFORMS POORLY! Local, gradient based (nearly) exact trust-region algorithm using matrix vector products with the hessian.
zfit.minimize.ScipyNewtonCG([tol, gradient, ...])	WARNING! This algorithm seems unstable and may does not perform well!
zfit.minimize.NLoptLBFGS([tol, maxcor, ...])	Local, gradient-based quasi-Newton minimizer using the low storage BFGS Hessian approximation.
zfit.minimize.NLoptTruncNewton([tol, ...])	Local, gradient-based truncated Newton minimizer using an inexact algorithm.
zfit.minimize.NLoptSLSQP([tol, verbosity, ...])	Local gradient based minimizer using a sequential quadratic programming.
zfit.minimize.NLoptMMA([tol, verbosity, ...])	Method-of-moving-asymptotes for gradient-based local minimization.
zfit.minimize.NLoptCCSAQ([tol, verbosity, ...])	MMA-like minimizer with simpler, quadratic local approximations.
zfit.minimize.NLoptSubplex([tol, verbosity, ...])	Local derivative free minimizer which improves on the Nealder-Mead algorithm.
zfit.minimize.NLoptCOBYLA([tol, verbosity, ...])	Derivative free simplex minimizer using a linear approximation with trust region steps.
zfit.minimize.NLoptMLSL([tol, population, ...])	Global minimizer using local optimization by randomly selecting points.
zfit.minimize.NLoptStoGO([tol, randomized, ...])	Global minimizer which divides the space into smaller rectangles and uses a local BFGS variant inside.
zfit.minimize.NLoptBOBYQA([tol, verbosity, ...])	Derivative-free local minimizer that iteratively constructed quadratic approximation for the loss.
zfit.minimize.NLoptISRES([tol, population, ...])	Improved Stochastic Ranking Evolution Strategy using a mutation rule and differential variation.
zfit.minimize.NLoptESCH([tol, verbosity, ...])	Global minimizer using an evolutionary algorithm.
zfit.minimize.NLoptShiftVar([tol, maxcor, ...])	Local, gradient-based minimizer using a shifted limited-memory variable-metric.