B 0d-@sdZddlZddlmZmZddlmZmZm Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZmZmZmZdd Zd d Zd d ZddZdS)a Dogleg algorithm with rectangular trust regions for least-squares minimization. The description of the algorithm can be found in [Voglis]_. The algorithm does trust-region iterations, but the shape of trust regions is rectangular as opposed to conventional elliptical. The intersection of a trust region and an initial feasible region is again some rectangle. Thus, on each iteration a bound-constrained quadratic optimization problem is solved. A quadratic problem is solved by well-known dogleg approach, where the function is minimized along piecewise-linear "dogleg" path [NumOpt]_, Chapter 4. If Jacobian is not rank-deficient then the function is decreasing along this path, and optimization amounts to simply following along this path as long as a point stays within the bounds. A constrained Cauchy step (along the anti-gradient) is considered for safety in rank deficient cases, in this situations the convergence might be slow. If during iterations some variable hit the initial bound and the component of anti-gradient points outside the feasible region, then a next dogleg step won't make any progress. At this state such variables satisfy first-order optimality conditions and they are excluded before computing a next dogleg step. Gauss-Newton step can be computed exactly by `numpy.linalg.lstsq` (for dense Jacobian matrices) or by iterative procedure `scipy.sparse.linalg.lsmr` (for dense and sparse matrices, or Jacobian being LinearOperator). The second option allows to solve very large problems (up to couple of millions of residuals on a regular PC), provided the Jacobian matrix is sufficiently sparse. But note that dogbox is not very good for solving problems with large number of constraints, because of variables exclusion-inclusion on each iteration (a required number of function evaluations might be high or accuracy of a solution will be poor), thus its large-scale usage is probably limited to unconstrained problems. References ---------- .. [Voglis] C. Voglis and I. E. Lagaris, "A Rectangular Trust Region Dogleg Approach for Unconstrained and Bound Constrained Nonlinear Optimization", WSEAS International Conference on Applied Mathematics, Corfu, Greece, 2004. .. [NumOpt] J. Nocedal and S. J. Wright, "Numerical optimization, 2nd edition". N)lstsqnorm)LinearOperatoraslinearoperatorlsmr)OptimizeResult) step_size_to_bound in_boundsupdate_tr_radiusevaluate_quadraticbuild_quadratic_1dminimize_quadratic_1d compute_gradcompute_jac_scalecheck_terminationscale_for_robust_loss_functionprint_header_nonlinearprint_iteration_nonlinearcs>j\}}fdd}fdd}t||f||tdS)zCompute LinearOperator to use in LSMR by dogbox algorithm. `active_set` mask is used to excluded active variables from computations of matrix-vector products. cs"|}d|<|S)Nr)Zravelcopymatvec)xx_free)Jop active_setdL/var/www/html/venv/lib/python3.7/site-packages/scipy/optimize/_lsq/dogbox.pyr@s zlsmr_operator..matveccs|}d|<|S)Nr)rmatvec)rr)rrrrrrEszlsmr_operator..rmatvec)rrdtype)shaperfloat)rrrmnrrr)rrrr lsmr_operator8s r%c Csl||}||}t|| }t||}t||}t||} t|| } t||} |||| | | fS)aFind intersection of trust-region bounds and initial bounds. Returns ------- lb_total, ub_total : ndarray with shape of x Lower and upper bounds of the intersection region. orig_l, orig_u : ndarray of bool with shape of x True means that an original bound is taken as a corresponding bound in the intersection region. tr_l, tr_u : ndarray of bool with shape of x True means that a trust-region bound is taken as a corresponding bound in the intersection region. )npmaximumminimumequal) r tr_boundslbubZ lb_centeredZ ub_centeredlb_totalub_totalorig_lorig_utr_ltr_urrrfind_intersectionMs    r3cCst||||\}} } } } } tj|td}t||| r>||dfStt|| || \}}t||d|d |}||}t|||| \}}d||dk| @<d||dk| @<t|dk| @|dk| @B}|||||fS)aFind dogleg step in a rectangular region. Returns ------- step : ndarray, shape (n,) Computed dogleg step. bound_hits : ndarray of int, shape (n,) Each component shows whether a corresponding variable hits the initial bound after the step is taken: * 0 - a variable doesn't hit the bound. * -1 - lower bound is hit. * 1 - upper bound is hit. tr_hit : bool Whether the step hit the boundary of the trust-region. )r Frr)r3r& zeros_likeintr r rany)r newton_stepgabr*r+r,r-r.r/r0r1r2Z bound_hitsZ to_bounds_Z cauchy_stepZ step_diffZ step_sizehitstr_hitrrr dogleg_stepjs   r?c= Cs|}|}d}|}d}| dk rL| |}dt|d}t|||\}}ndt||}t||}t| tov| dk}|rt|\}}n| d| }}t ||tj d}|dkrd}tj |t d}d|t ||<d|t ||<|}t|}| dkr|jd } d}d} d}!d}"|d kr&tx.||dk}#|#}$||$}%|}&d||#<t |tj d}'|'| krpd}|d krt| |||"|!|'|dk s|| krP||$}(||$})||$}*||$}+| d kr|dd|$f},t|,| dd d}-t|,|%|% \}.}/nP| d krTt|}0t|0||#}1t|1|f|d|$ }-|-|+9}-t|0|| \}.}/d}"xD|"dkr|| kr||+}2t|(|-|%|.|/|2|)|*\}3}4}5|d|3||$<| d krt|,|%|3 }6n| d krt|0|| }6t||||}7||7}8|d7}t ||tj d}9tt|8s*d|9}q\| dk rB| |8dd}:ndt|8|8}:||:}"t||"|6|9|5\}};t |}!t|"||!t ||;||}|dk r\Pq\W|"dkrB|4||$<|7}|dk}<||<||<<|dk}<||<||<<|8}|}|:}|||}|d7}| dk r"| |}t|||\}}t||}|rJt||\}}nd}!d}"| d7} q*W|dkrfd}t|||||&|'||||d S)Nrg?rjac)ordg?)r r4dexact)Zrcondrggg?T)Z cost_only) rcostfunr@ZgradZ optimalityZ active_masknfevnjevstatus) rr&sumrdotr isinstancestrrrinfr5r6r)Z empty_likesizerrrr rr%rr?fillr Zclipallisfiniter rr)=rFr@Zx0Zf0ZJ0r+r,ZftolZxtolZgtolZmax_nfevZx_scaleZ loss_functionZ tr_solverZ tr_optionsverbosefZf_truerGJrHrhorEr9Z jac_scalescaleZ scale_invDeltaZon_boundrstepZtermination_status iterationZ step_normZactual_reductionrZfree_setZg_freeZg_fullZg_normrZlb_freeZub_freeZ scale_freeZJ_freer8r:r;rZlsmr_opr*Z step_freeZ on_bound_freer>Zpredicted_reductionZx_newZf_newZ step_h_normZcost_newratiomaskrrrdogboxs                           r])__doc__numpyr&Z numpy.linalgrrZscipy.sparse.linalgrrrZscipy.optimizercommonr r r r r rrrrrrrr%r3r?r]rrrr*s 8+