B ӻdt@s0dZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd l mZdd lmZdd lmZGd d d eZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeejZeZeZ eZ!eZ"eZ#eZ$eZ%dS)znLegacy v1 optimizer classes. For more examples see the base class `tf.compat.v1.keras.optimizers.Optimizer`. )distribution_strategy_context)backprop)ops)backend)clip_ops)math_ops) state_ops)base) training_util)nestc@sXeZdZdZddZdZddZddZd d Zd d Z d dZ ddZ e ddZ dS) OptimizeraAbstract optimizer base class. Note: this is the parent class of all optimizers, not an actual optimizer that can be used for training models. All Keras optimizers support the following keyword arguments: clipnorm: float >= 0. Gradients will be clipped when their L2 norm exceeds this value. clipvalue: float >= 0. Gradients will be clipped when their absolute value exceeds this value. cKsjddh}xD|D]<}||kr*tdt|||dkrtd|||qW|j|g|_g|_dS)Nclipnorm clipvaluez1Unexpected keyword argument passed to optimizer: rzExpected {} >= 0, received: {}) TypeErrorstr ValueErrorformat__dict__updateupdatesweights)selfkwargsZallowed_kwargskrV/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v1.py__init__0s    zOptimizer.__init__FcCstdS)zCreates and sets all optimizer weights. Args: params: list or tuple of `Variable` objects that will be minimized using this optimizer. Returns: Specific weight values that are used in `get_updates` N)NotImplementedError)rparamsrrr_create_all_weightsAs zOptimizer._create_all_weightscCstdS)N)r)rlossrrrr get_updatesMszOptimizer.get_updatescsbt||}tdd|Dr&tdtdrBfdd|D}tdr^fdd|D}|S) a3Returns gradients of `loss` with respect to `params`. Args: loss: Loss tensor. params: List of variables. Returns: List of gradient tensors. Raises: ValueError: In case any gradient cannot be computed (e.g. if gradient function not implemented). css|]}|dkVqdS)Nr).0grrr _sz*Optimizer.get_gradients..zAn operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: backend.argmax, backend.round, backend.eval.r csg|]}t|jqSr)rZ clip_by_normr )r"r#)rrr fsz+Optimizer.get_gradients..rcs g|]}t|j jqSr)rZ clip_by_valuer)r"r#)rrrr%is)rZ gradientsanyrhasattr)rr rgradsr)rr get_gradientsPs    zOptimizer.get_gradientscCs|j}t|t|kr>tdtt|dtt|dg}t|}xTt|||D]D\}}}|j|jkrtdt|jdt|j|||fqZWt |dS)aSets the weights of the optimizer, from Numpy arrays. Should only be called after computing the gradients (otherwise the optimizer has no weights). Args: weights: a list of Numpy arrays. The number of arrays and their shape must match number of the dimensions of the weights of the optimizer (i.e. it should match the output of `get_weights`). Raises: ValueError: in case of incompatible weight shapes. z%Length of the specified weight list (z9) does not match the number of weights of the optimizer ()zOptimizer weight shape z+ not compatible with provided weight shape N) rlenrrrbatch_get_valuezipshapeappendZbatch_set_value)rrrZweight_value_tuplesZ param_valuespvpwrrr set_weightsns&  zOptimizer.set_weightscCs t|jS)zmReturns the current value of the weights of the optimizer. Returns: A list of numpy arrays. )rr,r)rrrr get_weightsszOptimizer.get_weightscCs0i}t|dr|j|d<t|dr,|j|d<|S)Nr r)r'r r)rconfigrrr get_configs     zOptimizer.get_configcCs |f|S)Nr)clsr5rrr from_configszOptimizer.from_configN)__name__ __module__ __qualname____doc__rZ_HAS_AGGREGATE_GRADrr!r)r3r4r6 classmethodr8rrrrr "s  r cs>eZdZdZd fdd ZddZd d Zfd d ZZS)SGDaStochastic gradient descent optimizer. Includes support for momentum, learning rate decay, and Nesterov momentum. Args: lr: float >= 0. Learning rate. momentum: float >= 0. Parameter that accelerates SGD in the relevant direction and dampens oscillations. decay: float >= 0. Learning rate decay over each update. nesterov: boolean. Whether to apply Nesterov momentum. {Gz?Fc s~tt|jf|t|jjHtjdddd|_tj|dd|_ tj|dd|_ tj|dd|_ WdQRX||_ ||_ dS) Nrint64 iterations)dtypenamelr)rDmomentumdecay)superr>rr name_scope __class__r9variablerBrErFrG initial_decaynesterov)rrErFrGrMr)rJrrrsz SGD.__init__cCs.dd|D}dd|D}|jg||_|S)NcSsg|]}t|qSr)r int_shape)r"r1rrrr%sz+SGD._create_all_weights..cSsg|]}t|qSr)rzeros)r"r.rrrr%s)rBr)rrshapesmomentsrrrrszSGD._create_all_weightsc Cs|||}t|jdg|_|j}|jdkrV|dd|jt |jt |j}| |}xt |||D]\}}}|j|||} |jt|| |jr||j| ||} n|| } t|dddk r|| } |jt|| qnW|jS)Nrg? constraint)r)r assign_addrBrrErLrGrcastrrCrr-rFr/assignrMgetattrrS) rr rr(rErQr1r#mvnew_prrrr!s(     zSGD.get_updatescs^tt|jtt|jtt|j|jd}tt| }t t | t | S)N)rErFrGrM) floatr get_valuerErFrGrMrHr>r6dictlistitems)rr5 base_config)rJrrr6s  zSGD.get_config)r?r@r@F) r9r:r;r<rrr!r6 __classcell__rr)rJrr>s   r>cs>eZdZdZdfdd Zdd Zd d Zfd d ZZS)RMSpropa}RMSProp optimizer. It is recommended to leave the parameters of this optimizer at their default values (except the learning rate, which can be freely tuned). Args: lr: float >= 0. Learning rate. rho: float >= 0. epsilon: float >= 0. Fuzz factor. If `None`, defaults to `backend.epsilon()`. decay: float >= 0. Learning rate decay over each update. MbP??Nc stt|jf|t|jjHtj|dd|_tj|dd|_ tj|dd|_ tjdddd|_ WdQRX|dkr~t }||_ ||_ dS) NrE)rDrhorGrrArB)rCrD)rHrbrrrIrJr9rKrErfrGrBepsilonrL)rrErfrgrGr)rJrrrszRMSprop.__init__cCsdd|D}||_|S)NcSs&g|]}tjt|t|dqS))rC)rrOrNrC)r"r1rrrr%sz/RMSprop._create_all_weights..)r)rr accumulatorsrrrrszRMSprop._create_all_weightsc Cs|||}||}t|jdg|_|j}|jdkr`|dd|jt |jt |j}xt |||D]\}}}|j|d|jt |} |jt|| |||t | |j} t|dddk r|| } |jt|| qnW|jS)NrRrg?rS)r)rrrTrBrrErLrGrrUrrCr-rfsquarer/rVsqrtrgrWrS) rr rr(rhrEr1r#anew_arZrrrr! s$     zRMSprop.get_updatescs^tt|jtt|jtt|j|jd}tt| }t t | t | S)N)rErfrGrg) r[rr\rErfrGrgrHrbr6r]r^r_)rr5r`)rJrrr6#s  zRMSprop.get_config)rcrdNre) r9r:r;r<rrr!r6rarr)rJrrbs   rbcs>eZdZdZd fdd ZddZd d Zfd d ZZS)AdagradaAdagrad optimizer. Adagrad is an optimizer with parameter-specific learning rates, which are adapted relative to how frequently a parameter gets updated during training. The more updates a parameter receives, the smaller the updates. It is recommended to leave the parameters of this optimizer at their default values. # Arguments lr: float >= 0. Initial learning rate. epsilon: float >= 0. If `None`, defaults to `backend.epsilon()`. decay: float >= 0. Learning rate decay over each update. # References - [Adaptive Subgradient Methods for Online Learning and Stochastic Optimization](http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf) {Gz?Nc s~tt|jf|t|jj8tj|dd|_tj|dd|_ tjdddd|_ WdQRX|dkrnt }||_ ||_ dS)NrE)rDrGrrArB)rCrD) rHrmrrrIrJr9rKrErGrBrgrL)rrErgrGr)rJrrrCszAdagrad.__init__cCs&dd|D}dd|D}||_|S)NcSsg|]}t|qSr)rrN)r"r1rrrr%Osz/Adagrad._create_all_weights..cSsg|]}t|qSr)rrO)r"r.rrrr%Ps)r)rrrPrhrrrrNszAdagrad._create_all_weightsc Cs|||}||}t|jdg|_|j}|jdkr`|dd|jt |jt |j}xt |||D]v\}}}|t |} |jt|| |||t | |j} t|dddk r|| } |jt|| qnW|jS)NrRrg?rS)r)rrrTrBrrErLrGrrUrrCr-rir/rVrjrgrWrS) rr rr(rhrEr1r#rkrlrZrrrr!Ts$     zAdagrad.get_updatescsPtt|jtt|j|jd}tt|}t t | t | S)N)rErGrg) r[rr\rErGrgrHrmr6r]r^r_)rr5r`)rJrrr6ns  zAdagrad.get_config)rnNro) r9r:r;r<rrr!r6rarr)rJrrm.s  rmcs>eZdZdZdfdd Zdd Zd d Zfd d ZZS)Adadeltaa0Adadelta optimizer. Adadelta is a more robust extension of Adagrad that adapts learning rates based on a moving window of gradient updates, instead of accumulating all past gradients. This way, Adadelta continues learning even when many updates have been done. Compared to Adagrad, in the original version of Adadelta you don't have to set an initial learning rate. In this version, initial learning rate and decay factor can be set, as in most other Keras optimizers. It is recommended to leave the parameters of this optimizer at their default values. Arguments: lr: float >= 0. Initial learning rate, defaults to 1. It is recommended to leave it at the default value. rho: float >= 0. Adadelta decay factor, corresponding to fraction of gradient to keep at each time step. epsilon: float >= 0. Fuzz factor. If `None`, defaults to `backend.epsilon()`. decay: float >= 0. Initial learning rate decay. References: - [Adadelta - an adaptive learning rate method](http://arxiv.org/abs/1212.5701) ?ffffff?Nc stt|jf|t|jj8tj|dd|_tj|dd|_ tjdddd|_ WdQRX|dkrnt }||_ ||_ ||_ dS)NrE)rDrGrrArB)rCrD)rHrprrrIrJr9rKrErGrBrgrfrL)rrErfrgrGr)rJrrrszAdadelta.__init__cCs<dd|D}dd|D}dd|D}|||_||fS)NcSsg|]}t|qSr)rrN)r"r1rrrr%sz0Adadelta._create_all_weights..cSsg|]}t|qSr)rrO)r"r.rrrr%scSsg|]}t|qSr)rrO)r"r.rrrr%s)r)rrrPrhdelta_accumulatorsrrrrs  zAdadelta._create_all_weightsc CsL|||}t|jdg|_||\}}|j}|jdkrd|dd|jt |jt |j}xt ||||D]\}}} } |j| d|jt |} |jt| | |t | |jt | |j} ||| } t|dddk r|| } |jt|| |j| d|jt | }|jt| |qtW|jS)NrRrg?rS)r)rrTrBrrrErLrGrrUrrCr-rfrir/rVrjrgrWrS)rr rr(rhrtrEr1r#rkZd_arlrrZZnew_d_arrrr!s,     zAdadelta.get_updatescsTtt|j|jtt|j|jd}tt| }t t | t | S)N)rErfrGrg) r[rr\rErfrGrgrHrpr6r]r^r_)rr5r`)rJrrr6s  zAdadelta.get_config)rqrrNrs) r9r:r;r<rrr!r6rarr)rJrrpxs  "rpcs>eZdZdZdfdd Zd d Zd d ZfddZZS)AdamaAdam optimizer. Default parameters follow those provided in the original paper. Args: lr: float >= 0. Learning rate. beta_1: float, 0 < beta < 1. Generally close to 1. beta_2: float, 0 < beta < 1. Generally close to 1. epsilon: float >= 0. Fuzz factor. If `None`, defaults to `backend.epsilon()`. decay: float >= 0. Learning rate decay over each update. amsgrad: boolean. Whether to apply the AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and Beyond". MbP??+?NFc stt|jf|t|jjXtjdddd|_tj|dd|_ tj|dd|_ tj|dd|_ tj|d d|_ WdQRX|dkrt }||_ ||_||_dS) NrrArB)rCrDrE)rDbeta_1beta_2rG)rHrurrrIrJr9rKrBrErzr{rGrgrLamsgrad)rrErzr{rgrGr|r)rJrrrsz Adam.__init__cCs`dd|D}dd|D}|jr2dd|D}ndd|D}|jg||||_|||fS)NcSs&g|]}tjt|t|dqS))rC)rrOrNrC)r"r1rrrr%sz,Adam._create_all_weights..cSs&g|]}tjt|t|dqS))rC)rrOrNrC)r"r1rrrr%scSs&g|]}tjt|t|dqS))rC)rrOrNrC)r"r1rrrr%scSsg|]}tdqS)rR)rrO)r"_rrrr%s)r|rBr)rrmsvsvhatsrrrrs zAdam._create_all_weightsc Cs|||}g|_|j}|jdkrJ|dd|jt|jt |j}t t |jdgt|jt}WdQRX|tdt|j|dt|j|}||\}}} xt||||| D]\} } } } }|j| d|j| }|j| d|jt| }|jr\t||}| ||t||j}|jt ||n| ||t||j}|jt | ||jt | ||}t| dddk r| |}|jt | |qW|jS)Nrg?rRrS)r)rrErLrGrrUrBrrCrcontrol_dependenciesrrTfloatxrjpowr{rzrr-rir|maximumrgr/rVrWrS)rr rr(rEtlr_tr~rrr1r#rXrYZvhatm_tv_tZvhat_tp_trZrrrr! s<   $  zAdam.get_updatescsptt|jtt|jtt|jtt|j|j|jd}t t | }t t |t |S)N)rErzr{rGrgr|)r[rr\rErzr{rGrgr|rHrur6r]r^r_)rr5r`)rJrrr61s zAdam.get_config)rvrwrxNryF) r9r:r;r<rrr!r6rarr)rJrrus(rucs>eZdZdZdfdd Zd d Zd d Zfd dZZS)AdamaxaAdamax optimizer from Adam paper's Section 7. It is a variant of Adam based on the infinity norm. Default parameters follow those provided in the paper. Args: lr: float >= 0. Learning rate. beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1. epsilon: float >= 0. Fuzz factor. If `None`, defaults to `backend.epsilon()`. decay: float >= 0. Learning rate decay over each update. Mb`??+?Nc stt|jf|t|jjXtjdddd|_tj|dd|_ tj|dd|_ tj|dd|_ tj|d d|_ WdQRX|dkrt }||_ ||_dS) NrrArB)rCrDrE)rDrzr{rG)rHrrrrIrJr9rKrBrErzr{rGrgrL)rrErzr{rgrGr)rJrrrLszAdamax.__init__cCsDdd|D}dd|D}dd|D}|jg|||_||fS)NcSsg|]}t|qSr)rrN)r"r1rrrr%asz.Adamax._create_all_weights..cSsg|]}t|qSr)rrO)r"r.rrrr%cscSsg|]}t|qSr)rrO)r"r.rrrr%es)rBr)rrrPr~usrrrr_s zAdamax._create_all_weightsc Csj|||}g|_|j}|jdkrJ|dd|jt|jt |j}t t |jdgt|jt}WdQRX|dt|j|}||\}}xt||||D]\} } } } |j| d|j| } t|j| t| }| || ||j}|jt | | |jt | ||}t| dddk rL| |}|jt | |qW|jS)Nrg?rRrS)r)rrErLrGrrUrBrrCrrrrTrrrzrr-rr{absrgr/rVrWrS)rr rr(rErrr~rr1r#rXurZu_trrZrrrr!is0    zAdamax.get_updatescsltt|jtt|jtt|jtt|j|jd}tt | }t t | t | S)N)rErzr{rGrg)r[rr\rErzr{rGrgrHrr6r]r^r_)rr5r`)rJrrr6s zAdamax.get_config)rrrNr) r9r:r;r<rrr!r6rarr)rJrr>s  #rcs>eZdZdZdfdd Zd d Zd d Zfd dZZS)NadamaNesterov Adam optimizer. Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum. Default parameters follow those provided in the paper. It is recommended to leave the parameters of this optimizer at their default values. Args: lr: float >= 0. Learning rate. beta_1/beta_2: floats, 0 < beta < 1. Generally close to 1. epsilon: float >= 0. Fuzz factor. If `None`, defaults to `backend.epsilon()`. Mb`??+?NMbp?c stt|jf|t|jjXtjdddd|_tjddd|_ tj|dd|_ tj|d d|_ tj|d d|_ WdQRX|dkrt }||_ ||_dS) NrrArB)rCrDg? m_schedule)rDrErzr{)rHrrrrIrJr9rKrBrrErzr{rgschedule_decay)rrErzr{rgrr)rJrrrszNadam.__init__cCsHdd|D}dd|D}dd|D}|j|jg|||_||fS)NcSsg|]}t|qSr)rrN)r"r1rrrr%sz-Nadam._create_all_weights..cSsg|]}t|qSr)rrO)r"r.rrrr%scSsg|]}t|qSr)rrO)r"r.rrrr%s)rBrr)rrrPr~rrrrrs zNadam._create_all_weightsc Cs|||}g|_tt|jdgt|jt }WdQRX|j ddt t d||j}|j ddt t d|d|j}|j|}|j||}|j|j|f||\} } xt||| | D]\} } } }| d|}|j | d|j | }|d|}|j|d|jt| }|dt |j|}d||||}|jt| ||jt||| |j|t ||j}|}t| dddk r| |}|jt| |qW|jS)NrRg?g?gQ?rS)r)rrrrrTrBrrUrrrzrZcast_to_floatxrrr/rr-r{rirVrErjrgrWrS)rr rr(rZmomentum_cache_tZmomentum_cache_t_1Zm_schedule_newZm_schedule_nextr~rr1r#rXrYZg_primerZ m_t_primerZ v_t_primeZm_t_barrrZrrrr!s>      zNadam.get_updatescsbtt|jtt|jtt|j|j|jd}tt | }t t | t | S)N)rErzr{rgr)r[rr\rErzr{rgrrHrr6r]r^r_)rr5r`)rJrrr6s zNadam.get_config)rrrNr) r9r:r;r<rrr!r6rarr)rJrrs,rc@s`eZdZdZdddZddZdddZd d Zd d Zd dZ e ddZ ddZ ddZ dS) TFOptimizerz/Wrapper class for native TensorFlow optimizers.Nc Csd||_|j|dd|dkrJt|jjtjdddd|_WdQRXn||_|j|jdddS)N optimizer)rDrrArB)rCrD global_step)rZ_track_trackablerrIrJr9rKrB)rrrBrrrrszTFOptimizer.__init__cCs|S)zBClip gradients according to the clipnorm and clipvalue attributes.r)rr(rrr_clip_gradients szTFOptimizer._clip_gradientsc Cst|s|dkrtd|dk r$|nt}t|rj|,t|sL|||}t|r`|}WdQRXt|}|r||||}tt ||}| |dS)z&Mimics the `OptimizerV2.minimize` API.Nz2`tape` is required when a `Tensor` loss is passed.) callablerrZ GradientTapewatchr flattenZgradientr^r-apply_gradients)rr Zvar_listZ grad_lossZtaper(grads_and_varsrrrminimizes  zTFOptimizer.minimizecCs|jj||jddS)N)r)rrrB)rrrrrr"szTFOptimizer.apply_gradientscCs|j||S)N)rcompute_gradients)rr rrrr get_grads%szTFOptimizer.get_gradscCstrFg|_|s |j|}n|j||}t}|j||}nB|sbt |j dg|_|jSg|_|j||}|jj||j d}|j ||jS)NrR)r) rZ has_strategyrrrr Zget_global_steprrrTrBr/)rr rr(rZ opt_updaterrrr!(s   zTFOptimizer.get_updatescCstdS)N)r)rrrrrDszTFOptimizer.weightscCstdS)N)r)rrrrr6HszTFOptimizer.get_configcCstdS)N)r)rr5rrrr8KszTFOptimizer.from_config)N)NN)r9r:r;r<rrrrrr!propertyrr6r8rrrrrs   rN)&r<Ztensorflow.python.distributerZtensorflow.python.eagerrZtensorflow.python.frameworkrZtensorflow.python.kerasrZtensorflow.python.opsrrrZtensorflow.python.trackabler Z trackableZtensorflow.python.trainingr Ztensorflow.python.utilr objectr r>rbrmrprurrZ TrackablerZsgdZrmspropZadagradZadadeltaZadamZadamaxZnadamrrrrs4          FGJ\jZdU