B ӻdwO@sdZddlmZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd l m Z dd l mZdd lmZdd lmZed Gddde jZGddde jZdS)zAdam optimizer implementation.) def_function)indexed_slices)ops)backend_config) optimizer_v2) array_ops)control_flow_ops)math_ops) state_ops)gen_training_ops) keras_exportzkeras.optimizers.AdamcsfeZdZdZdZdfdd Zd d Zfd d ZfddZdddZ dddZ fddZ Z S)Adama Optimizer that implements the Adam algorithm. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to [Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to 0.001. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to 0.9. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to 0.999. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to 1e-7. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to `"Adam"`. **kwargs: Keyword arguments. Allowed to be one of `"clipnorm"` or `"clipvalue"`. `"clipnorm"` (float) clips gradients by norm; `"clipvalue"` (float) clips gradients by value. Usage: >>> opt = tf.keras.optimizers.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 Reference: - [Kingma et al., 2014](http://arxiv.org/abs/1412.6980) - [Reddi et al., 2018]( https://openreview.net/pdf?id=ryQu7f-RZ) for `amsgrad`. Notes: The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). TMbP??+?Hz>Fc sftt|j|f||d|d||d|j|d||d||pXt|_||_dS)N learning_ratelrdecaybeta_1beta_2) superr __init__ _set_hyperget_initial_decayrepsilonamsgrad)selfrrrrrnamekwargs) __class__[/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/adam.pyris  z Adam.__init__cCsXx|D]}||dqWx|D]}||dq W|jrTx|D]}||dq@WdS)Nmvvhat)add_slotr)rvar_listvarr"r"r# _create_slotsys   zAdam._create_slotsc stt||||t|jd|}t|d|}t|d|}t ||}t ||}|||fdt d|d|} |||f t | t |j|||d|||d|ddS)Nrrlr_t)rrbeta_1_t beta_1_powerone_minus_beta_1_tbeta_2_t beta_2_powerone_minus_beta_2_t)rr _prepare_localr cast iterationsridentity _get_hyperpowsqrtupdatedictr"convert_to_tensor_v2_with_dispatchr) r var_device var_dtype apply_state local_stepr-r0r.r1r)r!r"r#r3s&   zAdam._prepare_localcsR|j}tt|dd}t|d|dkr>|dt|}tt||dS)Nr+)weightsintlenrr set_weights)rrCparamsnum_vars)r!r"r#rFs zAdam.set_weightsNc Cs|j|jj}}|pi||fp,|||}||d}||d}|jstj|j |j |j |d|d|d|d|d|d||j d S||d } tj |j |j |j | j |d|d|d|d|d|d||j d SdS) Nr$r%r.r1r,r-r0r) r)r$r% beta1_power beta2_powerrbeta1beta2rgrad use_lockingr&) r)r$r%r&rIrJrrKrLrrMrN) devicedtype base_dtyper_fallback_apply_stateget_slotrr ZResourceApplyAdamhandle _use_lockingZResourceApplyAdamWithAmsgrad) rrMr)r?r=r> coefficientsr$r%r&r"r"r#_resource_apply_denses@     zAdam._resource_apply_densec Cs|j|jj}}|pi||fp,|||}||d}||d} tj|||d|jd} t | g| ||| } WdQRX||d} |||d} tj| | |d|jd} t | g| | || } WdQRX|j s*t | }tj||d| ||d |jd}tj|| | gS||d }t || }t |gtj|||jd}WdQRXt |}tj||d| ||d |jd}tj|| | |gSdS) Nr$r/r-)rNr%r2r0rrr&)rOrPrQrrRrSr assignrUrZcontrol_dependenciesZ_resource_scatter_addrr r9 assign_subrgroupmaximum)rrMr)indicesr?r=r>rVr$m_scaled_g_valuesZm_tr%v_scaled_g_valuesZv_tZv_sqrtZ var_updatev_hatZv_hat_tZ v_hat_sqrtr"r"r#_resource_apply_sparses@            zAdam._resource_apply_sparsec sBtt|}||d|j|d|d|j|jd|S)Nrrr)rrrrrr)rr get_configr:_serialize_hyperparameterrrr)rconfig)r!r"r#ras zAdam.get_config)rrrrFr )N)N) __name__ __module__ __qualname____doc___HAS_AGGREGATE_GRADrr*r3rFrWr`ra __classcell__r"r")r!r#r sF   % (r cs~eZdZdZdZdfd d Zd d Zfd dZfddZe j dddddZ e j dddddZ fddZ ZS) NonFusedAdama Optimizer that implements the Adam algorithm without fused kernels. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to the paper [Adam: A Method for Stochastic Optimization. Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". For AMSGrad see [On The Convergence Of Adam And Beyond. Reddi et al., 5-8](https://openreview.net/pdf?id=ryQu7f-RZ). **If amsgrad = False**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{v_t} + \epsilon)$$ **If amsgrad = True**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector initialize $\hat{v}_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\hat{v}_t = \max(\hat{v}_{t-1}, v_t)$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{\hat{v}_t} + \epsilon)$$ The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). Usage: >>> opt = tf.keras.optimizers.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 TMbP??+?Hz>Fr c sftt|j|f||d|d||d|j|d||d||pXt|_||_dS)auConstruct a new Adam optimizer. Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to 0.001. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to 0.9. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to 0.999. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to 1e-7. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to "Adam". **kwargs: keyword arguments. Allowed to be {`clipnorm`, `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. rrrrrN) rrjrrrrrrr)rrrrrrrr )r!r"r#rIs$  zNonFusedAdam.__init__cCsXx|D]}||dqWx|D]}||dq W|jrTx|D]}||dq@WdS)Nr$r%r&)r'r)rr(r)r"r"r#r*us   zNonFusedAdam._create_slotsc stt||||t|jd|}t|d|}t|d|}t ||}t ||}|||fdt d|d|} |||f t | t |j|||d|||d|ddS)Nr+rrr,)rrr-r.r/r0r1r2)rrjr3r r4r5rr6r7r8r9r:r;rr<r) rr=r>r?r@r-r0r.r1r)r!r"r#r3s&   zNonFusedAdam._prepare_localcsR|j}tt|dd}t|d|dkr>|dt|}tt||dS)Nr+rArB)rCrDrErrjrF)rrCrGrH)r!r"r#rFs zNonFusedAdam.set_weights)Z jit_compileNc Cs|j|jj}}|pi||fp,|||}||d}||d}|dtd|dd|d} |||d|d|t ||d|d|j r||d } | t | || }| || t||d dS) Nr$r%r,r+r1r.r-r0r&r)rOrPrQrrRrSr r9Z assign_addZsquarerrXr[rY) rrMr)r?r=r>rVr$r%alphar&r"r"r#rWs     z"NonFusedAdam._resource_apply_densec Cs|j|jj}}|pi||fp,|||}||d}||d} |||d|t | |||d} |||d} | | |d| t | ||j s| |d|t | |dnB||d } | t | | | |d|t | |ddS) Nr$r/r-r%r2r0rrr&)rOrPrQrrRrSrXZ scatter_addrZ IndexedSlicesrrYr r9r[) rrMr)r\r?r=r>rVr$r]r%r^r_r"r"r#r`s$     z#NonFusedAdam._resource_apply_sparsec sBtt|}||d|j|d|d|j|jd|S)Nrrr)rrrrrr)rrjrar:rbrrr)rrc)r!r"r#ras zNonFusedAdam.get_config)rkrlrmrnFr )N)N)rdrerfrgrhrr*r3rFrfunctionrWr`rarir"r")r!r#rjs G&     rjN)rgZtensorflow.python.eagerrZtensorflow.python.frameworkrrZtensorflow.python.kerasrZ$tensorflow.python.keras.optimizer_v2rZtensorflow.python.opsrrr r Ztensorflow.python.trainingr Z tensorflow.python.util.tf_exportr Z OptimizerV2r rjr"r"r"r#s            _