B d(@sldZddlmmZddlmZddlmZddl m Z ddl m Z Gddde j Zd d Zd d ZdS) zBase class for attention layers that can be used in sequence DNN/CNN models. This file follows the terminology of https://arxiv.org/abs/1706.03762 Figure 2. Attention is formed by three tensors: Query, Key and Value. N)logging)backend) base_layer)control_flow_utilcspeZdZdZdfdd ZfddZddZdd d Zdd dZdddZ ddZ ddZ fddZ Z S)BaseDenseAttentionaXBase Attention class for Dense networks. This class is suitable for Dense or CNN networks, and not for RNN networks. Implementations of attention mechanisms should inherit from this class, and reuse the `apply_attention_scores()` method. Args: dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Call Args: inputs: List of the following tensors: * query: Query `Tensor` of shape `[batch_size, Tq, dim]`. * value: Value `Tensor` of shape `[batch_size, Tv, dim]`. * key: Optional key `Tensor` of shape `[batch_size, Tv, dim]`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: * query_mask: A boolean mask `Tensor` of shape `[batch_size, Tq]`. If given, the output will be zero at the positions where `mask==False`. * value_mask: A boolean mask `Tensor` of shape `[batch_size, Tv]`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). return_attention_scores: bool, if `True`, returns the attention scores (after masking and softmax) as an additional output argument. Output: Attention outputs of shape `[batch_size, Tq, dim]`. [Optional] Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. c s>d|krtd|dd|_tjf|||_d|_dS)Ncausalzi`causal` argument is deprecated. Please use `use_causal_mask` in call() method to specify causal masking.FT)rwarningpoprsuper__init__dropoutZsupports_masking)selfr kwargs) __class__]/var/www/html/venv/lib/python3.7/site-packages/keras/layers/attention/base_dense_attention.pyr BszBaseDenseAttention.__init__cs |jdkrt|d|_dS)NrT)r r buildZbuilt)r input_shape)rrrrOs  zBaseDenseAttention.buildcCstS)zCalculates attention scores. Args: query: Query tensor of shape `[batch_size, Tq, dim]`. key: Key tensor of shape `[batch_size, Tv, dim]`. Returns: Tensor of shape `[batch_size, Tq, Tv]`. )NotImplementedError)rquerykeyrrr_calculate_scoresVs z$BaseDenseAttention._calculate_scoresNcs|dk rPt|}|jtjkr8|dtj||jd8}n|dtj||jd8}|dkr`t}tj|j dkrfdd}t ||fdd t |fS) aMApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `[batch_size, Tq]` and `key` tensor of shape `[batch_size, Tv]` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `[batch_size, Tq, Tv]`. value: Value tensor of shape `[batch_size, Tv, dim]`. scores_mask: A boolean mask `Tensor` of shape `[batch_size, 1, Tv]` or `[batch_size, Tq, Tv]`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `[batch_size, Tq, dim]`. Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. Ng@)dtypegeArcsjjjdS)N)Zrate)Z_random_generatorr r)rweightsrrdropped_weightssz9BaseDenseAttention._apply_scores..dropped_weightscs tS)N)tfidentityr)rrrz2BaseDenseAttention._apply_scores..) rZ logical_notrZfloat16castrZlearning_phasennZsoftmaxr rZ smart_condmatmul)rscoresvalue scores_masktrainingZ padding_maskrr)rrr _apply_scoresbs    z BaseDenseAttention._apply_scoresFcCs"|j||d|d}|d}t|dkr2|dn|}|rB|dnd} |rR|dnd} |j||d} | dk rztj| dd} |js|rt| } tjt| dd| ddgdd} t | }nd}t | |}|j | |||d\}}| dk rtj| d d} |tj | |j d 9}|r||fS|S) N)inputsmaskr)rr)axis)r#r$r%r&)r)_validate_call_argslenrrZ expand_dimsrshapeconcatZ ones_like_lower_triangular_mask _merge_masksr'r r)rr(r)r&Zreturn_attention_scoresZuse_causal_maskqvkq_maskZv_maskr#Z scores_shapeZcausal_mask_shapeZ causal_maskr%resultZattention_scoresrrrcalls0  $   zBaseDenseAttention.callcCs4|j||d|r0|d}|dkr&dSt|SdS)N)r(r)r)r/rZconvert_to_tensor)rr(r)r8rrr compute_masks zBaseDenseAttention.compute_maskcCst|dS)Nr)rZ TensorShape)rrrrrcompute_output_shapesz'BaseDenseAttention.compute_output_shapecCs|jj}t|ts&t|d|dt|dks>t|dkrVt|dt|d|rt|tsxt|d|dt|dkst|t|krt|dt|ddS) z'Validates arguments of the call method.zc layer must be called on a list of inputs, namely [query, value] or [query, value, key]. Received: .r+zl layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Received length: zG layer mask must be a list, namely [query_mask, value_mask]. Received: zZ layer mask must be a list of length 2, namely [query_mask, value_mask]. Received length: N)r__name__ isinstancelist ValueErrorr0)rr(r) class_namerrrr/s  z&BaseDenseAttention._validate_call_argscs0d|ji}t}tt|t|S)Nr )r r get_configdictrAitems)rconfigZ base_config)rrrrDs  zBaseDenseAttention.get_config)r)NN)NNFF)N)r? __module__ __qualname____doc__r rrr'r:r;r<r/rD __classcell__rr)rrrs#  8 $ rcCs@tjtj|tjddd}tjtj|tjddd}t||S)zCCreates a lower-triangular boolean mask over the last 2 dimensions.)r1rr,)r-r.)rZcumsumZonesZint32Z greater_equal)r1Z row_indexZ col_indexrrrr3sr3cCs$|dkr |S|dkr|St||S)N)r logical_and)xyrrrr4s r4)rJZtensorflow.compat.v2compatZv2rZabslrZkerasrZ keras.enginerZ keras.utilsrZBaseRandomLayerrr3r4rrrrs    W