B ӻdQ@sdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l m Z dd l mZdd l mZdd lmZGd ddeZedGdddeZedGdddeZddZddZdS)zAttention layers that can be used in sequence DNN/CNN models. This file follows the terminology of https://arxiv.org/abs/1706.03762 Figure 2. Attention is formed by three tensors: Query, Key and Value. )dtypes)ops) tensor_shape)backend)Layer)control_flow_util) array_ops)init_ops)math_ops)nn) keras_exportcs\eZdZdZdfdd ZddZdd d Zdd d Zdd dZddZ fddZ Z S)BaseDenseAttentionaBase Attention class for Dense networks. This class is suitable for Dense or CNN networks, and not for RNN networks. Implementations of attention mechanisms should inherit from this class, and reuse the `apply_attention_scores()` method. Args: causal: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Call Args: inputs: List of the following tensors: * query: Query `Tensor` of shape `[batch_size, Tq, dim]`. * value: Value `Tensor` of shape `[batch_size, Tv, dim]`. * key: Optional key `Tensor` of shape `[batch_size, Tv, dim]`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: * query_mask: A boolean mask `Tensor` of shape `[batch_size, Tq]`. If given, the output will be zero at the positions where `mask==False`. * value_mask: A boolean mask `Tensor` of shape `[batch_size, Tv]`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. Output: Attention outputs of shape `[batch_size, Tq, dim]`. [Optional] Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. Fc s(tt|jf|||_||_d|_dS)NT)superr __init__causaldropoutZsupports_masking)selfrrkwargs) __class__`/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/layers/dense_attention.pyrLszBaseDenseAttention.__init__cCstS)zCalculates attention scores. Args: query: Query tensor of shape `[batch_size, Tq, dim]`. key: Key tensor of shape `[batch_size, Tv, dim]`. Returns: Tensor of shape `[batch_size, Tq, Tv]`. )NotImplementedError)rquerykeyrrr_calculate_scoresSs z$BaseDenseAttention._calculate_scoresNcs|dk rPt|}|jtjkr8|dtj||jd8}n|dtj||jd8}|dkr`t}t |fdd}t ||fddt |fS) aApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `[batch_size, Tq]` and `key` tensor of shape `[batch_size, Tv]` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `[batch_size, Tq, Tv]`. value: Value tensor of shape `[batch_size, Tv, dim]`. scores_mask: A boolean mask `Tensor` of shape `[batch_size, 1, Tv]` or `[batch_size, Tq, Tv]`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `[batch_size, Tq, dim]`. Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. Ng@)dtypegeAcstjjdS)N)Zrate)r rr)rweightsrrdropped_weightssz9BaseDenseAttention._apply_scores..dropped_weightscs tS)N)ridentityr)rrrz2BaseDenseAttention._apply_scores..) r Z logical_notrrZfloat16castrZlearning_phaser ZsoftmaxrZ smart_condmatmul)rscoresvalue scores_masktrainingZ padding_maskrr)rrr _apply_scores_s   z BaseDenseAttention._apply_scorescCs|j||d|d}|d}t|dkr2|dn|}|rB|dnd}|rR|dnd} |j||d} | dk rztj| dd} |jrt| } tjt| dd| ddgdd} t | } nd} t | | }|j | |||d\}}|dk r tj|d d}|t j ||jd 9}|r||fS|S) N)inputsmaskr)rr)axis)r$r%r&r')r)_validate_call_argslenrr expand_dimsrshapeconcatZ ones_like_lower_triangular_mask _merge_masksr(r r"r)rr)r*r'Zreturn_attention_scoresqvkq_maskZv_maskr$Z scores_shapeZcausal_mask_shapeZ causal_maskr&resultZattention_scoresrrrcalls2    zBaseDenseAttention.callcCs4|j||d|r0|d}|dkr&dSt|SdS)N)r)r*r)r0rZ"convert_to_tensor_v2_with_dispatch)rr)r*r:rrr compute_masks zBaseDenseAttention.compute_maskcCs|jj}t|ts td|t|dks8t|dkrLtd|t||rt|tshtd|t|dkst|t|krtd|t|dS)z'Validates arguments of the call method.zZ{} layer must be called on a list of inputs, namely [query, value] or [query, value, key].r,zm{} layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Given length: {}z>{} layer mask must be a list, namely [query_mask, value_mask].z[{} layer mask must be a list of length 2, namely [query_mask, value_mask]. Given length: {}N)r__name__ isinstancelist ValueErrorformatr1)rr)r* class_namerrrr0s$  z&BaseDenseAttention._validate_call_argscs8|j|jd}tt|}tt|t|S)N)rr)rrrr get_configdictrAitems)rconfig base_config)rrrrEs zBaseDenseAttention.get_config)Fr)NN)NNF)N) r? __module__ __qualname____doc__rrr(r<r=r0rE __classcell__rr)rrr "s( 1 " r zkeras.layers.AttentioncsBeZdZdZd fdd ZfddZddZfd d ZZS) AttentionaDot-product attention layer, a.k.a. Luong-style attention. Inputs are `query` tensor of shape `[batch_size, Tq, dim]`, `value` tensor of shape `[batch_size, Tv, dim]` and `key` tensor of shape `[batch_size, Tv, dim]`. The calculation follows the steps: 1. Calculate scores with shape `[batch_size, Tq, Tv]` as a `query`-`key` dot product: `scores = tf.matmul(query, key, transpose_b=True)`. 2. Use scores to calculate a distribution with shape `[batch_size, Tq, Tv]`: `distribution = tf.nn.softmax(scores)`. 3. Use `distribution` to create a linear combination of `value` with shape `[batch_size, Tq, dim]`: `return tf.matmul(distribution, value)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. causal: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Call Args: inputs: List of the following tensors: * query: Query `Tensor` of shape `[batch_size, Tq, dim]`. * value: Value `Tensor` of shape `[batch_size, Tv, dim]`. * key: Optional key `Tensor` of shape `[batch_size, Tv, dim]`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: * query_mask: A boolean mask `Tensor` of shape `[batch_size, Tq]`. If given, the output will be zero at the positions where `mask==False`. * value_mask: A boolean mask `Tensor` of shape `[batch_size, Tv]`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Output: Attention outputs of shape `[batch_size, Tq, dim]`. [Optional] Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. The meaning of `query`, `value` and `key` depend on the application. In the case of text similarity, for example, `query` is the sequence embeddings of the first piece of text and `value` is the sequence embeddings of the second piece of text. `key` is usually the same tensor as `value`. Here is a code example for using `Attention` in a CNN+Attention network: ```python # Variable-length int sequences. query_input = tf.keras.Input(shape=(None,), dtype='int32') value_input = tf.keras.Input(shape=(None,), dtype='int32') # Embedding lookup. token_embedding = tf.keras.layers.Embedding(input_dim=1000, output_dim=64) # Query embeddings of shape [batch_size, Tq, dimension]. query_embeddings = token_embedding(query_input) # Value embeddings of shape [batch_size, Tv, dimension]. value_embeddings = token_embedding(value_input) # CNN layer. cnn_layer = tf.keras.layers.Conv1D( filters=100, kernel_size=4, # Use 'same' padding so outputs have the same shape as inputs. padding='same') # Query encoding of shape [batch_size, Tq, filters]. query_seq_encoding = cnn_layer(query_embeddings) # Value encoding of shape [batch_size, Tv, filters]. value_seq_encoding = cnn_layer(value_embeddings) # Query-value attention of shape [batch_size, Tq, filters]. query_value_attention_seq = tf.keras.layers.Attention()( [query_seq_encoding, value_seq_encoding]) # Reduce over the sequence axis to produce encodings of shape # [batch_size, filters]. query_encoding = tf.keras.layers.GlobalAveragePooling1D()( query_seq_encoding) query_value_attention = tf.keras.layers.GlobalAveragePooling1D()( query_value_attention_seq) # Concatenate query and document encodings to produce a DNN input layer. input_layer = tf.keras.layers.Concatenate()( [query_encoding, query_value_attention]) # Add DNN layers, and create Model. # ... ``` Fc stt|jf|||_dS)N)rrNr use_scale)rrOr)rrrrAszAttention.__init__cs>|jr$|jddt|jdd|_nd|_tt||dS)z*Creates scale variable if use_scale==True.scalerT)namer3 initializerr trainableN) rO add_weightr Zones_initializerrrPrrNbuild)r input_shape)rrrrUEs zAttention.buildcCs(tj||dd}|jdk r$||j9}|S)zCalculates attention scores as a query-key dot product. Args: query: Query tensor of shape `[batch_size, Tq, dim]`. key: Key tensor of shape `[batch_size, Tv, dim]`. Returns: Tensor of shape `[batch_size, Tq, Tv]`. T)Z transpose_bN)r r#rP)rrrr$rrrrRs   zAttention._calculate_scorescs4d|ji}tt|}tt|t|S)NrO)rOrrNrErFrArG)rrHrI)rrrrE`s zAttention.get_config)F) r?rJrKrLrrUrrErMrr)rrrNs c rNzkeras.layers.AdditiveAttentioncsBeZdZdZd fdd ZfddZddZfd d ZZS) AdditiveAttentionaAdditive attention layer, a.k.a. Bahdanau-style attention. Inputs are `query` tensor of shape `[batch_size, Tq, dim]`, `value` tensor of shape `[batch_size, Tv, dim]` and `key` tensor of shape `[batch_size, Tv, dim]`. The calculation follows the steps: 1. Reshape `query` and `value` into shapes `[batch_size, Tq, 1, dim]` and `[batch_size, 1, Tv, dim]` respectively. 2. Calculate scores with shape `[batch_size, Tq, Tv]` as a non-linear sum: `scores = tf.reduce_sum(tf.tanh(query + value), axis=-1)` 3. Use scores to calculate a distribution with shape `[batch_size, Tq, Tv]`: `distribution = tf.nn.softmax(scores)`. 4. Use `distribution` to create a linear combination of `value` with shape `[batch_size, Tq, dim]`: `return tf.matmul(distribution, value)`. Args: use_scale: If `True`, will create a variable to scale the attention scores. causal: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Call Args: inputs: List of the following tensors: * query: Query `Tensor` of shape `[batch_size, Tq, dim]`. * value: Value `Tensor` of shape `[batch_size, Tv, dim]`. * key: Optional key `Tensor` of shape `[batch_size, Tv, dim]`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: * query_mask: A boolean mask `Tensor` of shape `[batch_size, Tq]`. If given, the output will be zero at the positions where `mask==False`. * value_mask: A boolean mask `Tensor` of shape `[batch_size, Tv]`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. Output: Attention outputs of shape `[batch_size, Tq, dim]`. [Optional] Attention scores after masking and softmax with shape `[batch_size, Tq, Tv]`. The meaning of `query`, `value` and `key` depend on the application. In the case of text similarity, for example, `query` is the sequence embeddings of the first piece of text and `value` is the sequence embeddings of the second piece of text. `key` is usually the same tensor as `value`. Here is a code example for using `AdditiveAttention` in a CNN+Attention network: ```python # Variable-length int sequences. query_input = tf.keras.Input(shape=(None,), dtype='int32') value_input = tf.keras.Input(shape=(None,), dtype='int32') # Embedding lookup. token_embedding = tf.keras.layers.Embedding(max_tokens, dimension) # Query embeddings of shape [batch_size, Tq, dimension]. query_embeddings = token_embedding(query_input) # Value embeddings of shape [batch_size, Tv, dimension]. value_embeddings = token_embedding(value_input) # CNN layer. cnn_layer = tf.keras.layers.Conv1D( filters=100, kernel_size=4, # Use 'same' padding so outputs have the same shape as inputs. padding='same') # Query encoding of shape [batch_size, Tq, filters]. query_seq_encoding = cnn_layer(query_embeddings) # Value encoding of shape [batch_size, Tv, filters]. value_seq_encoding = cnn_layer(value_embeddings) # Query-value attention of shape [batch_size, Tq, filters]. query_value_attention_seq = tf.keras.layers.AdditiveAttention()( [query_seq_encoding, value_seq_encoding]) # Reduce over the sequence axis to produce encodings of shape # [batch_size, filters]. query_encoding = tf.keras.layers.GlobalAveragePooling1D()( query_seq_encoding) query_value_attention = tf.keras.layers.GlobalAveragePooling1D()( query_value_attention_seq) # Concatenate query and document encodings to produce a DNN input layer. input_layer = tf.keras.layers.Concatenate()( [query_encoding, query_value_attention]) # Add DNN layers, and create Model. # ... ``` Tc stt|jf|||_dS)N)rrWrrO)rrOr)rrrrszAdditiveAttention.__init__csht|d}|d}t|tjr(|j}|jrN|jd|gt|j dd|_ nd|_ t t | |dS)Nr+r/rPT)rQr3rRrrS)rZ TensorShaper@Z Dimensionr%rOrTr Zglorot_uniform_initializerrrPrrWrU)rrVZv_shapedim)rrrrUs  zAdditiveAttention.buildcCsJtj|dd}tj|dd}|jr*|j}nd}tj|t||ddS)zCalculates attention scores as a nonlinear sum of query and key. Args: query: Query tensor of shape `[batch_size, Tq, dim]`. key: Key tensor of shape `[batch_size, Tv, dim]`. Returns: Tensor of shape `[batch_size, Tq, Tv]`. r-)r.g?r/)rr2rOrPr Z reduce_sumtanh)rrrZ q_reshapedZ k_reshapedrPrrrrs z#AdditiveAttention._calculate_scorescs4d|ji}tt|}tt|t|S)NrO)rOrrWrErFrArG)rrHrI)rrrrEs zAdditiveAttention.get_config)T) r?rJrKrLrrUrrErMrr)rrrWfs e rWcCs@tjtj|tjddd}tjtj|tjddd}t||S)zCCreates a lower-triangular boolean mask over the last 2 dimensions.)r3rr-)r.r/)r ZcumsumrZonesrZint32Z greater_equal)r3Z row_indexZ col_indexrrrr5s r5cCs$|dkr |S|dkr|St||S)N)r logical_and)xyrrrr6s r6N)rLZtensorflow.python.frameworkrrrZtensorflow.python.kerasrZ)tensorflow.python.keras.engine.base_layerrZtensorflow.python.keras.utilsrZtensorflow.python.opsrr r r Z tensorflow.python.util.tf_exportr r rNrWr5r6rrrrs*           ;