B d1t@sdZddlZddlZddlZddlZddlmmZ ddl m Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZejZd dZddZddZedGdddeZ dS)z'Keras-based multi-head attention layer.N) constraints) initializers) regularizers)Layer) activation)core)regularization)tf_utils) tf_logging) keras_exportc std|ttt|||df}|}dxFt|D]:}||ksT||dkrb|7q<t|7|d7}q, , num_heads, channels)`. `bs` and `` are treated as ``. The attention operations can be generalized: (1) Query-key dot product: `(, , num_heads, channels), (, , num_heads, channels) -> (, num_heads, , )` (2) Combination: `(, num_heads, , ), (, , num_heads, channels) -> (, , num_heads, channels)` Args: rank: Rank of query, key, value tensors. attn_axes: List/tuple of axes, `[-1, rank)`, that attention will be applied to. Returns: Einsum equations. Ncsg|] }|qSr).0i)target_notationr]/var/www/html/venv/lib/python3.7/site-packages/keras/layers/attention/multi_head_attention.py Osz-_build_attention_equation..csg|] }|qSrr)rr)rrrrPscsg|] }|qSrr)rr)source_notationrrrQsz %s,%s->%s)_CHR_IDXtuplenpdeleterangejoinlen) rank attn_axesZ batch_dims letter_offsetrZproduct_notationZdot_product_equationattn_scores_rankZcombine_equationr)rrr_build_attention_equation)s*   8r c Csd}d}d}d}d}x,t|D] }t||} || 7}|| 7}qW||7}x,t|D] }t||} || 7}|| 7}qTW||7}x4t|D](}t||} || 7}|| 7}|| 7}qW|d|d|} | |t|fS)zFBuilds an einsum equation for projections inside multi-head attention.r r,z->)rrr) free_dims bound_dims output_dimsZ input_strZ kernel_strZ output_str bias_axesrrcharZequationrrr_build_proj_equationas,      r'cCsdg|t|t|S)N)rlist) output_rankZknown_last_dimsrrr_get_output_shape~sr*zkeras.layers.MultiHeadAttentionc seZdZdZd"fdd Zfd d Zed d Zd#d dZddZ d$ddZ ddZ d%ddZ d&ddZ d'ddZd(ddZd)ddZd*d d!ZZS)+MultiHeadAttentionaFMultiHeadAttention layer. This is an implementation of multi-headed attention as described in the paper "Attention is all you Need" (Vaswani et al., 2017). If `query`, `key,` `value` are the same, then this is self-attention. Each timestep in `query` attends to the corresponding sequence in `key`, and returns a fixed-width vector. This layer first projects `query`, `key` and `value`. These are (effectively) a list of tensors of length `num_attention_heads`, where the corresponding shapes are `(batch_size, , key_dim)`, `(batch_size, , key_dim)`, `(batch_size, , value_dim)`. Then, the query and key tensors are dot-producted and scaled. These are softmaxed to obtain attention probabilities. The value tensors are then interpolated by these probabilities, then concatenated back to a single tensor. Finally, the result tensor with the last dimension as value_dim can take an linear projection and return. When using `MultiHeadAttention` inside a custom layer, the custom layer must implement its own `build()` method and call `MultiHeadAttention`'s `_build_from_signature()` there. This enables weights to be restored correctly when the model is loaded. Examples: Performs 1D cross-attention over two sequence inputs with an attention mask. Returns the additional attention weights over heads. >>> layer = MultiHeadAttention(num_heads=2, key_dim=2) >>> target = tf.keras.Input(shape=[8, 16]) >>> source = tf.keras.Input(shape=[4, 16]) >>> output_tensor, weights = layer(target, source, ... return_attention_scores=True) >>> print(output_tensor.shape) (None, 8, 16) >>> print(weights.shape) (None, 2, 8, 4) Performs 2D self-attention over a 5D input tensor on axes 2 and 3. >>> layer = MultiHeadAttention( ... num_heads=2, key_dim=2, attention_axes=(2, 3)) >>> input_tensor = tf.keras.Input(shape=[5, 3, 4, 16]) >>> output_tensor = layer(input_tensor, input_tensor) >>> print(output_tensor.shape) (None, 5, 3, 4, 16) Args: num_heads: Number of attention heads. key_dim: Size of each attention head for query and key. value_dim: Size of each attention head for value. dropout: Dropout probability. use_bias: Boolean, whether the dense layers use bias vectors/matrices. output_shape: The expected shape of an output tensor, besides the batch and sequence dims. If not specified, projects back to the key feature dim. attention_axes: axes over which the attention is applied. `None` means attention over all axes, but batch, heads, and features. kernel_initializer: Initializer for dense layer kernels. bias_initializer: Initializer for dense layer biases. kernel_regularizer: Regularizer for dense layer kernels. bias_regularizer: Regularizer for dense layer biases. activity_regularizer: Regularizer for dense layer activity. kernel_constraint: Constraint for dense layer kernels. bias_constraint: Constraint for dense layer kernels. Call arguments: query: Query `Tensor` of shape `(B, T, dim)`. value: Value `Tensor` of shape `(B, S, dim)`. key: Optional key `Tensor` of shape `(B, S, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. attention_mask: a boolean mask of shape `(B, T, S)`, that prevents attention to certain positions. The boolean mask specifies which query elements can attend to which key elements, 1 indicates attention and 0 indicates no attention. Broadcasting can happen for the missing batch dimensions and the head dimension. return_attention_scores: A boolean to indicate whether the output should be `(attention_output, attention_scores)` if `True`, or `attention_output` if `False`. Defaults to `False`. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Defaults to either using the training mode of the parent layer/model, or False (inference) if there is no parent layer. use_causal_mask: A boolean to indicate whether to apply a causal mask to prevent tokens from attending to future tokens (e.g., used in a decoder Transformer). Returns: attention_output: The result of the computation, of shape `(B, T, E)`, where `T` is for target sequence shapes and `E` is the query input last dimension if `output_shape` is `None`. Otherwise, the multi-head outputs are projected to the shape specified by `output_shape`. attention_scores: [Optional] multi-head attention coefficients over attention axes. NTglorot_uniformzerosc stjf|d|_||_||_|r(|n||_||_||_||_t ||_ t | |_ t | |_t | |_t | |_t | |_t ||_|dk rt|tjjs|f|_n||_d|_d\|_|_|_dS)NTF)NNN)super__init__Zsupports_masking _num_heads_key_dim _value_dim_dropout _use_bias _output_shaperget_kernel_initializer_bias_initializerr_kernel_regularizer_bias_regularizer_activity_regularizerr_kernel_constraint_bias_constraint isinstance collectionsabcSized_attention_axes_built_from_signature _query_shape _key_shape _value_shape)self num_headskey_dim value_dimdropoutuse_bias output_shapeattention_axeskernel_initializerbias_initializerkernel_regularizerbias_regularizeractivity_regularizerkernel_constraintbias_constraintkwargs) __class__rrr0s*          zMultiHeadAttention.__init__cs|j|j|j|j|j|j|jt|j t|j t |j t |j t |jt|jt|j|j|j|jd}t}tt|t|S)N)rIrJrKrLrMrNrOrPrQrRrSrTrUrV query_shape key_shape value_shape)r1r2r3r4r5r6rCr serializer8r9rr:r;r<rr=r>rErFrGr/ get_configdictr(items)rHconfigZ base_config)rXrrr]s,      zMultiHeadAttention.get_configcCsZ|d}|d}|d}|f|}d|||gkrHtdt|n|||||S)NrYrZr[zOne of dimensions of the input shape is missing. It should have been memorized when the layer was serialized. %s is created without weights.)poploggingwarningstr_build_from_signature)clsr`rYrZr[layerrrr from_config/s     zMultiHeadAttention.from_configc Csd|_t|dr t|j|_n t||_t|drFt|j|_n t||_|dkrd|j|_n&t|dr~t|j|_n t||_t |4|jj d}t |ddd\}}}t j |ft|d|j|jg|jr|nddd||_t |jj dddd\}}}t j |ft|d|j|jg|jr8|ndd d||_t |jj dddd\}}}t j |ft|d|j|jg|jr|ndd d||_|||||d |_WdQRXdS) aBuilds layers and variables. Once the method is called, self._built_from_signature will be set to True. Args: query: Query tensor or TensorShape. value: Value tensor or TensorShape. key: Key tensor or TensorShape. TshapeNr )r#r$query)rNr%namekeyvalueattention_output)rDhasattrtf TensorShaperirErGrFr Zmaybe_init_scoperr'r EinsumDenser*r1r2r5_get_common_kwargs_for_sublayer _query_dense _key_denser3 _value_dense_build_attention_make_output_dense _output_dense)rHrkrnrmr"einsum_equationr%r)rrrreBs\           z(MultiHeadAttention._build_from_signaturecCsXt|j|j|j|j|jd}|jj|j }|j j|j }||d<||d<|S)N)rRrSrTrUrVrPrQ) r^r:r;r<r=r>r8rXrhr]r9)rH common_kwargsrPrQrrrrts   z2MultiHeadAttention._get_common_kwargs_for_sublayercCsz|jr(t|jtjjs |jg}q4|j}n |jdg}t|dt|d\}}}tj |ft |d||j rj|nd|d|S)a*Builds the output projection matrix. Args: free_dims: Number of free dimensions for einsum equation building. common_kwargs: Common keyword arguments for einsum layer. name: Name for the projection layer. Returns: Projection layer. rj)r#r$r N)rNr%rl) r6r?r@rArBrEr'rrrsr*r5)rHr"r|rlrNr{r%r)rrrrys     z%MultiHeadAttention._make_output_densecCs~|jdkr ttd|d|_n t|j|_t||jd\|_|_}tt|t|j|}tj|d|_ t j |j d|_ dS)a1Builds multi-head dot-product attention computations. This function builds attributes necessary for `_compute_attention` to customize attention computation to replace the default dot-product attention. Args: rank: the rank of query, key, value tensors. Nr rj)r)axis)Zrate)rCrrr _dot_product_equation_combine_equationrrZSoftmax_softmaxrZDropoutr4_dropout_layer)rHrrZ norm_axesrrrrxs  z#MultiHeadAttention._build_attentioncCsX|dk rLt|j dd}x.tt|jt|jD]}tj||d}q6W|||S)Nrjr )r~)rrCrrirqZ expand_dimsr)rHattention_scoresattention_maskZmask_expansion_axis_rrr_masked_softmaxsz"MultiHeadAttention._masked_softmaxc Cs^t|dtt|j}t|j||}|||}|j ||d}t|j ||}||fS)aApplies Dot-product attention with query, key, value tensors. This function defines the computation inside `call` with projected multi-head Q, K, V inputs. Users can override this function for customized attention implementation. Args: query: Projected query `Tensor` of shape `(B, T, N, key_dim)`. key: Projected key `Tensor` of shape `(B, S, N, key_dim)`. value: Projected value `Tensor` of shape `(B, S, N, value_dim)`. attention_mask: a boolean mask of shape `(B, T, S)`, that prevents attention to certain positions. It is generally not needed if the `query` and `value` (and/or `key`) are masked. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (doing nothing). Returns: attention_output: Multi-headed outputs of attention computation. attention_scores: Multi-headed attention weights. g?)training) rqmultiplymathsqrtfloatr2Zeinsumrrrr) rHrkrmrnrrrZattention_scores_dropoutrorrr_compute_attentions  z%MultiHeadAttention._compute_attentionFcCs:|j|||||d}|js*|j|||d|dkr6|}t|tj}|rV|} |}t|tj} t|tj} | r| rtj | | } |j| d}|j| d}n.| r|jt |d}n| r|jt |d}| |}| |}||}||||||\} }|| } |r(tjj| | d} |r6| |fS| S)N)rmruse_causal_mask)rkrnrm)ri)lengths)_compute_attention_maskrDrer?rqZ RaggedTensorZnested_row_lengthsZ to_tensorrmaximumbounding_shaperirurvrwrrzZ from_tensor)rHrkrnrmrZreturn_attention_scoresrrZquery_is_raggedZ query_lengthsZ key_is_raggedZvalue_is_raggedrrorrrrcallsJ          zMultiHeadAttention.callc Cs*t|dd}t|dd}t|dd}d} |dk rVt|tj}|ddddtjf} |dk rt|tj}|ddtjddf} | dkr| n| | @} |dk rt|tj}|ddtjddf} | dkr| n| | @} |r|||} | dkr| n| | @} | dk r&|dkr| nt|t| @}|S)aComputes the attention mask, using the Keras masks of the inputs. * The `query`'s mask is reshaped from [B, T] to [B, T, 1]. * The `value`'s mask is reshaped from [B, S] to [B, 1, S]. * The `key`'s mask is reshaped from [B, S] to [B, 1, S]. The `key`'s mask is ignored if `key` is `None` or if `key is value`. * If `use_causal_mask=True`, then the causal mask is computed. Its shape is [1, T, S]. All defined masks are merged using a logical AND operation (`&`). In general, if the `query` and `value` are masked, then there is no need to define the `attention_mask`. Args: query: Projected query `Tensor` of shape `(B, T, N, key_dim)`. key: Projected key `Tensor` of shape `(B, T, N, key_dim)`. value: Projected value `Tensor` of shape `(B, T, N, value_dim)`. attention_mask: a boolean mask of shape `(B, T, S)`, that prevents attention to certain positions. use_causal_mask: A boolean to indicate whether to apply a causal mask to prevent tokens from attending to future tokens (e.g., used in a decoder Transformer). Returns: attention_mask: a boolean mask of shape `(B, T, S)`, that prevents attention to certain positions, based on the Keras masks of the `query`, `key`, `value`, and `attention_mask` tensors, and the causal mask if `use_causal_mask=True`. Z _keras_maskN)getattrrqcastboolZnewaxis_compute_causal_mask) rHrkrnrmrrZ query_maskZ value_maskZkey_maskZ auto_maskmaskrrrras,!     z*MultiHeadAttention._compute_attention_maskcCsHt|d}|dkr|n t|d}tjtd||ftjddS)aComputes a causal mask (e.g., for masked self-attention layers). For example, if query and value both contain sequences of length 4, this function returns a boolean `Tensor` equal to: ``` [[[True, False, False, False], [True, True, False, False], [True, True, True, False], [True, True, True, True]]] ``` Args: query: query `Tensor` of shape `(B, T, ...)`. value: value `Tensor` of shape `(B, S, ...)` (optional, defaults to query). Returns: mask: a boolean `Tensor` of shape [1, T, S] containing a lower triangular matrix of shape [T, S]. r Nr}r)rqriZlinalgZ band_partZonesr)rHrkrnZ q_seq_lengthZ v_seq_lengthrrrrsz'MultiHeadAttention._compute_causal_maskcCs|dkr |}t|}t|}t|}|d|dkrXtd|dd|dd|dd|ddkrtd|d||jr|dd|jS|S)Nr}zMThe last dimension of `query_shape` and `value_shape` must be equal, but are z, z@. Received: query_shape={query_shape}, value_shape={value_shape}r zRAll dimensions of `value` and `key`, except the last one, must be equal. Received z and )rqrr ValueErrorr6Z concatenate)rHrYr[rZrrrcompute_output_shapes   z'MultiHeadAttention.compute_output_shape) Nr,TNNr-r.NNNNN)N)N)N)NN)NNFNF)NNF)N)N)__name__ __module__ __qualname____doc__r0r] classmethodrhrertryrxrrrrrr __classcell__rr)rXrr+s:d   O   3 < ? r+)!rr@rstringnumpyrZtensorflow.compat.v2compatZv2rqZkerasrrrZkeras.engine.base_layerrZ keras.layersrrrZ keras.utilsr Ztensorflow.python.platformr rbZ tensorflow.python.util.tf_exportr ascii_lowercaserr r'r*r+rrrrs*          8