B ӻd@sdZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddl mZddl mZdd l mZdd l mZdd l mZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddl"m#Z$ddl%m&Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-dZ.e+dGdddeZ/e+dGd d!d!eZ0e+d"Gd#d$d$eZ1e-j2Gd%d&d&e3Z4e+d'Gd(d)d)e4eZ5e+d*Gd+d,d,e0Z6e+d-gd.Gd/d0d0e4eZ7e+d1gd.Gd2d3d3e0Z8e+d4gd.Gd5d6d6e4eZ9e+d7Gd8d9d9e9Z:e+d:gd.Gd;d<dd?ZdDdEZ?dFdGZ@dHdIZAdJdKZBdS)Mz(Recurrent layers and their base classes.N)distribution_strategy_context)context)ops) tensor_shape) activations)backend) constraints) initializers) regularizers)Layer) InputSpec)layer_serialization)control_flow_util) generic_utils)tf_utils) array_ops)control_flow_ops)math_ops) state_ops) tf_logging)base)nest) keras_export) doc_controlszbRNN `implementation=2` is not supported when `recurrent_dropout` is set. Using `implementation=1`.zkeras.layers.StackedRNNCellscsteZdZdZfddZeddZeddZdd d Zdd d Z e j d dZ fddZ edddZZS)StackedRNNCellsaMWrapper allowing a stack of RNN cells to behave as a single cell. Used to implement efficient stacked RNNs. Args: cells: List of RNN cell instances. Examples: ```python batch_size = 3 sentence_max_length = 5 n_features = 2 new_shape = (batch_size, sentence_max_length, n_features) x = tf.constant(np.reshape(np.arange(30), new_shape), dtype = tf.float32) rnn_cells = [tf.keras.layers.LSTMCell(128) for _ in range(2)] stacked_lstm = tf.keras.layers.StackedRNNCells(rnn_cells) lstm_layer = tf.keras.layers.RNN(stacked_lstm) result = lstm_layer(x) ``` c stx8|D]0}dt|kr td|dt|krtd|qW||_|dd|_|jr^tdtt|j f|dS)Ncallz4All cells must have a `call` method. received cells: state_sizez=All cells must have a `state_size` attribute. received cells:reverse_state_orderFzreverse_state_order=True in StackedRNNCells will soon be deprecated. Please update the code to work with the natural order of states if you rely on the RNN states, eg RNN(return_state=True).) dir ValueErrorcellspoprloggingwarningsuperr__init__)selfr kwargscell) __class__Z/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/layers/recurrent.pyr%Qs     zStackedRNNCells.__init__cCs*tdd|jr|jdddn|jDS)Ncss|] }|jVqdS)N)r).0cr*r*r+ isz-StackedRNNCells.state_size..)tuplerr )r&r*r*r+rgszStackedRNNCells.state_sizecCsRt|jddddk r"|jdjSt|jdjrB|jdjdS|jdjSdS)Nr/ output_sizer)getattrr r1_is_multiple_stater)r&r*r*r+r1ls  zStackedRNNCells.output_sizeNcCslg}x^|jr|jdddn|jD]>}t|dd}|rL|||||dq"|t||||q"Wt|S)Nr/get_initial_state)inputs batch_sizedtype)rr r2append$_generate_zero_filled_state_for_cellr0)r&r5r6r7Zinitial_statesr(get_initial_state_fnr*r*r+r4us" z!StackedRNNCells.get_initial_statec Ks |jr|jdddn|j}t|t|}g}xt|j|D]\} }t|rV|n|g}t| dddk } t |dkr| r|dn|}t | j dr||d<n | ddt| r| jn| j } t | j dr| ||fd|i|\}}n| ||f|\}}||q@W|t|t|fS)Nr/_is_tf_rnn_cellrtraining constants)rrrpack_sequence_asflattenzipr is_nestedr2lenrhas_argrr!callable__call__r8) r&r5statesr>r=r'rZ nested_statesZnew_nested_statesr(is_tf_rnn_cell cell_call_fnr*r*r+rs$   zStackedRNNCells.callc Cst|tr|d}x|jD]}t|trV|jsVt|j||d|_WdQRXt |dddk rn|j }nt |j r|j d}n|j }t |dgt|}qWd|_dS)NrTr1) isinstancelistr r builtr name_scopenamebuildr2r1r3rr0r TensorShapeas_list)r& input_shaper( output_dimr*r*r+rOs      zStackedRNNCells.buildcsVg}x|jD]}|t|q Wd|i}tt|}tt| t| S)Nr ) r r8rserialize_keras_objectr$r get_configdictrKitems)r&r r(config base_config)r)r*r+rUs  zStackedRNNCells.get_configcCsBddlm}g}x$|dD]}||||dqW||f|S)Nr) deserializer )custom_objects)tensorflow.python.keras.layersrZr!r8)clsrXr[deserialize_layerr Z cell_configr*r*r+ from_configs  zStackedRNNCells.from_config)NNN)NN)N)__name__ __module__ __qualname____doc__r%propertyrr1r4rrshape_type_conversionrOrU classmethodr_ __classcell__r*r*)r)r+r7s    rzkeras.layers.RNNcseZdZdZd'fdd ZefddZeddZeje j d dZd d Z d d Z ddZ eddZejddZd(fdd Zd)ddZddZddZddZd*dd Zfd!d"Zed+d#d$Zed%d&ZZS),RNNa#Base class for recurrent layers. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. Args: cell: A RNN cell instance or a list of RNN cell instances. A RNN cell is a class that has: - A `call(input_at_t, states_at_t)` method, returning `(output_at_t, states_at_t_plus_1)`. The call method of the cell can also take the optional argument `constants`, see section "Note on passing external constants" below. - A `state_size` attribute. This can be a single integer (single state) in which case it is the size of the recurrent state. This can also be a list/tuple of integers (one size per state). The `state_size` can also be TensorShape or tuple/list of TensorShape, to represent high dimension state. - A `output_size` attribute. This can be a single integer or a TensorShape, which represent the shape of the output. For backward compatible reason, if this attribute is not available for the cell, the value will be inferred by the first element of the `state_size`. - A `get_initial_state(inputs=None, batch_size=None, dtype=None)` method that creates a tensor meant to be fed to `call()` as the initial state, if the user didn't specify any initial state via other means. The returned initial state should have a shape of [batch_size, cell.state_size]. The cell might choose to create a tensor full of zeros, or full of other values based on the cell's implementation. `inputs` is the input tensor to the RNN layer, which should contain the batch size as its shape[0], and also dtype. Note that the shape[0] might be `None` during the graph construction. Either the `inputs` or the pair of `batch_size` and `dtype` are provided. `batch_size` is a scalar tensor that represents the batch size of the inputs. `dtype` is `tf.DType` that represents the dtype of the inputs. For backward compatibility, if this method is not implemented by the cell, the RNN layer will create a zero filled tensor with the size of [batch_size, cell.state_size]. In the case that `cell` is a list of RNN cell instances, the cells will be stacked on top of each other in the RNN, resulting in an efficient stacked RNN. return_sequences: Boolean (default `False`). Whether to return the last output in the output sequence, or the full sequence. return_state: Boolean (default `False`). Whether to return the last state in addition to the output. go_backwards: Boolean (default `False`). If True, process the input sequence backwards and return the reversed sequence. stateful: Boolean (default `False`). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. unroll: Boolean (default `False`). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. time_major: The shape format of the `inputs` and `outputs` tensors. If True, the inputs and outputs will be in shape `(timesteps, batch, ...)`, whereas in the False case, it will be `(batch, timesteps, ...)`. Using `time_major = True` is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. However, most TensorFlow data is batch-major, so by default this function accepts input and emits output in batch-major form. zero_output_for_mask: Boolean (default `False`). Whether the output should use zeros for the masked timesteps. Note that this field is only used when `return_sequences` is True and mask is provided. It can useful if you want to reuse the raw output sequence of the RNN without interference from the masked timesteps, eg, merging bidirectional RNNs. Call arguments: inputs: Input tensor. mask: Binary tensor of shape `[batch_size, timesteps]` indicating whether a given timestep should be masked. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. This argument is passed to the cell when calling it. This is for use with cells that use dropout. initial_state: List of initial state tensors to be passed to the first call of the cell. constants: List of constant tensors to be passed to the cell at each timestep. Input shape: N-D tensor with shape `[batch_size, timesteps, ...]` or `[timesteps, batch_size, ...]` when time_major is True. Output shape: - If `return_state`: a list of tensors. The first tensor is the output. The remaining tensors are the last states, each with shape `[batch_size, state_size]`, where `state_size` could be a high dimension tensor shape. - If `return_sequences`: N-D tensor with shape `[batch_size, timesteps, output_size]`, where `output_size` could be a high dimension tensor shape, or `[timesteps, batch_size, output_size]` when `time_major` is True. - Else, N-D tensor with shape `[batch_size, output_size]`, where `output_size` could be a high dimension tensor shape. Masking: This layer supports masking for input data with a variable number of timesteps. To introduce masks to your data, use an [tf.keras.layers.Embedding] layer with the `mask_zero` parameter set to `True`. Note on using statefulness in RNNs: You can set RNN layers to be 'stateful', which means that the states computed for the samples in one batch will be reused as initial states for the samples in the next batch. This assumes a one-to-one mapping between samples in different successive batches. To enable statefulness: - Specify `stateful=True` in the layer constructor. - Specify a fixed batch size for your model, by passing If sequential model: `batch_input_shape=(...)` to the first layer in your model. Else for functional model with 1 or more Input layers: `batch_shape=(...)` to all the first layers in your model. This is the expected shape of your inputs *including the batch size*. It should be a tuple of integers, e.g. `(32, 10, 100)`. - Specify `shuffle=False` when calling `fit()`. To reset the states of your model, call `.reset_states()` on either a specific layer, or on your entire model. Note on specifying the initial state of RNNs: You can specify the initial state of RNN layers symbolically by calling them with the keyword argument `initial_state`. The value of `initial_state` should be a tensor or list of tensors representing the initial state of the RNN layer. You can specify the initial state of RNN layers numerically by calling `reset_states` with the keyword argument `states`. The value of `states` should be a numpy array or list of numpy arrays representing the initial state of the RNN layer. Note on passing external constants to RNNs: You can pass "external" constants to the cell using the `constants` keyword argument of `RNN.__call__` (as well as `RNN.call`) method. This requires that the `cell.call` method accepts the same keyword argument `constants`. Such constants can be used to condition the cell transformation on additional static inputs (not changing over time), a.k.a. an attention mechanism. Examples: ```python # First, let's define a RNN Cell, as a layer subclass. class MinimalRNNCell(keras.layers.Layer): def __init__(self, units, **kwargs): self.units = units self.state_size = units super(MinimalRNNCell, self).__init__(**kwargs) def build(self, input_shape): self.kernel = self.add_weight(shape=(input_shape[-1], self.units), initializer='uniform', name='kernel') self.recurrent_kernel = self.add_weight( shape=(self.units, self.units), initializer='uniform', name='recurrent_kernel') self.built = True def call(self, inputs, states): prev_output = states[0] h = backend.dot(inputs, self.kernel) output = h + backend.dot(prev_output, self.recurrent_kernel) return output, [output] # Let's use this cell in a RNN layer: cell = MinimalRNNCell(32) x = keras.Input((None, 5)) layer = RNN(cell) y = layer(x) # Here's how to use the cell to build a stacked RNN: cells = [MinimalRNNCell(32), MinimalRNNCell(64)] x = keras.Input((None, 5)) layer = RNN(cells) y = layer(x) ``` Fc  st|ttfrt|}dt|kr,td|dt|kr@td|dd|_d|krd|ksfd |kr|d d|ddf} | |d<tt |j f|||_ ||_ ||_ ||_||_||_||_d |_d|_d|_d|_d|_d |_|rtrtd dS) Nrz7`cell` should have a `call` method. The RNN was passed:rzbThe RNN cell should have an attribute `state_size` (tuple of integers, one integer per RNN state).zero_output_for_maskFrR input_dim input_lengthTrzFRNNs with stateful=True not yet supported with tf.distribute.Strategy.)rJrKr0rrrr!rir$rhr%r(return_sequences return_state go_backwardsstatefulunroll time_majorZsupports_masking input_spec state_spec_statesconstants_spec_num_constants ds_contextZ has_strategy) r&r(rlrmrnrorprqr'rR)r)r*r+r%s<    z RNN.__init__cs|jr dStt|jS)NF)rpr$rh!_use_input_spec_as_call_signature)r&)r)r*r+rxsz%RNN._use_input_spec_as_call_signaturecCs<|jdkr6tdd|jj}t|jjr0|S|gS|jS)NcSsdS)Nr*)_r*r*r+zRNN.states..)rtr map_structurer(rrB)r&stater*r*r+rGs z RNN.statescCs ||_dS)N)rt)r&rGr*r*r+rGsc s"t|tr|d}yt|}Wn&ttfk rFt|d}YnX|d|djrht j j r~j j }n j j g}fdd}t j dddk rtt |j j}t|dkr|dn|}n ||d}jrfdd}t ||}t|t|S|SdS)Nrr<csVt|}jrBjr.tg|}qRtg|}ntg|}|S)N)rrPrQrlrq)Zflat_output_sizerS output_shape)batchr& time_stepr*r+_get_output_shapesz3RNN.compute_output_shape.._get_output_shaper1csgt|}t|S)N)rrPrQ)Z flat_state state_shape)rr*r+_get_state_shapesz2RNN.compute_output_shape.._get_state_shape)rJrKrrPr TypeErrorrr@rqr3r(rr2r|r1rCrmrto_list)r&rRrrr~rrr*)rr&rr+compute_output_shapes0         zRNN.compute_output_shapecCsDt|d}|jr|nd}|jrsz$RNN.compute_mask..)rr@rlrmrG)r&r5maskZ output_maskZ state_maskr*r*r+ compute_masks  zRNN.compute_maskc s|t|tr|d}fdd}fdd}yt|}Wnttfk rPYnXt|sjdk rv||jd<n ||g_||}n>jdk rt ||jd<nt t ||_t ||}tj t rj jstj jj |dj _WdQRXtj jr,tj j}n j jg}jdk rR|jndd|D_jrrd_dS) NrcsTt|tjr|}nt|}jr(dnd\}}js>d||<d||<tt|dS)z!Convert input shape to InputSpec.)r<r)rr<N)shape) rJrrPrQrKrqror r0)rZinput_spec_shapeZ batch_indexZtime_step_index)r&r*r+get_input_specs  z!RNN.build..get_input_speccs@t|tjrt|}jr*|ddS|df|ddS)Nr<r)rJrrPr0rQrq)r)r&r*r+get_step_input_shape)s  z'RNN.build..get_step_input_shapeTcSs&g|]}tdgt|dqS)N)r)r rrPrQ)r,dimr*r*r+rXszRNN.build..)rJrKrrPrrrrBrrr|rrr(r rLrrMrNrOr3rrs_validate_state_specro reset_states)r&rRrrZstep_input_shaperr*)r&r+rOs@            z RNN.buildcCsxtd||}t|}t|}t|t|kr8|x:t||D],\}}t|jdd t|sD|qDWdS)aeValidate the state spec between the initial_state and the state_size. Args: cell_state_sizes: list, the `state_size` attribute from the cell. init_state_specs: list, the `state_spec` from the initial_state that is passed in `call()`. Raises: ValueError: When initial state spec is not compatible with the state size. zAn `initial_state` was passed that is not compatible with `cell.state_size`. Received `state_spec`={}; however `cell.state_size` is {}r<N) rformatrr@rCrArrPrZis_compatible_with)Zcell_state_sizesZinit_state_specsZvalidation_errorZflat_cell_state_sizesZflat_state_specsZcell_state_specZcell_state_sizer*r*r+r_s     zRNN._validate_state_speccCst|jdd}t|r&t|d}t|}|jr>|dn|d}|j}|r`|d||d}nt ||jj |}t|s|g}t |S)Nr4rr<)r5r6r7) r2r(rrBr@rrrqr7_generate_zero_filled_staterrK)r&r5r:rRr6r7Z init_stater*r*r+r4}s    zRNN.get_initial_stateNc  st||||j\}}}|dkr:|dkr:tt|j|f|Sg}g}|dk rn||7}tdd||_||j7}|dk r||7}dd|D|_t ||_||j7}t |}|rt |dnd}x"|D]} t | |krt dqW|rX|g|} |jr |j|} nttdd||} | |_tt|j| f|} |jdt | |_| S|dk rj||d <|dk r|||d <tt|j|f|SdS) NcSstt|dS)N)r)r r int_shape)sr*r*r+rzr{zRNN.__call__..cSsg|]}tt|dqS))r)r rr)r,Zconstantr*r*r+rsz RNN.__call__..rTzThe initial state or constants of an RNN layer cannot be specified with a mix of Keras tensors and non-Keras tensors (a "Keras tensor" is a tensor that was returned by a Keras layer, or by `Input`)cSsdS)Nr*)ryr*r*r+rzr{ initial_stater>)_standardize_argsrvr$rhrFrr|rsrurCr@ris_keras_tensorrrLrrrr) r&r5rr>r'Zadditional_inputsZadditional_specsZflat_additional_inputsrZtensorZ full_inputZfull_input_specoutput)r)r*r+rFsL          z RNN.__call__c s2t|\}}|dk }|||||\}}}jtjtrjxjjD]}|qXW|dk rt |d}t |rt t |d} n t |} j r| dn| d} jr| dkrtditjjdr|d<tjdddk tjrjjnjj|rTtjjds@tdfdd } nfd d } tj| |||j|j|dk r|n| j jd \} } }jrd d tt jt |D}|jrtj|| |jd}n| }jr*t|t t!fs|g}nt |}t"||S|SdS)Nrr<aCannot unroll a RNN if the time dimension is undefined. - If using a Sequential model, specify the time dimension by passing an `input_shape` or `batch_input_shape` argument to your first layer. If your first layer is an Embedding, you can also use the `input_length` argument. - If using the functional API, specify the time dimension by passing a `shape` or `batch_shape` argument to your Input layer.r=r;r>z#RNN cell does not support constantscsn|j d}|dj }t|dkr8r8|dn|}||fd|i\}}t|sf|g}||fS)Nr<rr>)rvrCrrB)r5rGr>r new_states)rIrHr'r&r*r+steps zRNN.call..stepcsFt|dkrr|dn|}||f\}}t|s>|g}||fS)Nr<r)rCrrB)r5rGrr)rIrHr'r*r+rs  )r>rnrrprkrqricSsg|]\}}t||qSr*)rZassign)r,Z self_stater}r*r*r+r0szRNN.call..)rn)#rZconvert_inputs_if_ragged_validate_args_if_ragged_process_inputs_maybe_reset_cell_dropout_maskr(rJrr rr@rBrrqrprrrDrr2rErFZrnnrnrirorArGZ add_updaterlZmaybe_convert_to_raggedrmrKr0r)r&r5rr=rr>Z row_lengthsis_ragged_inputr(rRZ timestepsrZ last_outputoutputsrGZupdatesrr*)rIrHr'r&r+rsh        zRNN.callcst|tjjrdt|tsdjs,|ddn |dj |j d}tdkr\d|d}jrdk rt ddt j D}t j|dkfddfdddd qЈj ndkrЈ|ttj krtd ttj d ttd ||fS) Nr<rcSsg|]}t|qSr*)rZcount_nonzero_v2)r,rr*r*r+rZsz'RNN._process_inputs..csjS)N)rGr*)r&r*r+rz^r{z%RNN._process_inputs..csS)Nr*r*)rr*r+rz_r{T)Ztrue_fnZfalse_fnstrictz Layer has z states but was passed z initial states.)rJ collectionsabcSequencer0rvrCrorZadd_nrr@rGrZcondr4rstr)r&r5rr>Znon_zero_countr*)rr&r+rDs2         zRNN._process_inputscCs6|sdS|dk r$tdt|d|jr2tddS)Nz The mask that was passed in was zs and cannot be applied to RaggedTensor inputs. Please make sure that there is no mask passed in by upstream layers.zThe input received contains RaggedTensors and does not support unrolling. Disable unrolling by passing `unroll=False` in the RNN Layer constructor.)rrrp)r&rrr*r*r+rls zRNN._validate_args_if_raggedcCst|tr||dS)N)rJDropoutRNNCellMixinreset_dropout_maskreset_recurrent_dropout_mask)r&r(r*r*r+rzs z"RNN._maybe_reset_cell_dropout_maskc CsJ|jstdd}|jdk r2t|jddj}|dkr@d}n|jrN|dn|d}|sbtdt|jddkrt |j ddrt|j j d||j pt d}n tt||j j|j pt }tt j|}t|j j||_t|jsF|jg|_n@|dkr`xLtt|jt|j jD],\}}t |t|gt|q.Wnt|j}t|} t| t|krtd|jd tt|d tt| d t|g} xttt| |D]b\} \} }| j|jkr(td t| d |jdt||fdt| j| || fqWt !| dS)aReset the recorded states for the stateful RNN layer. Can only be used when RNN layer is constructed with `stateful` = `True`. Args: states: Numpy arrays that contains the value for the initial state, which will be feed to cell at the first time step. When the value is None, zero filled numpy array will be created based on the cell state size. Raises: AttributeError: When the RNN layer is not stateful. ValueError: When the batch size of the RNN layer is unknown. ValueError: When the input numpy array is not compatible with the RNN layer state, either size wise or dtype wise. zLayer must be stateful.Nrr<aIIf a RNN is stateful, it needs to know its batch size. Specify the batch size of your input tensors: - If using a Sequential model, specify the batch size by passing a `batch_input_shape` argument to your first layer. - If using the functional API, specify the batch size by passing a `batch_shape` argument to your Input layer.r4)r5r6r7zLayer z expects z states, but it received z state values. Input received: zState z is incompatible with layer z: expected shape=z, found shape=)"roAttributeErrorrrrr@rrqrrGr2r(r4r7rZfloatxrrr|variabler?rBrA set_valuenpzerosrrPrQrCrNr enumerater8Zbatch_set_value) r&rGZ spec_shaper6Zflat_init_state_valuesZflat_states_variablesr}sizeZ flat_statesZflat_input_statesZset_value_tuplesivaluer*r*r+rsT      $  86zRNN.reset_statescsx|j|j|j|j|j|jd}|jr.|j|d<|jr>|j|d<t |j |d<t t | }tt|t|S)N)rlrmrnrorprq num_constantsrir()rlrmrnrorprqrvrirrTr(r$rhrUrVrKrW)r&rXrY)r)r*r+rUs   zRNN.get_configcCs@ddlm}||d|d}|dd}||f|}||_|S)Nr)rZr()r[r)r\rZr!rv)r]rXr[r^r(rlayerr*r*r+r_s    zRNN.from_configcCs t|S)N)r ZRNNSavedModelSaver)r&r*r*r+_trackable_saved_model_saversz RNN._trackable_saved_model_saver)FFFFFF)NN)NNNN)N)N)r`rarbrcr%rdrxrGsetter trackable no_automatic_dependency_trackingrrrO staticmethodrrZdo_not_doc_inheritabler4rFrrrrrrUrfr_rrgr*r*)r)r+rhs<@. 5I D h( N  rhzkeras.layers.AbstractRNNCellc@s:eZdZdZddZeddZeddZd d d ZdS) AbstractRNNCellaAbstract object representing an RNN cell. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. This is the base class for implementing RNN cells with custom behavior. Every `RNNCell` must have the properties below and implement `call` with the signature `(output, next_state) = call(input, state)`. Examples: ```python class MinimalRNNCell(AbstractRNNCell): def __init__(self, units, **kwargs): self.units = units super(MinimalRNNCell, self).__init__(**kwargs) @property def state_size(self): return self.units def build(self, input_shape): self.kernel = self.add_weight(shape=(input_shape[-1], self.units), initializer='uniform', name='kernel') self.recurrent_kernel = self.add_weight( shape=(self.units, self.units), initializer='uniform', name='recurrent_kernel') self.built = True def call(self, inputs, states): prev_output = states[0] h = backend.dot(inputs, self.kernel) output = h + backend.dot(prev_output, self.recurrent_kernel) return output, output ``` This definition of cell differs from the definition used in the literature. In the literature, 'cell' refers to an object with a single scalar output. This definition refers to a horizontal array of such units. An RNN cell, in the most abstract setting, is anything that has a state and performs some operation that takes a matrix of inputs. This operation results in an output matrix with `self.output_size` columns. If `self.state_size` is an integer, this operation also results in a new state matrix with `self.state_size` columns. If `self.state_size` is a (possibly nested tuple of) TensorShape object(s), then it should return a matching structure of Tensors having shape `[batch_size].concatenate(s)` for each `s` in `self.batch_size`. cCs tddS)aThe function that contains the logic for one RNN step calculation. Args: inputs: the input tensor, which is a slide from the overall RNN input by the time dimension (usually the second dimension). states: the state tensor from previous step, which has the same shape as `(batch, state_size)`. In the case of timestep 0, it will be the initial state user specified, or zero filled tensor otherwise. Returns: A tuple of two tensors: 1. output tensor for the current timestep, with size `output_size`. 2. state tensor for next step, which has the shape of `state_size`. zAbstract methodN)NotImplementedError)r&r5rGr*r*r+r%szAbstractRNNCell.callcCs tddS)zsize(s) of state(s) used by this cell. It can be represented by an Integer, a TensorShape or a tuple of Integers or TensorShapes. zAbstract methodN)r)r&r*r*r+r6szAbstractRNNCell.state_sizecCs tddS)z>Integer or TensorShape: size of outputs produced by this cell.zAbstract methodN)r)r&r*r*r+r1?szAbstractRNNCell.output_sizeNcCst||||S)N)r9)r&r5r6r7r*r*r+r4Dsz!AbstractRNNCell.get_initial_state)NNN) r`rarbrcrrdrr1r4r*r*r*r+rs 6 rcs~eZdZdZfddZejddZddZdd Z dd d Z dd dZ dddZ dddZ fddZfddZZS)ra_Object that hold dropout related fields for RNN Cell. This class is not a standalone RNN cell. It suppose to be used with a RNN cell by multiple inheritance. Any cell that mix with class should have following fields: dropout: a float number within range [0, 1). The ratio that the input tensor need to dropout. recurrent_dropout: a float number within range [0, 1). The ratio that the recurrent state weights need to dropout. This object will create and cache created dropout masks, and reuse them for the incoming data, so that the same mask is used for every batch input. cs|tt|j||dS)N) _create_non_trackable_mask_cacher$rr%)r&argsr')r)r*r+r%WszDropoutRNNCellMixin.__init__cCs t|j|_t|j|_dS)aCreate the cache for dropout and recurrent dropout mask. Note that the following two masks will be used in "graph function" mode, e.g. these masks are symbolic tensors. In eager mode, the `eager_*_mask` tensors will be generated differently than in the "graph function" case, and they will be cached. Also note that in graph mode, we still cache those masks only because the RNN could be created with `unroll=True`. In that case, the `cell.call()` function will be invoked multiple times, and we want to ensure same mask is used every time. Also the caches are created without tracking. Since they are not picklable by python when deepcopy, we don't want `layer._obj_reference_counts_dict` to track it by default. N)rContextValueCache_create_dropout_mask_dropout_mask_cache_create_recurrent_dropout_mask_recurrent_dropout_mask_cache)r&r*r*r+r[s z4DropoutRNNCellMixin._create_non_trackable_mask_cachecCs|jdS)aReset the cached dropout masks if any. This is important for the RNN layer to invoke this in it `call()` method so that the cached mask is cleared before calling the `cell.call()`. The mask should be cached across the timestep within the same batch, but shouldn't be cached between batches. Otherwise it will introduce unreasonable bias against certain index of data within the batch. N)rclear)r&r*r*r+rrs z&DropoutRNNCellMixin.reset_dropout_maskcCs|jdS)aReset the cached recurrent dropout masks if any. This is important for the RNN layer to invoke this in it call() method so that the cached mask is cleared before calling the cell.call(). The mask should be cached across the timestep within the same batch, but shouldn't be cached between batches. Otherwise it will introduce unreasonable bias against certain index of data within the batch. N)rr)r&r*r*r+r}s z0DropoutRNNCellMixin.reset_recurrent_dropout_maskr<cCstt||j||dS)N)r=count)_generate_dropout_maskr ones_likedropout)r&r5r=rr*r*r+rs z(DropoutRNNCellMixin._create_dropout_maskcCstt||j||dS)N)r=r)rrrrecurrent_dropout)r&r5r=rr*r*r+rs z2DropoutRNNCellMixin._create_recurrent_dropout_maskcCs*|jdkrdSt|||d}|jj|dS)awGet the dropout mask for RNN cell's input. It will create mask based on context if there isn't any existing cached mask. If a new mask is generated, it will update the cache in the cell. Args: inputs: The input tensor whose shape will be used to generate dropout mask. training: Boolean tensor, whether its in training mode, dropout will be ignored in non-training mode. count: Int, how many dropout mask will be generated. It is useful for cell that has internal weights fused together. Returns: List of mask tensor, generated or cached mask based on context. rN)r5r=r)r')rrVr setdefault)r&r5r=r init_kwargsr*r*r+get_dropout_mask_for_cells z-DropoutRNNCellMixin.get_dropout_mask_for_cellcCs*|jdkrdSt|||d}|jj|dS)ayGet the recurrent dropout mask for RNN cell. It will create mask based on context if there isn't any existing cached mask. If a new mask is generated, it will update the cache in the cell. Args: inputs: The input tensor whose shape will be used to generate dropout mask. training: Boolean tensor, whether its in training mode, dropout will be ignored in non-training mode. count: Int, how many dropout mask will be generated. It is useful for cell that has internal weights fused together. Returns: List of mask tensor, generated or cached mask based on context. rN)r5r=r)r')rrVrr)r&r5r=rrr*r*r+#get_recurrent_dropout_mask_for_cells z7DropoutRNNCellMixin.get_recurrent_dropout_mask_for_cellcs*tt|}|dd|dd|S)Nrr)r$r __getstate__r!)r&r})r)r*r+rs  z DropoutRNNCellMixin.__getstate__cs4t|j|d<t|j|d<tt||dS)Nrr)rrrrr$r __setstate__)r&r})r)r*r+rs   z DropoutRNNCellMixin.__setstate__)r<)r<)r<)r<)r`rarbrcr%rrrrrrrrrrrrgr*r*)r)r+rHs        rzkeras.layers.SimpleRNNCellc sPeZdZdZdfd d Zejd d Zdd dZdddZ fddZ Z S) SimpleRNNCella: Cell class for SimpleRNN. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. This class processes one step within the whole time sequence input, whereas `tf.keras.layer.SimpleRNN` processes the whole sequence. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, (default `True`), whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. Default: `glorot_uniform`. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. Default: `orthogonal`. bias_initializer: Initializer for the bias vector. Default: `zeros`. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. Default: `None`. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_regularizer: Regularizer function applied to the bias vector. Default: `None`. kernel_constraint: Constraint function applied to the `kernel` weights matrix. Default: `None`. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_constraint: Constraint function applied to the bias vector. Default: `None`. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. Default: 0. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Default: 0. Call arguments: inputs: A 2D tensor, with shape of `[batch, feature]`. states: A 2D tensor with shape of `[batch, units]`, which is the state from the previous time step. For timestep 0, the initial state provided by user will be feed to cell. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. Only relevant when `dropout` or `recurrent_dropout` is used. Examples: ```python inputs = np.random.random([32, 10, 8]).astype(np.float32) rnn = tf.keras.layers.RNN(tf.keras.layers.SimpleRNNCell(4)) output = rnn(inputs) # The output has shape `[32, 4]`. rnn = tf.keras.layers.RNN( tf.keras.layers.SimpleRNNCell(4), return_sequences=True, return_state=True) # whole_sequence_output has shape `[32, 10, 4]`. # final_state has shape `[32, 4]`. whole_sequence_output, final_state = rnn(inputs) ``` tanhTglorot_uniform orthogonalrNc s |dkrtd|dtr0|dd|_n|dd|_tt|jf|||_t ||_ ||_ t ||_t ||_t ||_t ||_t ||_t | |_t | |_t | |_t | |_tdtd| |_tdtd||_|j|_|j|_dS) NrzFReceived an invalid value for units, expected a positive integer, got .enable_caching_deviceTFg?g)rr#executing_eagerly_outside_functionsr!_enable_caching_devicer$rr%unitsrget activationuse_biasr kernel_initializerrecurrent_initializerbias_initializerr kernel_regularizerrecurrent_regularizerbias_regularizerrkernel_constraintrecurrent_constraintbias_constraintminmaxrrrr1)r&rrrrrrrrrrrrrrr')r)r*r+r%s,          zSimpleRNNCell.__init__cCst|}|j|d|jfd|j|j|j|d|_|j|j|jfd|j|j|j |d|_ |j r|j|jfd|j |j |j|d|_nd|_d|_dS)Nr/kernel)rrN initializer regularizer constraintcaching_devicerecurrent_kernelbiasT)_caching_device add_weightrrrrrrrrrrrrrrrL)r&rRdefault_caching_devicer*r*r+rOCs2     zSimpleRNNCell.buildc Cst|r|dn|}|||}|||}|dk rJt|||j}nt||j}|jdk rpt||j}|dk r||}|t||j }|j dk r| |}t|r|gn|} || fS)Nr) rrBrrrdotrrbias_addrr) r&r5rGr=Z prev_outputdp_mask rec_dp_maskhr new_stater*r*r+r`s     zSimpleRNNCell.callcCst||||S)N)r9)r&r5r6r7r*r*r+r4vszSimpleRNNCell.get_initial_statecs|jt|j|jt|jt|jt|jt |j t |j t |j t |jt |jt |j|j|jd}|t|tt|}tt|t|S)N)rrrrrrrrrrrrrr)rr serializerrr rrrr rrrrrrrrrupdate!_config_for_enable_caching_devicer$rrUrVrKrW)r&rXrY)r)r*r+rUys"           zSimpleRNNCell.get_config) rTrrrNNNNNNrr)N)NNN) r`rarbrcr%rrerOrr4rUrgr*r*)r)r+rs$C   rzkeras.layers.SimpleRNNcseZdZdZd.fd d Zd/fd d ZeddZeddZeddZ eddZ eddZ eddZ eddZ eddZeddZed d!Zed"d#Zed$d%Zed&d'Zed(d)Zfd*d+Zed,d-ZZS)0 SimpleRNNaFully-connected RNN where the output is to be fed back to input. See [the Keras RNN API guide](https://www.tensorflow.org/guide/keras/rnn) for details about the usage of RNN API. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass None, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, (default `True`), whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. Default: `glorot_uniform`. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. Default: `orthogonal`. bias_initializer: Initializer for the bias vector. Default: `zeros`. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. Default: `None`. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_regularizer: Regularizer function applied to the bias vector. Default: `None`. activity_regularizer: Regularizer function applied to the output of the layer (its "activation"). Default: `None`. kernel_constraint: Constraint function applied to the `kernel` weights matrix. Default: `None`. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. Default: `None`. bias_constraint: Constraint function applied to the bias vector. Default: `None`. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. Default: 0. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Default: 0. return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence. Default: `False`. return_state: Boolean. Whether to return the last state in addition to the output. Default: `False` go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence. stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. Call arguments: inputs: A 3D tensor, with shape `[batch, timesteps, feature]`. mask: Binary tensor of shape `[batch, timesteps]` indicating whether a given timestep should be masked. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. This argument is passed to the cell when calling it. This is only relevant if `dropout` or `recurrent_dropout` is used. initial_state: List of initial state tensors to be passed to the first call of the cell. Examples: ```python inputs = np.random.random([32, 10, 8]).astype(np.float32) simple_rnn = tf.keras.layers.SimpleRNN(4) output = simple_rnn(inputs) # The output has shape `[32, 4]`. simple_rnn = tf.keras.layers.SimpleRNN( 4, return_sequences=True, return_state=True) # whole_sequence_output has shape `[32, 10, 4]`. # final_state has shape `[32, 4]`. whole_sequence_output, final_state = simple_rnn(inputs) ``` rTrrrNFc sd|kr|dtdd|kr4d|di}ni}t|f|||||||| | | | |||d|ddd|}tt|j|f|||||d|t| |_ t d d g|_ dS) NimplementationzhThe `implementation` argument in `SimpleRNN` has been deprecated. Please remove it from your layer call.rr7 trainableT)rrrrrrrrrrrrrr7r)rlrmrnrorp)ndim) r!r"r#rrr$rr%r activity_regularizerr rr)r&rrrrrrrrrrrrrrrrlrmrnrorpr' cell_kwargsr()r)r*r+r%sF    zSimpleRNN.__init__cstt|j||||dS)N)rr=r)r$rr)r&r5rr=r)r)r*r+r2s zSimpleRNN.callcCs|jjS)N)r(r)r&r*r*r+r6szSimpleRNN.unitscCs|jjS)N)r(r)r&r*r*r+r:szSimpleRNN.activationcCs|jjS)N)r(r)r&r*r*r+r>szSimpleRNN.use_biascCs|jjS)N)r(r)r&r*r*r+rBszSimpleRNN.kernel_initializercCs|jjS)N)r(r)r&r*r*r+rFszSimpleRNN.recurrent_initializercCs|jjS)N)r(r)r&r*r*r+rJszSimpleRNN.bias_initializercCs|jjS)N)r(r)r&r*r*r+rNszSimpleRNN.kernel_regularizercCs|jjS)N)r(r)r&r*r*r+rRszSimpleRNN.recurrent_regularizercCs|jjS)N)r(r)r&r*r*r+rVszSimpleRNN.bias_regularizercCs|jjS)N)r(r)r&r*r*r+rZszSimpleRNN.kernel_constraintcCs|jjS)N)r(r)r&r*r*r+r^szSimpleRNN.recurrent_constraintcCs|jjS)N)r(r)r&r*r*r+rbszSimpleRNN.bias_constraintcCs|jjS)N)r(r)r&r*r*r+rfszSimpleRNN.dropoutcCs|jjS)N)r(r)r&r*r*r+rjszSimpleRNN.recurrent_dropoutcs|jt|j|jt|jt|jt|jt |j t |j t |j t |j t|jt|jt|j|j|jd}tt|}|t|j|d=tt|t|S)N)rrrrrrrrrrrrrrrr()rrrrrr rrrr rrrrrrrrrrr$rrUrrr(rVrKrW)r&rXrY)r)r*r+rUns&            zSimpleRNN.get_configcCsd|kr|d|f|S)Nr)r!)r]rXr*r*r+r_s zSimpleRNN.from_config)rTrrrNNNNNNNrrFFFFF)NNN)r`rarbrcr%rrdrrrrrrrrrrrrrrrUrfr_rgr*r*)r)r+rsJV)               &rzkeras.layers.GRUCell)Zv1csPeZdZdZdfd d Zejd dZdddZfddZ dddZ Z S)GRUCellazCell class for the GRU layer. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass None, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: hard sigmoid (`hard_sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. bias_initializer: Initializer for the bias vector. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. bias_regularizer: Regularizer function applied to the bias vector. kernel_constraint: Constraint function applied to the `kernel` weights matrix. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. bias_constraint: Constraint function applied to the bias vector. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. reset_after: GRU convention (whether to apply reset gate after or before matrix multiplication). False = "before" (default), True = "after" (CuDNN compatible). Call arguments: inputs: A 2D tensor. states: List of state tensors corresponding to the previous timestep. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. Only relevant when `dropout` or `recurrent_dropout` is used. r hard_sigmoidTrrrNFc sX|dkrtd|dtr0|dd|_n|dd|_tt|jf|||_t ||_ t ||_ ||_ t ||_t ||_t ||_t ||_t | |_t | |_t | |_t | |_t | |_tdtd||_tdtd||_|d d }|jdkr8|d kr8tt d |_!n||_!||_"|j|_#|j|_$dS) NrzFReceived an invalid value for units, expected a positive integer, got rrTFg?grr<)%rrrr!rr$rr%rrrrrecurrent_activationrr rrrr rrrrrrrrrrrr"debugRECURRENT_DROPOUT_WARNING_MSGr reset_afterrr1)r&rrrrrrrrrrrrrrrrr'r)r)r*r+r%s:             zGRUCell.__init__cCs|d}t|}|j||jdfd|j|j|j|d|_|j|j|jdfd|j|j|j |d|_ |j r|j s|d|jf}ndd|jf}|j|d|j |j|j|d|_nd|_d|_dS) Nr/rr)rrNrrrrrrrT)rrrrrrrrrrrrrrrrrrL)r&rRrjrZ bias_shaper*r*r+rOs8    z GRUCell.buildcCsdt|r|dn|}|j||dd}|j||dd}|jr`|jsP|jd}}nt|j\}}|j dkrd|j krdkrnn&||d} ||d} ||d} n |} |} |} t | |j ddd|jf} t | |j dd|j|jdf} t | |j dd|jddf}|jrvt | |d|j} t | ||j|jd} t |||jdd}d|jkrdkrnn&||d}||d}||d}n |}|}|}t ||jddd|jf}t ||jdd|j|jdf}|jrN|jrNt ||d|j}t |||j|jd}|| |}|| |}|jrt ||jdd|jddf}|jrt |||jdd}||}n(t |||jdd|jddf}|||}n6d|j krdkr&nn ||d}t ||j }|jrHt ||}tj|ddd \} } }|jrt ||j}|jrt ||}n$t ||jdddd|jf}tj||j|jdgdd \}}}|| |}|| |}|jr||}n(t |||jddd|jdf}|||}||d||}t|rX|gn|}||fS) Nrr)rr<gg?rr/)axis)rrBrrrrrrZunstackrrrrrrrrrrrsplit)r&r5rGr=h_tm1rrZ input_biasZrecurrent_biasZinputs_zZinputs_rZinputs_hZx_zZx_rZx_hZh_tm1_zZh_tm1_rZh_tm1_hZ recurrent_zZ recurrent_rzrZ recurrent_hhhZmatrix_xZ matrix_innerrrr*r*r+r+s     &$   "  $    $z GRUCell.callcs|jt|jt|j|jt|jt|jt|j t |j t |j t |j t|jt|jt|j|j|j|j|jd}|t|tt|}tt|t|S)N)rrrrrrrrrrrrrrrrr)rrrrrrr rrrr rrrrrrrrrrrrrr$rrUrVrKrW)r&rXrY)r)r*r+rUs(            zGRUCell.get_configcCst||||S)N)r9)r&r5r6r7r*r*r+r4szGRUCell.get_initial_state)rrTrrrNNNNNNrrF)N)NNN) r`rarbrcr%rrerOrrUr4rgr*r*)r)r+rs(/)& i rzkeras.layers.GRUcseZdZdZd5fd d Zd6fd d ZeddZeddZeddZ eddZ eddZ eddZ eddZ eddZedd Zed!d"Zed#d$Zed%d&Zed'd(Zed)d*Zed+d,Zed-d.Zed/d0Zfd1d2Zed3d4ZZS)7GRUaUGated Recurrent Unit - Cho et al. 2014. There are two variants. The default one is based on 1406.1078v3 and has reset gate applied to hidden state before matrix multiplication. The other one is based on original 1406.1078v1 and has the order reversed. The second variant is compatible with CuDNNGRU (GPU-only) and allows inference on CPU. Thus it has separate biases for `kernel` and `recurrent_kernel`. Use `'reset_after'=True` and `recurrent_activation='sigmoid'`. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: hard sigmoid (`hard_sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. bias_initializer: Initializer for the bias vector. kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. bias_regularizer: Regularizer function applied to the bias vector. activity_regularizer: Regularizer function applied to the output of the layer (its "activation").. kernel_constraint: Constraint function applied to the `kernel` weights matrix. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. bias_constraint: Constraint function applied to the bias vector. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence. return_state: Boolean. Whether to return the last state in addition to the output. go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence. stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. time_major: The shape format of the `inputs` and `outputs` tensors. If True, the inputs and outputs will be in shape `(timesteps, batch, ...)`, whereas in the False case, it will be `(batch, timesteps, ...)`. Using `time_major = True` is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. However, most TensorFlow data is batch-major, so by default this function accepts input and emits output in batch-major form. reset_after: GRU convention (whether to apply reset gate after or before matrix multiplication). False = "before" (default), True = "after" (CuDNN compatible). Call arguments: inputs: A 3D tensor. mask: Binary tensor of shape `(samples, timesteps)` indicating whether a given timestep should be masked. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. This argument is passed to the cell when calling it. This is only relevant if `dropout` or `recurrent_dropout` is used. initial_state: List of initial state tensors to be passed to the first call of the cell. rrTrrrNFc s|dd}|dkrtdd|kr6d|di}ni}t|f|||||||| | | | ||||||d|ddd |}tt|j|f|||||d |t| |_ t d d g|_ dS) Nrr<rzm`implementation=0` has been deprecated, and now defaults to `implementation=1`.Please update your layer call.rr7rT)rrrrrrrrrrrrrrrrr7r)rlrmrnrorpr)r) r!r"r#rrr$r r%r rr rr)r&rrrrrrrrrrrrrrrrrlrmrnrorprr'rrr()r)r*r+r%sL    z GRU.__init__cstt|j||||dS)N)rr=r)r$r r)r&r5rr=r)r)r*r+rPs zGRU.callcCs|jjS)N)r(r)r&r*r*r+rTsz GRU.unitscCs|jjS)N)r(r)r&r*r*r+rXszGRU.activationcCs|jjS)N)r(r)r&r*r*r+r\szGRU.recurrent_activationcCs|jjS)N)r(r)r&r*r*r+r`sz GRU.use_biascCs|jjS)N)r(r)r&r*r*r+rdszGRU.kernel_initializercCs|jjS)N)r(r)r&r*r*r+rhszGRU.recurrent_initializercCs|jjS)N)r(r)r&r*r*r+rlszGRU.bias_initializercCs|jjS)N)r(r)r&r*r*r+rpszGRU.kernel_regularizercCs|jjS)N)r(r)r&r*r*r+rtszGRU.recurrent_regularizercCs|jjS)N)r(r)r&r*r*r+rxszGRU.bias_regularizercCs|jjS)N)r(r)r&r*r*r+r|szGRU.kernel_constraintcCs|jjS)N)r(r)r&r*r*r+rszGRU.recurrent_constraintcCs|jjS)N)r(r)r&r*r*r+rszGRU.bias_constraintcCs|jjS)N)r(r)r&r*r*r+rsz GRU.dropoutcCs|jjS)N)r(r)r&r*r*r+rszGRU.recurrent_dropoutcCs|jjS)N)r(r)r&r*r*r+rszGRU.implementationcCs|jjS)N)r(r)r&r*r*r+rszGRU.reset_aftercs|jt|jt|j|jt|jt|jt|j t |j t |j t |j t |jt|jt|jt|j|j|j|j|jd}|t|jtt|}|d=tt|t|S)N)rrrrrrrrrrrrrrrrrrr() rrrrrrr rrrr rrrrrrrrrrrrrrr(r$r rUrVrKrW)r&rXrY)r)r*r+rUs,             zGRU.get_configcCs&d|kr|ddkrd|d<|f|S)Nrrr<r*)r]rXr*r*r+r_szGRU.from_config)rrTrrrNNNNNNNrrFFFFFF)NNN)r`rarbrcr%rrdrrrrrrrrrrrrrrrrrrUrfr_rgr*r*)r)r+r sTX,                  ,r zkeras.layers.LSTMCellcs`eZdZdZdfd d Zejd d ZddZddZ dddZ fddZ dddZ Z S)LSTMCellaCell class for the LSTM layer. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: hard sigmoid (`hard_sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. bias_initializer: Initializer for the bias vector. unit_forget_bias: Boolean. If True, add 1 to the bias of the forget gate at initialization. Setting it to true will also force `bias_initializer="zeros"`. This is recommended in [Jozefowicz et al., 2015]( http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf) kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. bias_regularizer: Regularizer function applied to the bias vector. kernel_constraint: Constraint function applied to the `kernel` weights matrix. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. bias_constraint: Constraint function applied to the bias vector. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. Call arguments: inputs: A 2D tensor. states: List of state tensors corresponding to the previous timestep. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. Only relevant when `dropout` or `recurrent_dropout` is used. rrTrrrNc s^|dkrtd|dtr0|dd|_n|dd|_tt|jf|||_t ||_ t ||_ ||_ t ||_t ||_t ||_||_t | |_t | |_t | |_t | |_t | |_t ||_tdtd||_tdtd||_|d d }|jdkr>|d kr>t t!d |_"n||_"|j|jg|_#|j|_$dS) NrzFReceived an invalid value for units, expected a positive integer, got rrTFg?grr<)%rrrr!rr$rr%rrrrrrr rrrunit_forget_biasr rrrrrrrrrrrr"rrrrr1)r&rrrrrrrrrrrrrrrrr'r)r)r*r+r%s:             zLSTMCell.__init__cst}|d}j|jdfdjjj|d_jjjdfdjjj |d_ j rj r|fdd}nj }jjdfd|jj|d_nd_d _dS) Nr/r)rrNrrrrrcsNtjjff||tdjff||jjdff||gS)Nonesr)rZ concatenaterrr r)ryrr')r&r*r+rM sz(LSTMCell.build..bias_initializerrT)rrrrrrrrrrrrrrrrrrL)r&rRrrjrr*)r&r+rO7 s:     zLSTMCell.buildc Cs|\}}}}|\}} } } ||t||jddd|jf} ||t| |jdd|j|jdf} | || ||t| |jdd|jd|jdf}||t| |jdd|jddf}||fS)z.Computes carry and output using split kernels.Nrr)rrrrrr)r&xr c_tm1x_ix_fx_cx_oh_tm1_ih_tm1_fh_tm1_ch_tm1_orfr-or*r*r+_compute_carry_and_output` s  & &.*z"LSTMCell._compute_carry_and_outputc CsH|\}}}}||}||}|||||} ||} | | fS)z.Computes carry and output using fused kernels.)rr) r&r rz0z1z2z3rrr-rr*r*r+_compute_carry_and_output_fusedn s     z(LSTMCell._compute_carry_and_output_fusedc!CsZ|d}|d}|j||dd}|j||dd}|jdkrd|jkrRdkrnn2||d}||d} ||d} ||d} n|}|} |} |} tj|jddd\} } }}t|| }t| | }t| |}t| |}|j r4tj|j ddd\}}}}t ||}t ||}t ||}t ||}d|j krNdkrnn2||d}||d}||d}||d}n|}|}|}|}||||f}||||f}| |||\}}nd |jkrdkrnn ||d}t||j}|t||j7}|j r t ||j }tj|ddd}|||\}}|||} | | |gfS) Nrr<r)rg?rr)Znum_or_size_splitsrg)rrrrrrrrrrrrrr rr%r)!r&r5rGr=r rrrZinputs_iZinputs_fZinputs_cZinputs_oZk_iZk_fZk_cZk_orrrrZb_iZb_fZb_cZb_orrrrrr-rr rr*r*r+rw sd                   z LSTMCell.callcs|jt|jt|j|jt|jt|jt|j |j t |j t |j t |jt|jt|jt|j|j|j|jd}|t|tt|}tt|t|S)N)rrrrrrrrrrrrrrrrr)rrrrrrr rrrrr rrrrrrrrrrrrr$rrUrVrKrW)r&rXrY)r)r*r+rU s(            zLSTMCell.get_configcCstt||||S)N)rKr9)r&r5r6r7r*r*r+r4 szLSTMCell.get_initial_state)rrTrrrTNNNNNNrr)N)NNN)r`rarbrcr%rrerOr r%rrUr4rgr*r*)r)r+rs,2() < )rz#keras.experimental.PeepholeLSTMCellcs>eZdZdZdfd d Zfd d ZddZddZZS)PeepholeLSTMCella8Equivalent to LSTMCell class but adds peephole connections. Peephole connections allow the gates to utilize the previous internal state as well as the previous hidden state (which is what LSTMCell is limited to). This allows PeepholeLSTMCell to better learn precise timings over LSTMCell. From [Gers et al., 2002]( http://www.jmlr.org/papers/volume3/gers02a/gers02a.pdf): "We find that LSTM augmented by 'peephole connections' from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars." The peephole implementation is based on: [Sak et al., 2014](https://research.google.com/pubs/archive/43905.pdf) Example: ```python # Create 2 PeepholeLSTMCells peephole_lstm_cells = [PeepholeLSTMCell(size) for size in [128, 256]] # Create a layer composed sequentially of the peephole LSTM cells. layer = RNN(peephole_lstm_cells) input = keras.Input((timesteps, input_dim)) output = layer(input) ``` rrTrrrNc sPtdtt|jf||||||||| | | | | ||||ddd|dS)Nz`tf.keras.experimental.PeepholeLSTMCell` is deprecated and will be removed in a future version. Please use tensorflow_addons.rnn.PeepholeLSTMCell instead.rr<)rrrrrrrrrrrrrrrrr)warningswarnr$r&r%r!)r&rrrrrrrrrrrrrrrrr')r)r*r+r% s(  zPeepholeLSTMCell.__init__cs\tt|||j|jfd|jd|_|j|jfd|jd|_|j|jfd|jd|_dS)Ninput_gate_peephole_weights)rrNrforget_gate_peephole_weightsoutput_gate_peephole_weights) r$r&rOrrrr*r+r,)r&rR)r)r*r+rO+ s  zPeepholeLSTMCell.buildc Cs|\}}}}|\}} } } ||t||jddd|jf|j|} ||t| |jdd|j|jdf|j|} | || ||t| |jdd|jd|jdf}||t| |jdd|jddf|j|}||fS)Nrr) rrrrrr*r+rr,)r&rr rrrrrrrrrrrr-rr*r*r+r = s  "(.&z*PeepholeLSTMCell._compute_carry_and_outputc Csf|\}}}}|||j|}|||j|}|||||} |||j| } | | fS)N)rr*r+rr,) r&r rr!r"r#r$rrr-rr*r*r+r%M s z0PeepholeLSTMCell._compute_carry_and_output_fused)rrTrrrTNNNNNNr'r') r`rarbrcr%rOr r%rgr*r*)r)r+r& s& r&zkeras.layers.LSTMcseZdZdZd5fd d Zd6fd d ZeddZeddZeddZ eddZ eddZ eddZ eddZ eddZedd Zed!d"Zed#d$Zed%d&Zed'd(Zed)d*Zed+d,Zed-d.Zed/d0Zfd1d2Zed3d4ZZS)7LSTMaLong Short-Term Memory layer - Hochreiter 1997. Note that this cell is not optimized for performance on GPU. Please use `tf.compat.v1.keras.layers.CuDNNLSTM` for better performance on GPU. Args: units: Positive integer, dimensionality of the output space. activation: Activation function to use. Default: hyperbolic tangent (`tanh`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). recurrent_activation: Activation function to use for the recurrent step. Default: hard sigmoid (`hard_sigmoid`). If you pass `None`, no activation is applied (ie. "linear" activation: `a(x) = x`). use_bias: Boolean, whether the layer uses a bias vector. kernel_initializer: Initializer for the `kernel` weights matrix, used for the linear transformation of the inputs.. recurrent_initializer: Initializer for the `recurrent_kernel` weights matrix, used for the linear transformation of the recurrent state. bias_initializer: Initializer for the bias vector. unit_forget_bias: Boolean. If True, add 1 to the bias of the forget gate at initialization. Setting it to true will also force `bias_initializer="zeros"`. This is recommended in [Jozefowicz et al., 2015]( http://www.jmlr.org/proceedings/papers/v37/jozefowicz15.pdf). kernel_regularizer: Regularizer function applied to the `kernel` weights matrix. recurrent_regularizer: Regularizer function applied to the `recurrent_kernel` weights matrix. bias_regularizer: Regularizer function applied to the bias vector. activity_regularizer: Regularizer function applied to the output of the layer (its "activation"). kernel_constraint: Constraint function applied to the `kernel` weights matrix. recurrent_constraint: Constraint function applied to the `recurrent_kernel` weights matrix. bias_constraint: Constraint function applied to the bias vector. dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state. return_sequences: Boolean. Whether to return the last output. in the output sequence, or the full sequence. return_state: Boolean. Whether to return the last state in addition to the output. go_backwards: Boolean (default False). If True, process the input sequence backwards and return the reversed sequence. stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch. unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. time_major: The shape format of the `inputs` and `outputs` tensors. If True, the inputs and outputs will be in shape `(timesteps, batch, ...)`, whereas in the False case, it will be `(batch, timesteps, ...)`. Using `time_major = True` is a bit more efficient because it avoids transposes at the beginning and end of the RNN calculation. However, most TensorFlow data is batch-major, so by default this function accepts input and emits output in batch-major form. Call arguments: inputs: A 3D tensor. mask: Binary tensor of shape `(samples, timesteps)` indicating whether a given timestep should be masked. An individual `True` entry indicates that the corresponding timestep should be utilized, while a `False` entry indicates that the corresponding timestep should be ignored. training: Python boolean indicating whether the layer should behave in training mode or in inference mode. This argument is passed to the cell when calling it. This is only relevant if `dropout` or `recurrent_dropout` is used. initial_state: List of initial state tensors to be passed to the first call of the cell. rrTrrrNFc s|dd}|dkrtdd|kr6d|di}ni}t|f|||||||| | | | ||||||d|ddd |}tt|j|f|||||d |t| |_ t d d g|_ dS) Nrr<rzm`implementation=0` has been deprecated, and now defaults to `implementation=1`.Please update your layer call.rr7rT)rrrrrrrrrrrrrrrrr7r)rlrmrnrorpr)r) r!r"r#rrr$r-r%r rr rr)r&rrrrrrrrrrrrrrrrrrlrmrnrorpr'rrr()r)r*r+r% sL    z LSTM.__init__cstt|j||||dS)N)rr=r)r$r-r)r&r5rr=r)r)r*r+r s z LSTM.callcCs|jjS)N)r(r)r&r*r*r+r sz LSTM.unitscCs|jjS)N)r(r)r&r*r*r+r szLSTM.activationcCs|jjS)N)r(r)r&r*r*r+r szLSTM.recurrent_activationcCs|jjS)N)r(r)r&r*r*r+r sz LSTM.use_biascCs|jjS)N)r(r)r&r*r*r+r szLSTM.kernel_initializercCs|jjS)N)r(r)r&r*r*r+r szLSTM.recurrent_initializercCs|jjS)N)r(r)r&r*r*r+r szLSTM.bias_initializercCs|jjS)N)r(r)r&r*r*r+r szLSTM.unit_forget_biascCs|jjS)N)r(r)r&r*r*r+r szLSTM.kernel_regularizercCs|jjS)N)r(r)r&r*r*r+r szLSTM.recurrent_regularizercCs|jjS)N)r(r)r&r*r*r+r szLSTM.bias_regularizercCs|jjS)N)r(r)r&r*r*r+r! szLSTM.kernel_constraintcCs|jjS)N)r(r)r&r*r*r+r% szLSTM.recurrent_constraintcCs|jjS)N)r(r)r&r*r*r+r) szLSTM.bias_constraintcCs|jjS)N)r(r)r&r*r*r+r- sz LSTM.dropoutcCs|jjS)N)r(r)r&r*r*r+r1 szLSTM.recurrent_dropoutcCs|jjS)N)r(r)r&r*r*r+r5 szLSTM.implementationcs|jt|jt|j|jt|jt|jt|j |j t |j t |j t |jt |jt|jt|jt|j|j|j|jd}|t|jtt|}|d=tt|t|S)N)rrrrrrrrrrrrrrrrrrr() rrrrrrr rrrrr rrrrrrrrrrrrrr(r$r-rUrVrKrW)r&rXrY)r)r*r+rU9 s,             zLSTM.get_configcCs&d|kr|ddkrd|d<|f|S)Nrrr<r*)r]rXr*r*r+r_e szLSTM.from_config)rrTrrrTNNNNNNNr.r.FFFFF)NNN)r`rarbrcr%rrdrrrrrrrrrrrrrrrrrrUrfr_rgr*r*)r)r+r-X sTU,                  ,r-r<cs@fdd|dkr0fddt|DStjdS)Ncs tS)N)rrr*)rrater*r+dropped_inputsm sz._generate_dropout_mask..dropped_inputsr<csg|]}tjdqS))r=)rin_train_phase)r,ry)r0rr=r*r+rr sz*_generate_dropout_mask..)r=)rangerr1)rr/r=rr*)r0rr/r=r+rl s  rcCst|tr|dkr|dkst|r>|| d}|d| }t|dkrb|dd}|dd}t|dkrxt|}n|d}dd}||}||}|||fS)aStandardizes `__call__` to a single list of tensor inputs. When running a model loaded from a file, the input tensors `initial_state` and `constants` can be passed to `RNN.__call__()` as part of `inputs` instead of by the dedicated keyword arguments. This method makes sure the arguments are separated and that `initial_state` and `constants` are lists of tensors (or None). Args: inputs: Tensor or list/tuple of tensors. which may include constants and initial states. In that case `num_constant` must be specified. initial_state: Tensor or list of tensors or None, initial states. constants: Tensor or list of tensors or None, constant tensors. num_constants: Expected number of constants (if constants are passed as part of the `inputs` list. Returns: inputs: Single tensor or tuple of tensors. initial_state: List of tensors or None. constants: List of tensors or None. Nr<rcSs.|dkst|tr|St|tr(t|S|gS)N)rJrKr0)rr*r*r+to_list_or_none s  z*_standardize_args..to_list_or_none)rJrKAssertionErrorrCr0)r5rr>rr3r*r*r+rx s      rcCst|dot|tj S)z6Check whether the state_size contains multiple states.__len__)hasattrrJrrP)rr*r*r+r3 s r3cCs*|dk rt|d}|j}t||j|S)Nr)rrr7rr)r(r5r6r7r*r*r+r9 sr9csPdksdkr tdfdd}t|rDt||S||SdS)zBGenerate a zero filled tensor with shape [batch_size, state_size].Nz]batch_size and dtype cannot be None while constructing initial state: batch_size={}, dtype={}cs&t|}g|}tj|dS)N)r7)rrPrQrr)Zunnested_state_sizeZ flat_dimsZinit_state_size)batch_size_tensorr7r*r+ create_zeros s z1_generate_zero_filled_state..create_zeros)rrrrBr|)r7rr7r8r*)r7r7r+r s   rcCs^tr dSt|ddsdSttr8tddS|j j |j j krVtddSddS)aReturns the caching device for the RNN variable. This is useful for distributed training, when variable is not located as same device as the training worker. By enabling the device cache, this allows worker to read the variable once and cache locally, rather than read it every time step from remote when it is needed. Note that this is assuming the variable that cell needs for each time step is having the same value in the forward path, and only gets updated in the backprop. It is true for all the default cells (SimpleRNN, GRU, LSTM). If the cell body relies on any variable that gets updated every time step, then caching device will cause it to read the stall value. Args: rnn_cell: the rnn cell instance. NrFa$Variable read device caching has been disabled because the RNN is in tf.while_loop loop context, which will cause reading stalled value in forward path. This could slow down the training due to duplicated variable reads. Please consider updating your code to remove tf.while_loop if possible.a\Variable read device caching has been disabled since it doesn't work with the mixed precision API. This is likely to cause a slowdown for RNN training due to duplicated read of variable for each timestep, which will be significant in a multi remote worker setting. Please consider disabling mixed precision API if the performance has been affected.cSs|jS)N)Zdevice)opr*r*r+rz r{z!_caching_device..) rZexecuting_eagerlyr2rZ IsInWhileLooprZget_default_graphr"r#Z _dtype_policyZ compute_dtypeZvariable_dtype)rnn_cellr*r*r+r s  rcCs t}|j|krd|jiSiS)a]Return the dict config for RNN cell wrt to enable_caching_device field. Since enable_caching_device is a internal implementation detail for speed up the RNN variable read when running on the multi remote worker setting, we don't want this config to be serialized constantly in the JSON. We will only serialize this field when a none default value is used to create the cell. Args: rnn_cell: the RNN cell for serialize. Returns: A dict which contains the JSON config for enable_caching_device value or empty dict if the enable_caching_device value is same as the default value. r)rrr)r:Zdefault_enable_caching_devicer*r*r+r s  r)Nr<)Crcrr(numpyrZtensorflow.python.distributerrwZtensorflow.python.eagerrZtensorflow.python.frameworkrrZtensorflow.python.kerasrrrr r Z)tensorflow.python.keras.engine.base_layerr Z)tensorflow.python.keras.engine.input_specr Z*tensorflow.python.keras.saving.saved_modelr Ztensorflow.python.keras.utilsrrrZtensorflow.python.opsrrrrZtensorflow.python.platformrr"Ztensorflow.python.trackablerrZtensorflow.python.utilrZ tensorflow.python.util.tf_exportrZtensorflow.tools.docsrrrrhrZdo_not_generate_docsobjectrrrrr rr&r-rrr3r9rrrr*r*r*r+s                        -Z M~  v =1