B dN@sdZddlZddlZddlZddlZddlZddlmm Z ddl m Z ddl mZddlmZedd#ddZd$d d Zd d Zd%ddZd&ddZd'ddZddZddZddZGdddZedd(dd Zd!d"ZdS))z/Utilities related to layer/model functionality.N) initializers)io_utils) keras_exportzkeras.utils.get_source_inputscst|ds|S|dks|r&|j\}}}|js2|gS|j|}|jrPtj|jSg}xR|D]F\}}}}t |||}x,|D]$t fdd|Dr|| q|Wq^W|SdS)aReturns the list of input tensors necessary to compute `tensor`. Output will always be a list of tensors (potentially with 1 element). Args: tensor: The tensor to start from. layer: Origin layer of the tensor. Will be determined via tensor._keras_history if not provided. node_index: Origin node index of the tensor. Returns: List of input tensors. _keras_historyNc3s|]}|k VqdS)N).0t)xrI/var/www/html/venv/lib/python3.7/site-packages/keras/utils/layer_utils.py Csz$get_source_inputs..) hasattrr_inbound_nodesZis_inputtfnestflattenZ input_tensorsiterate_inboundget_source_inputsallappend)Ztensorlayer node_index_nodeZsource_tensorsZprevious_sourcesr)r r r!s       rFc Cs|r|dkrdS|r t|r dSt|tr6||kr6dS|r>dnd}||rLdnd7}|d|7}|rrd|d}nd}td|d |d |d |d | dS) z0Validates the correctness of a string-based arg.Nz`None`, za `Callable`, z or one of the following values: zIf restoring a model and `z` is a custom callable, please ensure the callable is registered as a custom object. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details. zUnkown value for `z` argument of layer z. zAllowed values are: z . Received: )callable isinstancestr ValueError)Z input_dataZallowable_stringsZ layer_namearg_name allow_noneZallow_callablesZ allowed_argsZ callable_noterrr validate_string_argHs   r cCsRdd|D}dd|D}dd|D}dd|D}ttdd|DS) zCount the total number of scalars composing the weights. Args: weights: An iterable containing the weights on which to compute params Returns: The total number of scalars composing the weights cSsi|]}|t|qSr)id)rwrrr tsz count_params..cSsg|]}t|dr|qS)shape)r )rr"rrr vsz count_params..cSsg|]}|jqSr)r$as_list)rr"rrr r%wscSsg|]}dd|DqS)cSsg|]}|dkrdn|qS)Nrr)rZw_irrr r%ysz+count_params...r)rr"rrr r%yscss|]}t|VqdS)N)npprod)rprrr r {szcount_params..)valuesintsum)weightsZunique_weightsZ weight_shapesZstandardized_weight_shapesrrr count_paramsks r.csdk rXtdkr,tddttdtrHtdtsftdndt|jgSfdd t|jD}fd d t|jD}|r|std t|t|krt|t|dgSt|t|dgS) a3Get the layer indexes from the model based on layer names. The layer indexes can be used to slice the model into sub models for display. Args: model: `tf.keras.Model` instance. layer_names: a list or tuple of 2 strings, the starting layer name and ending layer name (both inclusive) for the result. All layers will be included when `None` is provided. Returns: The index value of layer based on its unique name (layer_names). Output will be [first_layer_index, last_layer_index + 1]. NzIlayer_range must be a list or tuple of length 2. Received: layer_range = z of length rz7layer_range should contain string type only. Received: cs&g|]\}}td|jr|qS)r)rematchname)ridxr) layer_rangerr r%sz7get_layer_index_bound_by_layer_name..cs&g|]\}}td|jr|qS)r0)r1r2r3)rr4r)r5rr r%szHPassed layer_names do not match the layer names in the model. Received: )lenrrrlayers enumerateminmax)modelr5Z lower_indexZ upper_indexr)r5r #get_layer_index_bound_by_layer_name~s&     r<csdkrtj|jjdkr d n|js,d nd |j}g}xL|D]D} t| dkszt| dkrttj | dj dkrd P|| 7}qDW rx@|j D]6} d} x&| j D]} | |kr| rd Pqd} qW sPqW rpdpdd d gd dkrfd d Ddddg} njp"dp4dddd gd dkrVfdd Dddddg} gx|jD]} | 7qrW rd7| dt||}d*fdd d|jdd| dd+ fdd d, fd!d" d- fd#d$ x(|j |d|dD]} | qJWdt|d%rt|j}n t|j}t|j}d&||d'd(|d'd)|d'ddS).aPrints a summary of a model. Args: model: Keras model instance. line_length: Total length of printed lines (e.g. set this to adapt the display to different terminal window sizes). positions: Relative or absolute positions of log elements in each line. If not provided, defaults to `[.33, .55, .67, 1.]`. print_fn: Print function to use. It will be called on each line of the summary. You can set it to a custom function in order to capture the string summary. It defaults to `print` (prints to stdout). expand_nested: Whether to expand the nested models. If not provided, defaults to `False`. show_trainable: Whether to show if a layer is trainable. If not provided, defaults to `False`. layer_range: List or tuple containing two strings, the starting layer name and ending layer name (both inclusive), indicating the range of layers to be printed in the summary. The strings could also be regexes instead of an exact name. In this case, the starting layer will be the first layer that matches `layer_range[0]` and the ending layer will be the last element that matches `layer_range[1]`. By default (`None`) all layers in the model are included in the summary. NZ SequentialTr0rFAg?g333333?g?csg|]}t|qSr)r+)rr)) line_lengthrr r%sz!print_summary..z Layer (type)z Output ShapezParam #bgQ?g?gq= ףp?csg|]}t|qSr)r+)rr))r?rr r%sz Connected to Z Trainablec s^dd|D}xHt|rXd}xtt|D] }|dkrL||d}nd}||}|t|dkrldnd}|||} ||d| d} fdd| D} | rt| } d| |dkr|d |d 7}|7}||rd |nd7}||| d||<|r&|d ||t||7}q0|d ||t|7}q0W|d |7}|qWdS) NcSsg|] }t|qSr)r)rr rrr r%sz4print_summary..print_row..rrr0r/)z),z},z],z',cs,g|]$}|dkr|t|qS)r)findr6)rr ) fit_into_linerr r%)s| )anyranger6r9) fields positions nested_levelZ left_to_printlinecol start_posend_posspacecutoffZline_break_conditionsZcandidate_cutoffs)print_fn)rCr print_rows6    z print_summary..print_rowzModel: ""r=csy |j}Wn.tk r"d}Yntk r8d}YnX|j}|jj}|js`t|dds`d}n|}|d|d||g}r| |j rdnd ||d S) zPrints a summary for a single layer. Args: layer: target layer. nested_level: level of nesting of the layer inside its parent layer (e.g. 0 for a top-level layer, 1 for a nested layer). multiple?_is_graph_networkFz 0 (unused)z ()YNN) output_shapeAttributeError RuntimeErrorr3 __class____name__Zbuiltgetattrr.r trainable)rrJr[r3cls_nameparamsrH)rIrRshow_trainablerr print_layer_summaryDs  z*print_summary..print_layer_summaryc sy |j}Wntk r"d}YnXg}xT|jD]J}rB|krBq0x6|D]*\}}}}||jd|d|dqLWq0W|j} |jj} | d| d|||g} r| |j rdnd| |d S) aPrints a summary for a single layer (including topological connections). Args: layer: target layer. nested_level: level of nesting of the layer inside its parent layer (e.g. 0 for a top-level layer, 1 for a nested layer). rU[z][]z (rXrYrZN) r[r\r rrr3r^r_r.ra) rrJr[ connectionsrZ inbound_layerrZ tensor_indexrr3rbrH)rIrRrelevant_nodesrdrr $print_layer_summary_with_connectionsbs*    "z;print_summary..print_layer_summary_with_connectionscsr||n ||rt|dr|jrʈd|ddd|dd|d|j}d}x.print_layer_collected_trainable_weightszTotal params: ,zTrainable params: zNon-trainable params: )r)r)r)rF)rZ print_msgr^r_rWZ_nodes_by_depthr*r6rrrZ keras_inputsr7r rr<r3r r.rpZtrainable_weightsZnon_trainable_weights)r;r?rIrQrmrdr5Znodes_by_depthZnodesvrflagr to_displayZtrainable_countZnon_trainable_countr) rmr?rIrQrnrerjrRrirordr print_summarys$    &         )   *!     ruchannels_firstc Cs|dks t|\}}xt|jdD]}|dkrl|\}}}|||f} |dd|f| } t| d} n6|\}}}|||f} |dd|f| } t| d} t| t|f|dd|f<q(W|||gdS)a]Utility useful when changing a convnet's `data_format`. When porting the weights of a convnet from one data format to the other, if the convnet includes a `Flatten` layer (applied to the last convolutional feature map) followed by a `Dense` layer, the weights of that `Dense` layer should be updated to reflect the new dimension ordering. Args: dense: The target `Dense` layer. previous_feature_map_shape: A shape tuple of 3 integers, e.g. `(512, 7, 7)`. The shape of the convolutional feature map right before the `Flatten` layer that came before the target `Dense` layer. target_data_format: One of "channels_last", "channels_first". Set it "channels_last" if converting a "channels_first" model to "channels_last", or reciprocally. >rv channels_lastr0rvN)r/rr0)r0r/r) AssertionErrorZ get_weightsrGr$Zreshaper'Z transposer(Z set_weights) ZdenseZprevious_feature_map_shapeZtarget_data_formatZkernelZbiasrlchr"Zoriginal_fm_shapeZkirrr !convert_dense_weights_data_formats       $r{cCs$t|ddsdS|jdko"|jdkS)N_keras_api_namesF)zkeras.layers.Layer)r`r|Z_keras_api_names_v1)rrrr is_builtin_layers  r}cs*ttfdd}|_|S)a Lightweight decorator for caching lazily constructed properties. When to use: This decorator provides simple caching with minimal overhead. It is designed for properties which are expensive to compute and static over the life of a class instance, and provides no mechanism for cache invalidation. Thus it is best suited for lazily exposing derived properties of other static data. For classes with custom getattr / setattr behavior (such as trackable objects), storing cache results as object attributes is not performant. Instead, a specialized cache can significantly reduce property lookup overhead. (While still allowing the decorated property to be lazily computed.) Consider the following class: ``` class MyClass: def __setattr__(self, key, value): # Some expensive class specific code # ... # ... super(MyClass, self).__setattr__(key, value) @property def thing(self): # `thing` is expensive to compute (and may not even be requested), so we # want to lazily compute it and then cache it. output = getattr(self, '_thing', None) if output is None: self._thing = output = compute_thing(self) return output ``` It's also worth noting that ANY overriding of __setattr__, even something as simple as: ``` def __setattr__(self, key, value): super(MyClass, self).__setattr__(key, value) ``` Slows down attribute assignment by nearly 10x. By contrast, replacing the definition of `thing` with the following sidesteps the expensive __setattr__ altogether: ''' @property @tracking.cached_per_instance def thing(self): # `thing` is expensive to compute (and may not even be requested), so we # want to lazily compute it and then cache it. return compute_thing(self) ''' Performance: The overhead for this decorator is ~0.4 us / call. A much lower overhead implementation (~0.085 us / call) can be achieved by using a custom dict type: ``` def dict_based_cache(f): class Cache(dict): __slots__ = () def __missing__(self, key): self[key] = output = f(key) return output return property(Cache().__getitem__) ``` However, that implementation holds class instances as keys, and as a result blocks garbage collection. (And modifying it to use weakref's as keys raises the lookup overhead to ~0.4 us) As a result, the WeakKeyDictionary implementation below turns out to be more prudent. Args: f: The function to cache. Returns: f decorated with simple caching behavior. cs&|}|dkr"||<}|S)N)get)itemoutput)cachefrr wrappedEs z$cached_per_instance..wrapped)weakrefWeakKeyDictionary functoolswrapsr)rrr)rrr cached_per_instancesSrccst}|ddd}xl|r|}t||kr0q|t|t|drZt|tsZ|Vqt|ddphg}||dddqWdS)z4Filter out empty Layer-like containers and uniquify.Nr>Z _is_layerr7) setpopr!addr rtyper`extend)Z layer_listexistingZto_visitobjZ sub_layersrrr filter_empty_layer_containersPs rc@seZdZdZddZeddZeddZejddZee d d Z ed d Z e jd d Z eddZ e jddZ eddZ dddZdddZdddZddZdS) CallFunctionSpeczKCaches the spec and provides utilities for handling call function args.cCs||_t|jj|_|jr8|jddkr8|jdd|_|j|jjpHg7_|jjdk }d|jkpf||_d|jkpv||_|jjpg}t }x,t dt |dD]}||||j|<qW| |jj pi|d|_dS)zInitialies a `CallFunctionSpec`. Args: full_argspec: the FullArgSpec of a call function of a layer. rselfr0NZtrainingmaskr>) _full_argspeclistargs _arg_names kwonlyargsvarkw_expects_training_arg_expects_mask_argdefaultsdictrGr6updatekwonlydefaultsr~_default_training_arg)r full_argspecZcall_accepts_kwargsZcall_fn_defaultsrrlrrr __init__hs  zCallFunctionSpec.__init__cCs|jS)z-Returns the FullArgSpec of the call function.)r)rrrr rszCallFunctionSpec.full_argspeccCs|jS)z%List of names of args and kwonlyargs.)r)rrrr arg_namesszCallFunctionSpec.arg_namescCs ||_dS)N)r)rvaluerrr rscCs*t}xt|jD]\}}|||<qW|S)z:Returns a dict mapping arg names to their index positions.)rr8r)rZcall_fn_arg_positionsposargrrr arg_positionss zCallFunctionSpec.arg_positionscCs|jS)z9Whether the call function uses 'training' as a parameter.)r)rrrr expects_training_argsz%CallFunctionSpec.expects_training_argcCs ||_dS)N)r)rrrrr rscCs|jS)z5Whether the call function uses `mask` as a parameter.)r)rrrr expects_mask_argsz!CallFunctionSpec.expects_mask_argcCs ||_dS)N)r)rrrrr rscCs|jS)z3The default value given to the "training" argument.)r)rrrr default_training_argsz%CallFunctionSpec.default_training_argFcCs@|s |s dS||krdS|j}|s.|dd}|tt||kS)aReturns true if argument is present in `args` or `kwargs`. Args: arg_name: String name of the argument to find. args: Tuple of args passed to the call function. kwargs: Dictionary of kwargs passed to the call function. inputs_in_args: Whether the input argument (the first argument in the call function) is included in `args`. Defaults to `False`. Returns: True if argument with `arg_name` is present in `args` or `kwargs`. FTr0N)rrzip)rrrkwargsinputs_in_args call_fn_argsrrr arg_was_passeds zCallFunctionSpec.arg_was_passedcCs<||kr||S|j}|s&|dd}tt||}||S)akRetrieves the value for the argument with name `arg_name`. Args: arg_name: String name of the argument to find. args: Tuple of args passed to the call function. kwargs: Dictionary of kwargs passed to the call function. inputs_in_args: Whether the input argument (the first argument in the call function) is included in `args`. Defaults to `False`. Returns: The value of the argument with name `arg_name`, extracted from `args` or `kwargs`. Raises: KeyError if the value of `arg_name` cannot be found. r0N)rrr)rrrrrrZ args_dictrrr get_arg_values zCallFunctionSpec.get_arg_valuecCs|jjrNy*|jj|}|jjddkr0|d8}Wq\tk rJd}Yq\Xn|j|d}|dk r|sp|d}t||krt|}|||<t ||fS|dkr|r| |dn|||<||fS)aSets the value of an argument into the given args/kwargs. Args: arg_name: String name of the argument to find. new_value: New value to give to the argument. args: Tuple of args passed to the call function. kwargs: Dictionary of kwargs passed to the call function. inputs_in_args: Whether the input argument (the first argument in the call function) is included in `args`. Defaults to `False`. pop_kwarg_if_none: If the new value is `None`, and this is `True`, then the argument is deleted from `kwargs`. Returns: The updated `(args, kwargs)`. rrr0N) rvarargsrindexrrr~r6rtupler)rr new_valuerrrZpop_kwarg_if_noneZarg_posrrr set_arg_values&     zCallFunctionSpec.set_arg_valuecCsV|r|d}|dd}n2|jd|krDt|}||jd}ntd|||fS)z2Splits (args, kwargs) into (inputs, args, kwargs).rr0Nz9The first argument to `Layer.call` must always be passed.)rcopyrr)rrrinputsrrr split_out_first_args z$CallFunctionSpec.split_out_first_argN)F)F)FF)r_ __module__ __qualname____doc__rpropertyrrsetterrrrrrrrrrrrrr rds"!        )rz&keras.utils.warmstart_embedding_matrixuniformc Cst|}t|}t|}|t||jdf|jd}tt|tt|}g}g}x8t |D],\}} | |kr^| || | t |q^W|rt ||} t |t j|dd| }|S)aY Warm start embedding matrix with changing vocab. This util can be used to warmstart the embedding layer matrix when vocabulary changes between previously saved checkpoint and model. Vocabulary change could mean, the size of the new vocab is different or the vocabulary is reshuffled or new vocabulary has been added to old vocabulary. If the vocabulary size changes, size of the embedding layer matrix also changes. This util remaps the old vocabulary embeddings to the new embedding layer matrix. Example: Here is an example that demonstrates how to use the `warmstart_embedding_matrix` util. >>> import keras >>> vocab_base = tf.convert_to_tensor(["unk", "a", "b", "c"]) >>> vocab_new = tf.convert_to_tensor( ... ["unk", "unk", "a", "b", "c", "d", "e"]) >>> vectorized_vocab_base = np.random.rand(vocab_base.shape[0], 3) >>> vectorized_vocab_new = np.random.rand(vocab_new.shape[0], 3) >>> warmstarted_embedding_matrix = warmstart_embedding_matrix( ... base_vocabulary=vocab_base, ... new_vocabulary=vocab_new, ... base_embeddings=vectorized_vocab_base, ... new_embeddings_initializer=keras.initializers.Constant( ... vectorized_vocab_new)) Here is an example that demonstrates how to get vocabulary and embedding weights from layers, use the `warmstart_embedding_matrix` util to remap the layer embeddings and continue with model training. ``` # get old and new vocabulary by using layer.get_vocabulary() # for example assume TextVectorization layer is used base_vocabulary = old_text_vectorization_layer.get_vocabulary() new_vocabulary = new_text_vectorization_layer.get_vocabulary() # get previous embedding layer weights embedding_weights_base = model.get_layer('embedding').get_weights()[0] warmstarted_embedding = keras.utils.warmstart_embedding_matrix( base_vocabulary, new_vocabulary, base_embeddings=embedding_weights_base, new_embeddings_initializer="uniform") updated_embedding_variable = tf.Variable(warmstarted_embedding) # update embedding layer weights model.layers[1].embeddings = updated_embedding_variable model.fit(..) # continue with model training ``` Args: base_vocabulary: The list of vocabulary terms that the preexisting embedding matrix `base_embeddings` represents. It can be either a 1D array/tensor or a tuple/list of vocabulary terms (strings), or a path to a vocabulary text file. If passing a file path, the file should contain one line per term in the vocabulary. new_vocabulary: The list of vocabulary terms for the new vocabulary (same format as above). base_embeddings: NumPy array or tensor representing the preexisting embedding matrix. new_embeddings_initializer: Initializer for embedding vectors for previously unseen terms to be added to the new embedding matrix (see `keras.initializers`). Defaults to "uniform". new_embedding matrix needs to be specified with "constant" initializer. matrix. Default value is None. Returns: tf.tensor of remapped embedding layer matrix r0)r$dtype)Zaxis)convert_vocab_to_listrr~r6r$rrrrGr8rr+rgatherZtensor_scatter_nd_updateZ expand_dims) Zbase_vocabularyZnew_vocabularyZbase_embeddingsZnew_embeddings_initializerZ new_embeddingZbase_vocabulary_dictZindices_base_vocabularyZindices_new_vocabularyrkeyZvalues_to_updaterrr warmstart_embedding_matrix/s*O    rc Csg}t|rt|}nt|tjttfr8t|}nft|trtj j |s`t d|dtj j |d}|}WdQRXnt dt|dt|dkrt d|S) z!Convert input vacabulary to list.zVocabulary file z does not exist.rNzkVocabulary is expected to be either a NumPy array, list, 1D tensor or a vocabulary text file. Instead type z was received.rzVocabulary is expected to be either a NumPy array, list, 1D tensor or a vocabulary text file with at least one token. Received 0 instead.)rZ is_tensorrnumpyrr'ZndarrayrrioZgfileexistsrZGFileread splitlinesrr6)ZvocabZ vocab_listZvocabulary_filerrr rs     r)NN)FF)N)NNNFFN)rv)r)rrrr1rrr'Ztensorflow.compat.v2compatZv2rZkerasrZ keras.utilsrZ tensorflow.python.util.tf_exportrrr r.r<rur{r}rrrrrrrrr s@    +  8  & `L k