B ӻd] @s"dZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddl mZddl mZddl mZdd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlm Z!ddlm"Z#ddlm$Z%ddlm&Z'ddl(m)Z)ddl(m*Z*ddl+m,Z,ddZ-ddZ.dfddZ/ej0dgd d!Z1e2Z3de3_4de3_5de3_6de3_7e)j8d"d#Z9e)j8d$d%Z:d&d'Z;e)j8d(d)Zd.d/Z?dhd0d1Z@d2d3ZAGd4d5d5ejBZCGd6d7d7ejBZDd8d9ZEd:d;ZFdd?d?ejBZHGd@dAdAejBZIdidBdCZJGdDdEdEejKZLGdFdGdGejBZMGdHdIdIejBZNdjdJdKZOejPejQejRe!jSe%jTe'jUe#jVdLZWdMdNZXdkdPdQZYdRdSZZdTdUZ[dVdWZ\ej8dXdYZ]ej8dZd[Z^d\d]Z_d^d_Z`d`daZadldbdcZbdddeZcdS)mz!Utilities for unit-testing Keras.N)tf2)context)config)dtypes)ops) tensor_shape) tensor_spec) test_util)backend)layers)models)base_layer_utils)adadelta)adagrad)adam)adamax)gradient_descent)nadam)rmsprop) tf_contextlib) tf_inspect) tf_decoratorcCstj||dS)N)nptestingZassert_array_equal)actualexpectedrW/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/testing_utils.py string_test1srcCstjj||ddddS)NgMbP?gư>)ZrtolZatol)rrZassert_allclose)rrrrr numeric_test5src Cs|dk rtj|||}d|tj|f|}tjjd||fd}tj|f|tjd}x2t|D]&} ||| tjjdd|d|| <qjW|d||d|f||d||dffS)aGenerates test data to train a model on. Args: train_samples: Integer, how many training samples to generate. test_samples: Integer, how many test samples to generate. input_shape: Tuple of integers, shape of the inputs. num_classes: Integer, number of classes for the data and targets. random_seed: Integer, random seed used by numpy to generate data. Returns: A tuple of Numpy arrays: `(x_train, y_train), (x_test, y_test)`. Nr)size)dtypeg?)locscaler!)rrandomseedrandintzerosfloat32rangenormal) Z train_samplesZ test_samples input_shape num_classesZ random_seedZ num_sampleZ templatesyxirrr get_test_data9s &r1Tc !sB|dkr|dkrtd|s d}t|} x.t| D]"\}}|dkr2tjdd| |<q2Wdtj| }|dddkr|d 8}||}n|dkr|j}|dkr|j}|dkr|}t |t j kr| r| j }qt }n| r| j}nt}pif}| dk r&|j| kr&td j|j| f| dk r:|| |}||d tjkrp|d <f}tj|dd|d |}t||krtd jt||ffdd}|dk r|t||jt|}t |!t|"}|#t$j%||d }|&|}|j}|||||j||j|jkrptdj|j|jf|dk r||||'}tj(|| }|j)r|}|||&|}||||}|r(t|}t*j+dk r |j,dddgt-dn|j,dddgd|.|||'}||d<|j/(|}| dk rX|| t0}|1tj|dd|d |1||||&|}|j}xDt2||D]6\}} |dk r|| krtdj||fqW|dk r||||'}tj0(|| }|j)r>|}|||&|}||||S)aTest routine for a layer with a single input and single output. Args: layer_cls: Layer class object. kwargs: Optional dictionary of keyword arguments for instantiating the layer. input_shape: Input shape tuple. input_dtype: Data type of the input data. input_data: Numpy array of input data. expected_output: Numpy array of the expected output. expected_output_dtype: Data type expected for the output. expected_output_shape: Shape tuple for the expected shape of the output. validate_training: Whether to attempt to validate training on this layer. This might be set to False for non-differentiable layers that output string or integer values. adapt_data: Optional data for an 'adapt' call. If None, adapt() will not be tested for this layer. This is only relevant for PreprocessingLayers. custom_objects: Optional dictionary mapping name strings to custom objects in the layer class. This is helpful for testing custom layers. test_harness: The Tensorflow test, if any, that this function is being called in. supports_masking: Optional boolean to check the `supports_masking` property of the layer. If None, the check will not be performed. Returns: The output data (Numpy array) returned by the layer, for additional checks to be done by the calling code. Raises: ValueError: if `input_shape is None`. Nzinput_shape is Noner) floatg?zbWhen testing layer %s, the `supports_masking` property is %rbut expected to be %r. Full kwargs: %sweights)shaper"zcWhen testing layer %s, for input %s, found output dtype=%s but expected to find %s. Full kwargs: %scst|t|kr(tdj||fxbt||D]T\}}t|tjrN|j}t|tjr`|j}|dk r4||kr4tdj||fq4WdS)zFAsserts that the output shape from the layer matches the actual shape.zcWhen testing layer %s, for input %s, found output_shape=%s but expected to find %s. Full kwargs: %sN)lenAssertionError__name__zip isinstancerZ Dimensionvalue)rr expected_dim actual_dim)kwargs layer_clsr/rrassert_shapes_equals  z'layer_test..assert_shapes_equalzcWhen testing layer %s, for input %s, found output_dtype=%s but expected to find %s. Full kwargs: %srZmseacc)weighted_metrics run_eagerly)rEZbatch_input_shapezWhen testing layer %s **after deserialization**, for input %s, found output_shape=%s but expected to find inferred shape %s. Full kwargs: %s)3 ValueErrorlist enumeraterr%r'Zastyper8r"rZas_dtypestringZassertAllEqualrZassertAllClosersupports_maskingr:r;ZadaptZ get_weightsZ set_weightsr getargspec__init__r Inputr rZ TensorShaper ModeltupleZcompute_output_shapeas_listZcompute_output_signaturerZ TensorSpecZpredictZ get_config from_configr7_thread_local_datarFcompileshould_run_eagerlyZtrain_on_batch __class__ Sequentialaddr<)!rBrAr, input_dtypeZ input_dataZexpected_outputZexpected_output_dtypeZexpected_output_shapeZvalidate_trainingZ adapt_dataZcustom_objectsZ test_harnessrKZinput_data_shaper0eZ assert_equallayerr7r.rCmodelZcomputed_output_shapeZcomputed_output_signatureZ actual_outputZactual_output_shapeZ model_configZrecovered_modeloutputZ layer_weightsZ layer_configr?r@r)rArBr/r layer_testVs-                                     r^ccs$tj}z|t_|VWd|t_XdS)zProvides a scope within which the model type to test is equal to `value`. The model type gets restored to its original value upon exiting the scope. Args: value: model type value Yields: The provided value. N)rS model_type)r>previous_valuerrrmodel_type_scope;s  raccs$tj}z|t_|VWd|t_XdS)a.Provides a scope within which we compile models to run eagerly or not. The boolean gets restored to its original value upon exiting the scope. Args: value: Bool specifying if we should run models eagerly in the active test. Should be True or False. Yields: The provided value. N)rSrF)r>r`rrrrun_eagerly_scopePs  rbcCs tjdkrtdtjotS)z@Returns whether the models we are testing should be run eagerly.NziCannot call `should_run_eagerly()` outside of a `run_eagerly_scope()` or `run_all_keras_modes` decorator.)rSrFrGrexecuting_eagerlyrrrrrUfs rUcks6tj}tj}z|t_|t_dVWd|t_|t_XdS)a1Provides a scope within which the savde model format to test is `value`. The saved model format gets restored to its original value upon exiting the scope. Args: value: saved model format value **kwargs: optional kwargs to pass to the save function. Yields: The provided value. N)rSsaved_model_format save_kwargs)r>rAZprevious_formatZprevious_kwargsrrrsaved_model_format_scopeps rfcCstjdkrtdtjS)NzzCannot call `get_save_format()` outside of a `saved_model_format_scope()` or `run_with_all_saved_model_formats` decorator.)rSrdrGrrrrget_save_formats rgcCstjdkrtdtjpiS)NzzCannot call `get_save_kwargs()` outside of a `saved_model_format_scope()` or `run_with_all_saved_model_formats` decorator.)rSrerGrrrrget_save_kwargss rhcCstjdkrtdtjS)z*Gets the model type that should be tested.NziCannot call `get_model_type()` outside of a `model_type_scope()` or `run_with_all_model_types` decorator.)rSr_rGrrrrget_model_types ricCs`t}|r$|tj|d|dn|tj|dd|dkrDdnd}|tj||d|S)Nrelu) activation input_dim)rkr2sigmoidsoftmax)r rWrXr Dense) num_hiddenr-rlr\rkrrrget_small_sequential_mlpsrqcCsNtj|fd}tj|dd|}|dkr,dnd}tj||d|}t||S)N)r8rj)rkr2rmrn)r rNror rO)rpr-rlinputsoutputsrkrrrget_small_functional_mlps rtcs*eZdZdZdfdd ZddZZS)SmallSubclassMLPz!A subclass model based small MLP.Fc stt|jfddi|||_||_tj|dd|_|dkrBdnd}tj||d|_|jrht d|_ |jr|tj d d |_ dS) NnameZ test_modelrj)rkr2rmrng?)Zaxis) superrurMuse_bnuse_dpr rolayer_alayer_bZDropoutdpZBatchNormalizationbn)selfrpr-ryrzrArk)rVrrrMs zSmallSubclassMLP.__init__cKs4||}|jr||}|jr*||}||S)N)r{rzr}ryr~r|)rrrrAr/rrrcalls    zSmallSubclassMLP.call)FF)r; __module__ __qualname____doc__rMr __classcell__rr)rVrrusrucs0eZdZdZfddZddZddZZS)_SmallSubclassMLPCustomBuildz;A subclass model small MLP that uses a custom build method.cs*tt|d|_d|_||_||_dS)N)rxrrMr{r|rpr-)rrpr-)rVrrrMs z%_SmallSubclassMLPCustomBuild.__init__cCs:tj|jdd|_|jdkr dnd}tj|j|d|_dS)Nrj)rkr2rmrn)r rorpr{r-r|)rr,rkrrrbuildsz"_SmallSubclassMLPCustomBuild.buildcKs||}||S)N)r{r|)rrrrAr/rrrrs z!_SmallSubclassMLPCustomBuild.call)r;rrrrMrrrrr)rVrrs rcCs t||S)N)ru)rpr-rrrget_small_subclass_mlpsrcCs t||S)N)r)rpr-rrr(get_small_subclass_mlp_with_custom_buildsrcCsdt}|dkrt||S|dkr*t||S|dkr>t|||S|dkrRt|||Std|dS)z@Get a small mlp of the model type specified by `get_model_type`.subclasssubclass_custom_build sequential functionalzUnknown model type {}N)rirrrqrtrGformat)rpr-rlr_rrr get_small_mlps    rcs0eZdZdZfddZddZddZZS)_SubclassModelzA Keras subclass model.csf|dd}tt|j||x&t|D]\}}t||||q(Wt||_|dk rb| |dS)zInstantiate a model. Args: model_layers: a list of layers to be added to the model. *args: Model's args **kwargs: Model's keyword args, at most one of input_tensor -> the input tensor required for ragged/sparse input. input_tensorN) poprxrrMrIsetattr_layer_name_for_ir9 num_layersZ _set_inputs)r model_layersargsrArrr0r[)rVrrrMs  z_SubclassModel.__init__cCs d|S)Nzlayer{})r)rr0rrrrsz _SubclassModel._layer_name_for_icKs4|}x*t|jD]}t|||}||}qW|S)N)r*rgetattrr)rrrrAr/r0r[rrrrs  z_SubclassModel.call)r;rrrrMrrrrr)rVrrs rcs0eZdZdZfddZddZddZZS)_SubclassModelCustomBuildz7A Keras subclass model that uses a custom build method.cs"tt|j||d|_||_dS)N)rxrrM all_layers_layer_generating_func)rlayer_generating_funcrrA)rVrrrM(sz"_SubclassModelCustomBuild.__init__cCs*g}x|D]}||qW||_dS)N)rappendr)rr,rr[rrrr-sz_SubclassModelCustomBuild.buildcKs |}x|jD] }||}q W|S)N)r)rrrrAr/r[rrrr3s  z_SubclassModelCustomBuild.call)r;rrrrMrrrrr)rVrr%s rc s|dkrt}|dkrBd}|s"|r4tj||||d}t||dS|dkrbfdd}t||dS|d krtj|d} |r| tj||||d xD]} | | qW| S|d kr|st d tj||||d}|} xD] } | | } qWtj || |dSt d |dS)a-Builds a model from a sequence of layers. Args: model_layers: The layers used to build the network. input_shape: Shape tuple of the input or 'TensorShape' instance. input_dtype: Datatype of the input. name: Name for the model. input_ragged: Boolean, whether the input data is a ragged tensor. input_sparse: Boolean, whether the input data is a sparse tensor. model_type: One of "subclass", "subclass_custom_build", "sequential", or "functional". When None, defaults to `get_model_type`. Returns: A Keras model. Nr)r8r"raggedsparse)rvrrcsS)Nrr)rrr]z'get_model_from_layers..)rvr)r,r"rrrzACannot create a functional model from layers with no input shape.zUnknown model type {}) rir rNrrr rWrXZ InputLayerrGrOr) rr,rYrvZ input_raggedZ input_sparser_rrrr\r[rsr)rrget_model_from_layers:sN       rc@seZdZddZddZdS)BiascCs|jdddd|_dS)Nbias)r2r()Z initializer)Z add_variabler)rr,rrrrsz Bias.buildcCs ||jS)N)r)rrrrrrrsz Bias.callN)r;rrrrrrrrr~srcs*eZdZdZdfdd ZddZZS)_MultiIOSubclassModelzMulti IO Keras subclass model.Ncs.tt|j|d||_||_||_||_dS)N)rv)rxrrM_shared_input_branch _branch_a _branch_b_shared_output_branch)rbranch_abranch_bshared_input_branchshared_output_branchrv)rVrrrMs z_MultiIOSubclassModel.__init__cKs|jr(x|jD] }||}qW|}|}n$t|trD|d}|d}n|\}}x|jD] }||}qTWx|jD] }||}qlW||g}|jrx|jD] }||}qW|S)NZinput_1Zinput_2)rr=dictrrr)rrrrAr[aboutsrrrrs$          z_MultiIOSubclassModel.call)NNN)r;rrrrMrrrr)rVrrsrcs2eZdZdZd fdd ZddZddZZS) _MultiIOSubclassModelCustomBuildz>Multi IO Keras subclass model that uses a custom build method.NcsBtt|||_||_||_||_d|_d|_d|_ d|_ dS)N) rxrrM_shared_input_branch_func_branch_a_func_branch_b_func_shared_output_branch_funcrrrr)rZ branch_a_funcZ branch_b_funcZshared_input_branch_funcZshared_output_branch_func)rVrrrMsz)_MultiIOSubclassModelCustomBuild.__init__cCs<|r||_||_||_|r8||_dS)N)rrrrrrrr)rr,rrrrs    z&_MultiIOSubclassModelCustomBuild.buildcKs|jr(x|jD] }||}qW|}|}n|\}}x|jD] }||}q8Wx|jD] }||}qPW||f}|jrx|jD] }||}qvW|S)N)rrrr)rrrrAr[rrrrrrrs        z%_MultiIOSubclassModelCustomBuild.call)NN)r;rrrrMrrrrr)rVrrs   rc sNrd}ddn(ddf}ddddt}|dkr^tS|dkrtfddfddfd dfd dS|d krtd |d kr<r|}xD] }||}qW|}|} n|\}} xD] }||}qWxD] }|| } qW|| f} r0xD]}|| } qWt|| Std|dS)a Builds a multi-io model that contains two branches. The produced model will be of the type specified by `get_model_type`. To build a two-input, two-output model: Specify a list of layers for branch a and branch b, but do not specify any shared input branch or shared output branch. The resulting model will apply each branch to a different input, to produce two outputs. The first value in branch_a must be the Keras 'Input' layer for branch a, and the first value in branch_b must be the Keras 'Input' layer for branch b. example usage: ``` branch_a = [Input(shape=(2,), name='a'), Dense(), Dense()] branch_b = [Input(shape=(3,), name='b'), Dense(), Dense()] model = get_multi_io_model(branch_a, branch_b) ``` To build a two-input, one-output model: Specify a list of layers for branch a and branch b, and specify a shared output branch. The resulting model will apply each branch to a different input. It will then apply the shared output branch to a tuple containing the intermediate outputs of each branch, to produce a single output. The first layer in the shared_output_branch must be able to merge a tuple of two tensors. The first value in branch_a must be the Keras 'Input' layer for branch a, and the first value in branch_b must be the Keras 'Input' layer for branch b. example usage: ``` input_branch_a = [Input(shape=(2,), name='a'), Dense(), Dense()] input_branch_b = [Input(shape=(3,), name='b'), Dense(), Dense()] shared_output_branch = [Concatenate(), Dense(), Dense()] model = get_multi_io_model(input_branch_a, input_branch_b, shared_output_branch=shared_output_branch) ``` To build a one-input, two-output model: Specify a list of layers for branch a and branch b, and specify a shared input branch. The resulting model will take one input, and apply the shared input branch to it. It will then respectively apply each branch to that intermediate result in parallel, to produce two outputs. The first value in the shared_input_branch must be the Keras 'Input' layer for the whole model. Branch a and branch b should not contain any Input layers. example usage: ``` shared_input_branch = [Input(shape=(2,), name='in'), Dense(), Dense()] output_branch_a = [Dense(), Dense()] output_branch_b = [Dense(), Dense()] model = get_multi_io_model(output__branch_a, output_branch_b, shared_input_branch=shared_input_branch) ``` Args: branch_a: A sequence of layers for branch a of the model. branch_b: A sequence of layers for branch b of the model. shared_input_branch: An optional sequence of layers to apply to a single input, before applying both branches to that intermediate result. If set, the model will take only one input instead of two. Defaults to None. shared_output_branch: An optional sequence of layers to merge the intermediate results produced by branch a and branch b. If set, the model will produce only one output instead of two. Defaults to None. Returns: A multi-io model of the type specified by `get_model_type`, specified by the different branches. rr2NrrcsS)Nrr)rrrr<rz$get_multi_io_model..csS)Nrr)rrrr=rcsS)Nrr)rrrr>rcsS)Nrr)rrrr?rrz>Cannot use `get_multi_io_model` to construct sequential modelsrzUnknown model type {})rirrrGr rOr) rrrrrrr_Za_and_br[rrrsr)rrrrrget_multi_io_modelsFS              r)rrrrrrZsgdc Ks@yt|f|Stk r:td|ttYnXdS)aGet the v2 optimizer requested. This is only necessary until v2 are the default, as we are testing in Eager, and Eager + v1 optimizers fail tests. When we are in v2, the strings alone should be sufficient, and this mapping can theoretically be removed. Args: name: string name of Keras v2 optimizer. **kwargs: any kwargs to pass to the optimizer constructor. Returns: Initialized Keras v2 optimizer. Raises: ValueError: if an unknown name was passed. z;Could not find requested v2 optimizer: {} Valid choices: {}N)_V2_OPTIMIZER_MAPKeyErrorrGrrHkeys)rvrArrrget_v2_optimizeris rcs0tstrdd|DSfdd|DS)zEReturns expected metric variable names given names and prefix/suffix.cSsg|] }|dqS)z:0r).0nrrr sz6get_expected_metric_variable_names..csg|]}|dqS)z:0r)rr) name_suffixrrrs)renabledrrc)Z var_namesrr)rr"get_expected_metric_variable_namessrcCs t|dS)z=Decorator for enabling the layer V2 dtype behavior on a test.T)_set_v2_dtype_behavior)fnrrrenable_v2_dtype_behaviorsrcCs t|dS)z>Decorator for disabling the layer V2 dtype behavior on a test.F)r)rrrrdisable_v2_dtype_behaviorsrcs$tfdd}t|S)zCReturns version of 'fn' that runs with v2 dtype behavior on or off.cs$tj}t_z ||S|t_XdS)N)r ZV2_DTYPE_BEHAVIOR)rrAZv2_dtype_behavior)rrrrwrappers  z'_set_v2_dtype_behavior..wrapper) functoolswrapsrZmake_decorator)rrrr)rrrrs rc cs6|rtrd}nd}t| dVWdQRXdS)z&Uses gpu when requested and available.z /device:GPU:0z /device:CPU:0N)r Zis_gpu_availablerdevice)should_use_gpudevrrrrs   rc cs tdd dVWdQRXdS)z&Uses gpu when requested and available.T)rN)rrrrruse_gpus rcsfdd}|S)aGenerate class-level decorator from given method-level decorator. It is expected for the given decorator to take some arguments and return a method that is then called on the test method to produce a decorated method. Args: decorator: The decorator to apply. *args: Positional arguments **kwargs: Keyword arguments Returns: Function that will decorate a given classes test methods with the decorator. csPxJt|D]>}t||}t|r |dr |dkr t|||q W|S)z-Apply decorator to all test methods in class.testZ test_session)dirrcallable startswithr)clsrvr>)r decoratorrArrall_test_methods_impls  z3for_all_test_methods..all_test_methods_implr)rrrArr)rrrArfor_all_test_methodss rcCs dd}|S)aExecute test with TensorFloat-32 disabled. While almost every real-world deep learning model runs fine with TensorFloat-32, many tests use assertAllClose or similar methods. TensorFloat-32 matmuls typically will cause such methods to fail with the default tolerances. Args: description: A description used for documentation purposes, describing why the test requires TensorFloat-32 to be disabled. Returns: Decorator which runs a test with TensorFloat-32 disabled. cstfdd}|S)Nc s8t}ztd|f||Wdt|XdS)NF)rZ!tensor_float_32_execution_enabledZ enable_tensor_float_32_execution)rrrAallowed)frr decorateds  zArun_without_tensor_float_32..decorator..decorated)rr)rrr)rrrs z.run_without_tensor_float_32..decoratorr) descriptionrrrrrun_without_tensor_float_32s rcCs tt|S)z:Execute all tests in a class with TensorFloat-32 disabled.)rr)rrrrrun_all_without_tensor_float_32srcCsdd}|dk r||S|S)akExecute the decorated test only if running in v2 mode. This function is intended to be applied to tests that exercise v2 only functionality. If the test is run in v1 mode it will simply be skipped. See go/tf-test-decorator-cheatsheet for the decorators to use in different v1/v2/eager/graph combinations. Args: func: function to be annotated. If `func` is None, this method returns a decorator the can be applied to a function. If `func` is not None this returns the decorator applied to `func`. Returns: Returns a decorator that will conditionally skip the decorated test method. cs"trtdfdd}|S)Nz)`run_v2_only` only supports test methods.cs"ts|d|f||S)NzTest is only compatible with v2)rrZskipTest)rrrA)rrrr s z1run_v2_only..decorator..decorated)risclassrG)rrr)rrr s  zrun_v2_only..decoratorNr)funcrrrr run_v2_onlys rc sdd}g}xDt||dD]0\}t|ts6|g}|fdd|DqWddtj|D}g}x\|D]T}t|tjst d dd|D}|tt|d d |fgqpW|S) a5Generate combinations based on its keyword arguments using combine(). This function calls combine() and appends a testcase name to the list of dictionaries returned. The 'testcase_name' key is a required for named parameterized tests. Args: **kwargs: keyword arguments of form `option=[possibilities, ...]` or `option=the_only_possibility`. Returns: a list of dictionaries for each combination. Keys in the dictionaries are the keyword argument names. Each key has one value - one of the corresponding keyword argument values. cSs|dS)Nrr)krrrr+rz:generate_combinations_with_testcase_name..)keycsg|] }|fqSrr)rr>)rrrr0sz.cSsg|]}t|qSr) collections OrderedDict)rresultrrrr2src Ss<g|]4\}}ddttj|dttjt|qS)z_{}_{}r)rjoinfilterstrisalnum)rrr>rrrr8sZ testcase_namez_test{}) sorteditemsr=rHr itertoolsproductrrr:rr)rAZ sort_by_key combinationsvaluesZnamed_combinationsZ combinationrvr)rr(generate_combinations_with_testcase_names&   r)N) NNNNNNNTNNNN)N)NNNNNN)NN)r)N)drr contextlibrr threadingnumpyrZtensorflow.pythonrZtensorflow.python.eagerrZtensorflow.python.frameworkrrrrrr Ztensorflow.python.kerasr r r Ztensorflow.python.keras.enginer Z$tensorflow.python.keras.optimizer_v2rZ adadelta_v2rZ adagrad_v2rZadam_v2rZ adamax_v2rZgradient_descent_v2rZnadam_v2rZ rmsprop_v2Ztensorflow.python.keras.utilsrrZtensorflow.python.utilrrrr1Zdisable_cudnn_autotuner^localrSr_rFrdrecontextmanagerrarbrUrfrgrhrirqrtrOrurrrrrrrZLayerrrrrZAdadeltaZAdagradZAdamZAdamaxZNadamZRMSpropZSGDrrrrrrrrrrrrrrrrrs                        R    $ > $3    ! $