B һd"@sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl m Z ddl mZddl mZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddl"m#Z#e$a%dZ&e 'ej(e 'ej)e 'ej*fZ+ddZ,e-dddZ.ddZ/ddZ0dd Z1Gd!d"d"e2Z3e#d#e&d$ddfd%d&Z4e#d'd(d)Z5dS)*z>Dumping op callbacks: Enables dump-based features in tfdbg v2.N) tensor_pb2)debug_event_pb2)graph_debug_info_pb2)debug_events_writer)op_callbacks_common) source_utils)function) constant_op)dtypes) op_callbacks)ops) tensor_util) array_ops) gen_debug_ops) tf_logging)compat)tf_stack) tf_export NO_TENSORcCst|tS)N)ras_bytes startswith_FUNCTION_PREFIXES)op_typer^/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/debug/lib/dumping_callback.pyis_op_type_function7srZDebugIdentityV2cCs~|S)z-Gradient function for the DebugIdentityV2 op.r)opZdyrrr_debug_identity_v2_grad;srcCsttddS)N)struuiduuid4rrrr_get_tfdbg_run_idBsr"cCs ttS)zGet a short unique ID.)rr r!rrrr_get_idFsr#cCst|S)N)r Zmake_tensor_protonumpy)tensorrrr_concrete_tensor_to_protoKsr&c@seZdZdZddZddZeddZejddZed d Z ed d Z ed dZ ddZ ddZ ddZddZddZddZddZd*ddZd+d d!Zd"d#Zd$d%Zd&d'Zd(d)ZdS),_DumpingCallbackz>An object holding the states surrounding the dumping callback.cCs||_t|_||_||_||_||_t|_ g|_ t |_ t |_ t |_t |_d|_t |_t|_t|_t|_t|_t |_d|_dS)Nr) _dump_rootr" _tfdbg_run_id_tensor_debug_mode_circular_buffer_size _op_regex_tensor_dtypessocket gethostname _hostname_source_file_pathsdict_stack_frame_to_id_context_to_id_function_to_graph_id_op_type_to_context_id_symbolic_tensor_counter_tensor_aliases threadingLock_source_file_paths_lock_stack_frame_to_id_lock _context_lock_symbolic_tensor_counter_lock_placeholder_to_debug_tensor_writer)self dump_roottensor_debug_modecircular_buffer_sizeop_regex tensor_dtypesrrr__init__Rs(     z_DumpingCallback.__init__c Cs0~~~||}|j||j|<WdQRXdS)aA callback to be called on creation of Functions. Used to establish a join between function name and graph (context) ID. Args: function: The just-created Function. name: Name of the function. graph: FuncGraph, the graph containing the operations in the function. inputs: the tensors in the graph to be used as inputs to the function outputs: the tensors in the graph which will be outputs from the function N)_get_context_idr=r5)rArnamegraphinputsoutputsgraph_idrrrfunction_callbackzs  z"_DumpingCallback.function_callbackcCs|jS)N)r()rArrrrBsz_DumpingCallback.dump_rootcCs|j|kr||_d|_dS)N)r(r@)rArBrrrrBs cCs|jS)N)r))rArrr tfdbg_run_idsz_DumpingCallback.tfdbg_run_idcCs|jS)N)r*)rArrrrCsz"_DumpingCallback.tensor_debug_modecCs|jS)N)r+)rArrrrDsz%_DumpingCallback.circular_buffer_sizecCs$|jstj|j|j|jd|_|jS)zCGet the debug events writer for the currently configured dump root.)rD)r@rDebugEventsWriterr(r)r+)rArrr get_writers  z_DumpingCallback.get_writerc Cs~||jkr|j|Sd}|j$||jkr>d}t}||j|<WdQRX|rt|tj|t|dd||d|j|S)aGet a unique ID for an op-construction context (e.g., a graph). If the graph has been encountered before, reuse the same unique ID. When encountering a new context (graph), this methods writes a DebugEvent proto with the debugged_graph field to the proper DebugEvent file. Args: context: A context to get the unique ID for. Must be hashable. E.g., a Graph object. Returns: A unique ID for the context. FTNrI)rM graph_nameZouter_context_id) r4r=r#rQZWriteDebuggedGraphrZ DebuggedGraphgetattr_get_outer_context_id)rAcontextZ graph_is_new context_idrrrrHs     z _DumpingCallback._get_context_idcCs$t|dr|jr||jSdSdS)a:Get the ID of the immediate outer context of the input graph. Args: graph: The graph (context) in question. Returns: If an outer context exists, the immediate outer context name as a string. If such as outer context does not exist (i.e., `graph` is itself outermost), `None`. outer_graphN)hasattrrWrH)rArJrrrrTs  z&_DumpingCallback._get_outer_context_idc Cs||jkr|j|S|j||jkrd}t|rzyt|\}}Wn0tk rx}ztd||Wdd}~XYnX| }| t j ||j |d|j||j|SQRXdS)zSend the content of a source file via debug-events writer. Args: file_path: Path to the source file. Returns: An int index for the file. Nz4Failed to read source code from path: %s. Reason: %s) file_path host_namelines)r1indexr;rZ%is_extension_uncompiled_python_sourceZ load_sourceIOErrorloggingwarnrQZWriteSourceFilerZ SourceFiler0append)rArYr[_ewriterrrr_write_source_file_contents"     z+_DumpingCallback._write_source_file_contentc Cst}g}d}x|D]\}}}}tj|}|||f|jkrV||j|||fq|j~|||f|jkrt} | |j|||f<| |} t j j | ||d} t j| | d} |}|| ||j|||fWdQRXqWt j|j|d} | S)zProcess stack frames. Send the content of source-files, on a best-effort basis. Returns: A list of stack frame IDs. N) file_indexlinefunc)id file_line_col)rZstack_frame_ids)r extract_stackospathabspathr3r`r<r#rdrZGraphDebugInfoZ FileLineColrZStackFrameWithIdrQZWriteStackFrameWithIdZ CodeLocationr0)rAZ stack_framesrjrcrYlinenorgraabs_pathZstack_frame_idreriZstack_frame_with_id code_locationrrr_process_stack_framess2       z&_DumpingCallback._process_stack_framesc Cs|dkr||j|<|S|tjjkr<|dkr<|j|j|j<|S|jd|j}WdQRXtj ||d}|j |j |j|j|j<|SdS)aFor V1 graph mode, determine what tensor to output from callback. Args: op_type: Type of the op that outputs the original symbolic tensor. tensor: The original output symbolic tensor. debug_tensor: The debugger-instrumented tensor. tensor_debug_mode: Debug mode used, a tfdbg TensorDebugMode enum. Returns: A symbolic tensor to be returned by the dumping op_callback. ) PlaceholderPlaceholderWithDefaultConstztfdbg_identity_%dN)rI) r?rTensorDebugMode FULL_TENSORrIr8r>r7ridentityr_add_control_input)rArr% debug_tensorrCZ identity_namerxrrr_process_v1_graph_mode_tensors  z._DumpingCallback._process_v1_graph_mode_tensorc CsH|j}d|jg}t }|r$gnd} xt|D] \} } |jd|j} WdQRX||| |j|| |j|jd} |t j j kr| || j r| j js|r4| | q4|r| j js| | q4tjtjgtjdf| }|r| ||| ||q4|t j jt j jt j jt j jfkr| j }|t j jt j jt j jfkrB|jpd|t j jkod|jpd|jpd|j}| || j r||s|r4| | q4tjtj| || |jtjdf| }|r@| ||| ||q4|t j j kr2| || j r| j js|r4| | q4tj| f| }|r@| ||| ||q4t!d|jq4W| S)aAdd debugging instrumentation for symbolic (i.e., non-eager) tensors. The detailed fashion in which the tensors are instrumented is determined by the tensor_debug_mode configured for the currently enabled dumping callback. Args: tensors: A tuple of Tensors to instrument. It is assumed that their ordering corresponds to the ordering of output tensors of an original op. Output slot indices (0-based) will be generated based on the ordering. op_type: Type name of the op that emits the Tensors (e.g., "MatMul"). op_name: Name of the op that emits the Tensors (e.g., "dense_1/MatMul"). tfdbg_context_id: A unique ID for the context that the op belongs to (e.g., a graph). tensor_ids: A list of unique ID numbers for the tensors, for tfdbg's internal use. Returns: Non-eager Tensors that override the `tensors` as the output of the op that originally generated `tensors`. In some cases (e.g., non-V1 graph mode), this may be `None`, as the instrumentation can simply rely on automatic control dependencies (see `auto_control_deps.py`) instead of tensor overriding. z file://%sNzDebugIdentityV2_%d)tfdbg_context_idop_name output_slotrC debug_urlsrIrDrO)dtype)Z tensor_idrC output_dtypezDSymbolic tensor instrumentation is not implemented for debug mode %s)"r*r(r #executing_eagerly_outside_functions enumerater>r7r+r)rrvr_should_dump_tensorris_numpy_compatibler`rZdebug_identity_v2r Zconstantr Zfloat32r{ CURT_HEALTHCONCISE_HEALTH FULL_HEALTHSHAPE is_floating is_integeris_booldebug_numeric_summary_v2float64rwNotImplementedError)rAtensorsrr}r| tensor_idsrCris_v1_graph_modeZinstrumented_tensorsr~r%Zdebug_identity_nameZdebug_identity_op_kwargsrzrZdtype_is_dumpablerrr_instrument_symbolic_tensorsHs            z-_DumpingCallback._instrument_symbolic_tensorsNc Cs|j}dd|D}t|t|ks(t|tjjkrVtj||t||||||dS|tjjtjj tjj tjj tjj fkrtj|t|||||||d}x|D]} | || jr| jjr|tjjtjj tjj fkr| jjrttj| |tjd} nt} nd|tjj krV| jjs4| jjs4| jjrLttj| |tjd} nt} n|tjj krlt| } | r|j| qW|Std|jdS)aDump the value of eager tensors. The destination of the dumping is determined by the dump_root of the currently enabled dumping callback. The tensors may be transformed prior to dumping (e.g., reduced as summary statistics such as minimum, maximum and arithmetic mean). The details of this transformation (if any) depends on the tensor_debug_mode of the currently enabled dumping callback. Args: tensors: The EagerTensors whose values are to be dumped, with or without value transform. op_type: Type of the op that generates the tensors, as a string. input_tensor_ids: IDs of the input EagerTensors to the op. output_tensor_device_ids: Debugged-generated IDs for the devices on which the output tensors are allocated, as a `list` of `int`s. Must match `tensors` in length. graph_id: ID of the executed graph, applicable only to eager execution of a FuncGraph. Returns: A tfdbg Execution protocol buffer. cSsg|] }|jqSr)_id).0trrr sz8_DumpingCallback._dump_eager_tensors..)rrM num_outputsinput_tensor_idsoutput_tensor_idsoutput_tensor_device_idsrCrq)rrrMrrrrCrq)rCrz@Tensor instrumentation is not implemented for debug mode %s yet N)r*lenAssertionErrorrrvrZ Executionrrrrrrrwrrrrr&rrr rrZ TensorProtorrZ tensor_protosr`r) rArrrrrMrCrZexecution_protor%Z tensor_protorrr_dump_eager_tensorsst        z$_DumpingCallback._dump_eager_tensorsc sp~|rt }|}t|} |dkrL|djdd}|rx2|D]*} | jkrV|rV|dj j| j qVWt j ||t |dr|jnd|fdd|Dt|| d} | |rt|tjkr||||| Snvt|} | d krdS| tjkrdS|}d d|D} |rNfd d|Dng}j||| ||d dS) z;Op callback for tracing (dumping) a TF program's execution.)rursrtr:rINcsg|]}|qSr)_lookup_tensor_name)r input_tensor)rArrr:sz-_DumpingCallback.callback..)rr}rRrMZ input_namesrrrqsDebugNumericSummaryV2cSsg|] }|jqSr)r)rrrrrrMscsg|]}|jqSr)ZRegisterDeviceAndGetIdZdevice)routput)rcrrrNs)rM)rQr rrH_get_symbolic_tensor_idsrrIsplitr?rryrZGraphOpCreationrXrrZWriteGraphOpCreationrrrZOP_CALLBACK_SKIP_OPSr_func_graph_id_from_func_nameZWriteExecutionr)rArrKattrsrLr}rJrrVrrZgraph_op_creationZ op_type_bytesZ input_idsrr)rArcrcallbacksL           z_DumpingCallback.callbackcCs|j|j|jS)aLook up the name of a graph tensor. This method maps the name of a debugger-generated Identity or DebugIdentityV2 tensor to the name of the original instrumented tensor, if `tensor` is such a debugger-created tensor. Otherwise, it returns the name of `tensor` as is. Args: tensor: The graph tensor to look up the name for. Returns: Name of the orignal instrumented tensor as known to the debugger. )r8getrI)rAr%rrrrTsz$_DumpingCallback._lookup_tensor_namec Csvt|}t|rn||jkr&|j|S|j8x0|jD]&}|j|kr6|j|}||j|<|Sq6WWdQRXdSdSdS)aAttempt to get the ID of a FuncGraph based on an op type name. Also caches the ID for faster access later. Args: op_type: Op type string, which may be the name of a function. Returns: If the op_type name does not fit the pattern of a function name (e.g., one that starts with "__inference_"), `None` is returned immediately. Else, if the FuncGraph is found, ID of the underlying FuncGraph is returned as a string. Else, `None` is returned. N)rrrr6r=r5rI)rArrrMrrrrds       z._DumpingCallback._func_graph_id_from_func_namec CsJg}|rF|j2x*t|D]}|jd7_||jqWWdQRX|S)N)r>ranger7r`)rAZ num_tensorsrrarrrrsz)_DumpingCallback._get_symbolic_tensor_idscsbd}|jr|ot|j|}|jr^t|jttfrP|oLtfdd|jD}n|o\|}|S)a}Determine if the given tensor's value will be dumped. The determination is made given the configurations such as `op_regex`, `tensor_dtypes`. Args: op_type: Name of the op's type, as a string (e.g., "MatMul"). dtype: The dtype of the tensor, as a `dtypes.DType` object. Returns: A bool indicating whether the tensor's value will be dumped. Tc3s|]}|kVqdS)Nr)r dtype_item)rrr sz7_DumpingCallback._should_dump_tensor..)r,rematchr- isinstancelisttupleany)rArrZ should_dumpr)rrrs  z$_DumpingCallback._should_dump_tensor)N)NN)__name__ __module__ __qualname____doc__rGrNpropertyrBsetterrOrCrDrQrHrTrdrrr{rrrrrrrrrrrr'Os,(     #+z ] 4# r'z-debugging.experimental.enable_dump_debug_infoicCstj}||kr"td||ftj|}|tjjtjjtjjtjjtjj tjj fkrnt dtj ||dk rt |ttfst|std|ft |ttfrdd|D}ttdrtjj|krtdtjj|ftjj|krBtd tjj|tjj||fn.t|||||t_ttjjttjjtjj|krX|tj_td t !j"tjjtj |t#$t%tj&S) aEnable dumping debugging information from a TensorFlow program. The debugging information is dumped to a directory on the file system specified as `dump_root`. The dumped debugging information can be ingested by debugger UIs. The files in the dump directory contain the following information: - TensorFlow Function construction (e.g., compilation of Python functions decorated with @tf.function), the op types, names (if available), context, the input and output tensors, and the associated stack traces. - Execution of TensorFlow operations (ops) and Functions and their stack traces, op types, names (if available) and contexts. In addition, depending on the value of the `tensor_debug_mode` argument (see Args section below), the value(s) of the output tensors or more concise summaries of the tensor values will be dumped. - A snapshot of Python source files involved in the execution of the TensorFlow program. Once enabled, the dumping can be disabled with the corresponding `disable_dump_debug_info()` method under the same Python namespace. Calling this method more than once with the same `dump_root` is idempotent. Calling this method more than once with different `tensor_debug_mode`s leads to a `ValueError`. Calling this method more than once with different `circular_buffer_size`s leads to a `ValueError`. Calling this method with a different `dump_root` abolishes the previously-enabled `dump_root`. Usage example: ```py tf.debugging.experimental.enable_dump_debug_info('/tmp/my-tfdbg-dumps') # Code to build, train and run your TensorFlow model... ``` NOTE: If your code is running on TPUs, be sure to call `tf.config.set_soft_device_placement(True)` before calling `tf.debugging.experimental.enable_dump_debug_info()` as this API uses automatic outside compilation on TPUs. For example: ```py tf.config.set_soft_device_placement(True) tf.debugging.experimental.enable_dump_debug_info( logdir, tensor_debug_mode="FULL_HEALTH") resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') strategy = tf.distribute.TPUStrategy(resolver) with strategy.scope(): # ... ``` Args: dump_root: The directory path where the dumping information will be written. tensor_debug_mode: Debug mode for tensor values, as a string. The currently supported options are: - "NO_TENSOR": (Default) Only traces the output tensors of all executed ops (including those executed eagerly at the Python level or as a part of a TensorFlow graph) and functions, while not extracting any information from the values of the tensors. - "CURT_HEALTH": For each floating-dtype tensor (e.g., tensors of dtypes such as `float32`, `float64` and `bfloat16`), extracts a binary bit indicating whether it contains any -infinity, +infinity or NaN. - "CONCISE_HEALTH": For each floating-dtype tensor, extract total element count, and counts of -infinity, +infinity and NaN elements. - "FULL_HEALTH": For each floating-dtype tensor, extracts the dtype, rank (number of dimensions), total element count, and counts of -infinity, +infinity and NaN elements. - "SHAPE": For each tensor (regardless of dtype), extracts its dtype, rank, total element count and shape. circular_buffer_size: Size of the circular buffers for execution events. These circular buffers are designed to reduce the overhead of debugging dumping. They hold the most recent debug events concerning eager execution of ops and `tf.function`s and traces of tensor values computed inside `tf.function`s. They are written to the file system only when the proper flushing method is called (see description of return values below). Expected to be an integer. If <= 0, the circular-buffer behavior will be disabled, i.e., the execution debug events will be written to the file writers in the same way as non-execution events such as op creations and source-file snapshots. op_regex: Dump data from only the tensors from op types that matches to the regular expression (through Python's `re.match()`). "Op type" refers to the names of the TensorFlow operations (e.g., "MatMul", "LogSoftmax"), which may repeat in a TensorFlow function. It does *not* refer to the names of nodes (e.g., "dense/MatMul", "dense_1/MatMul_1") which are unique within a function. - Example 1: Dump tensor data from only MatMul and Relu ops `op_regex="^(MatMul|Relu)$"`. - Example 2: Dump tensors from all ops *except* Relu: `op_regex="(?!^Relu$)"`. This filter operates in a logical AND relation with `tensor_dtypes`. tensor_dtypes: Dump data from only the tensors of which the specified dtypes. This optional argument can be in any of the following format: - a list or tuple of `DType` objects or strings that can be converted to `DType` objects via `tf.as_dtype()`. Examples: - `tensor_dtype=[tf.float32, tf.float64]`, - `tensor_dtype=["float32", "float64"]`, - `tensor_dtypes=(tf.int32, tf.bool)`, - `tensor_dtypes=("int32", "bool")` - a callable that takes a single `DType` argument and returns a Python `boolean` indicating whether the dtype is to be included in the data dumping. Examples: - `tensor_dtype=lambda dtype: dtype.is_integer`. This filter operates in a logical AND relation with `op_regex`. Returns: A DebugEventsWriter instance used by the dumping callback. The caller may use its flushing methods, including `FlushNonExecutionFiles()` and `FlushExecutionFiles()`. z@Invalid value in tensor_debug_mode ('%s'). Valid options are: %szFtfdbg dumping: support for tensor debug mode %s is not implemented yetNzIf specified, tensor_dtypes is expected to be a list, a tuple, or a callable that takes a DType argument and returns a boolean, but received %scSsg|]}t|qSr)r Zas_dtype)rrrrrr@sz*enable_dump_debug_info..dumping_callbackzThere is already a dumping callback configured with a different circular-buffer size (%d). Therefore the newly request circular-buffer size (%d) will not be honored.zThere is already a dumping callback configured for dump root %s with a different tensor-debug mode (%s). Therefore the newly request tensor-debug mode (%s) size will not be honored.zLEnabled dumping callback in thread %s (dump root: %s, tensor debug mode: %s))'rrvkeys ValueErrorValuerrrrrrwrNamerrrcallablerX_staterrDrCrBr'r Zadd_op_callbackr function_libZadd_function_callbackrNr^infor9current_threadrIatexitregisterdisable_dump_debug_inforQ)rBrCrDrErFZtensor_debug_mode_keysrrrenable_dump_debug_infosfy          rz.debugging.experimental.disable_dump_debug_infocCshttdrdtjj}tjj}t||t tjj t tjj ttdtdtj|dS)aIDisable the currently-enabled debugging dumping. If the `enable_dump_debug_info()` method under the same Python namespace has been invoked before, calling this method disables it. If no call to `enable_dump_debug_info()` has been made, calling this method is a no-op. Calling this method more than once is idempotent. rz6Disabled dumping callback in thread %s (dump root: %s)N)rXrrrBrOrrPCloser Zremove_op_callbackrrZremove_function_callbackrNdelattrr^rr9rrI)rBrOrrrrjs   r)6rrrlrr.r9r Ztensorflow.core.frameworkrZtensorflow.core.protobufrrZtensorflow.python.debug.librrrZtensorflow.python.eagerrrZtensorflow.python.frameworkr r r r r Ztensorflow.python.opsrrZtensorflow.python.platformrr^Ztensorflow.python.utilrrZ tensorflow.python.util.tf_exportrlocalrZDEFAULT_TENSOR_DEBUG_MODErZ_FORWARD_PREFIXZ_BACKWARD_PREFIXZ_INFERENCE_PREFIXrrZRegisterGradientrr"r#r&objectr'rrrrrrs\                    `;