B һd @s~dZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZd Zd Zd ZdZdZdZdZddZGdddZ ddZ!ddZ"ddZ#ddZ$ddZ%d d!Z&d"d#Z'd$d%Z(d&d'Z)d(d)Z*e+d*d+d,d-d.d/d0gZ,d1d2Z-d3d4Z.d5d6Z/Gd7d8d8Z0Gd9d:d:e1Z2Gd;d<d<Z3dS)=zGClasses and functions to handle debug-dump data of TensorFlow Debugger.N) graph_pb2) types_pb2) event_pb2) debug_graphs) tensor_util)gfile) tf_logging)compatZ_tfdbg_Zcore_metadata_Zgraph_device_hashZ fetches_info_Zfeed_keys_info_cCs$tdkrt|St|SdS)NWindows)platformsystemglobrZGlob)Z glob_patternrX/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/debug/lib/debug_data.py_glob/s  rc@s.eZdZdZd ddZddZeddZd S) InconvertibleTensorProtoz@Represents a TensorProto that cannot be converted to np.ndarray.TcCs||_||_dS)zConstructor. Args: tensor_proto: the `TensorProto` object that cannot be represented as a `np.ndarray` object. initialized: (`bool`) whether the Tensor is initialized. N) _tensor_proto _initialized)self tensor_proto initializedrrr__init__9sz!InconvertibleTensorProto.__init__cCs |jr dnd}|t|j7}|S)NzUninitialized tensor: )rstrr)routputrrr__str__Dsz InconvertibleTensorProto.__str__cCs|jS)N)r)rrrrrIsz$InconvertibleTensorProto.initializedN)T)__name__ __module__ __qualname____doc__rrpropertyrrrrrr6s rc Cs6t}t|d}||t|SQRXdS)aLoad a tensor from an event file. Assumes that the event file contains a `Event` protobuf and the `Event` protobuf contains a `Tensor` value. Args: event_file_path: (`str`) path to the event file. Returns: The tensor value loaded from the event file, as a `numpy.ndarray`. For uninitialized Tensors, returns `None`. For Tensors of data types that cannot be converted to `numpy.ndarray` (e.g., `tf.resource`), return `None`. rbN)rEventrOpenParseFromStringreadload_tensor_from_event)event_file_patheventfrrrload_tensor_from_event_fileNsr,cCs|jjdj}t|j}d}x|D] }||9}q$W|jsD|jsD|s|jt j krZt |}qyt |}Wqt k rt |}YqXn t |d}|S)aLoad a tensor from an Event proto. Args: event: The Event proto, assumed to hold a tensor value in its summary.value[0] field. Returns: The tensor value loaded from the event file, as a `numpy.ndarray`, if representation of the tensor value by a `numpy.ndarray` is possible. For uninitialized Tensors, returns `None`. For Tensors of data types that cannot be represented as `numpy.ndarray` (e.g., `tf.resource`), return the `TensorProto` protobuf object without converting it to a `numpy.ndarray`. rF)summaryvaluetensorrZTensorShapeProtoToListZ tensor_shapeZtensor_contentZ string_valdtyperZ DT_RESOURCErZ MakeNdarrayKeyError)r*rshapeZ num_elementsZ shape_dimZ tensor_valuerrrr(ds      r(c Cs<t}t|d}||WdQRXtj|j S)Nr#) rr$rr%r&r'rZGraphDefZ FromString graph_def)r)r*r+rrr_load_graph_def_from_event_filesr5c Cs6t}t|d}||WdQRX|jjS)Nr#)rr$rr%r&r' log_messagemessage)r)r*r+rrr!_load_log_message_from_event_filesr8cCs|ttS)N) startswithMETADATA_FILE_PREFIXGRAPH_FILE_TAG) file_namerrr_is_graph_filesr=cCs |ttkS)N)r:FETCHES_INFO_FILE_TAG)r<rrr_is_run_fetches_info_filesr?cCs |ttkS)N)r:FEED_KEYS_INFO_FILE_TAG)r<rrr_is_run_feed_keys_info_filesrAcCs d||fS)zGet tensor name given node name and output slot index. Args: node_name: Name of the node that outputs the tensor, as a string. output_slot: Output slot index of the tensor, as an integer. Returns: Name of the tensor, as a string. z%s:%dr) node_name output_slotrrr_get_tensor_names rDcCsdt|||fS)aGet the string representation of a debug watch on a tensor. Args: node_name: Name of the node by which the watched tensor is produced, as a string. output_slot: Output slot index of the tensor, as an integer. debug_op: Name of the debug op that is used to watch the tensor, as a string. Returns: A string representing the debug watch on the tensor (i.e., the "watch key"). z%s:%s)rD)rBrCdebug_oprrr_get_tensor_watch_keysrFcCsj|}t|trdSt|jtjsBt|jtjsBt|jtjrbtt |p`tt |SdSdS)aA predicate for whether a tensor consists of any bad numerical values. This predicate is common enough to merit definition in this module. Bad numerical values include `nan`s and `inf`s. The signature of this function follows the requirement of the method `DebugDumpDir.find()`. Args: datum: (`DebugTensorDatum`) Datum metadata. tensor: (`numpy.ndarray` or None) Value of the tensor. None represents an uninitialized tensor. Returns: (`bool`) True if and only if tensor consists of any nan or inf values. FN) isinstancernpZ issubdtyper1ZfloatingZcomplexfloatingintegeranyisnanisinf)datumr0_rrrhas_inf_or_nans  rOZ CoreMetadata global_stepsession_run_indexexecutor_step_index input_names output_names target_nodescCs8t|jj}t|d|d|d|d|d|dS)NrPrQrRrSrTrU)jsonloadsr6r7 _CoreMetadata)r*Z json_metadatarrr&extract_core_metadata_from_event_protosrYcCs0t|d}dd|D}ttd|S)z#Convert device name to device path./cSsg|]}|ddqS):rN)replace).0itemrrr sz.device_name_to_device_path..,)r Zas_textsplitr: DEVICE_TAGjoin) device_nameZdevice_name_itemsrrrdevice_name_to_device_pathsrecCs:tj|ttttdd}ddd|DS)zParse device name from device path. Args: device_dir: (str) a directory name for the device. Returns: (str) parsed device name. Nr`rZcSs"g|]}|dddddqS)r zdevice:rNr[r-)r\)r] path_itemrrrr_sz.device_path_to_device_name..)ospathbasenamelenr:rbrarc) device_dirZ path_itemsrrrdevice_path_to_device_names rlc@seZdZdZddZddZddZdd Zed d Z ed d Z eddZ eddZ eddZ eddZeddZeddZeddZeddZdS)DebugTensorDatumaA single tensor dumped by TensorFlow Debugger (tfdbg). Contains metadata about the dumped tensor, including `timestamp`, `node_name`, `output_slot`, `debug_op`, and path to the dump file (`file_path`). This type does not hold the generally space-expensive tensor value (numpy array). Instead, it points to the file from which the tensor value can be loaded (with the `get_tensor` method) if needed. cCstj|tj}t|d|_|d}|ddkrDtd||dd|_ d|j kr|t |j d|j d|_ n t |j |_ |dd|_ t |dd |_d|ddd }d |d d|g|_tj|||_t|jr t|jjnd|_dS) a`DebugTensorDatum` constructor. Args: dump_root: (`str`) Debug dump root directory. This path should not include the path component that represents the device name (see also below). debug_dump_rel_path: (`str`) Path to a debug dump file, relative to the `dump_root`. The first item of this relative path is assumed to be a path representing the name of the device that the Tensor belongs to. See `device_path_to_device_name` for more details on the device path. For example, suppose the debug dump root directory is `/tmp/tfdbg_1` and the dump file is at `/tmp/tfdbg_1//>ns_1/node_a_0_DebugIdentity_123456789`, then the value of the debug_dump_rel_path should be `/ns_1/node_a_0_DebugIdentity_1234456789`. Raises: ValueError: If the base file name of the dump file does not conform to the dump file naming pattern: `node_name`_`output_slot`_`debug_op`_`timestamp` rrNz9Dump file path does not conform to the naming pattern: %s-NrZr-)rgrhnormpathraseprl _device_namecount ValueError_extended_timestampintfind _timestamp _debug_op _output_slotrc _node_name _file_pathrZExistsZStatlength_dump_size_bytes)r dump_rootdebug_dump_rel_pathZpath_componentsbaseZnode_base_namerrrrs"   zDebugTensorDatum.__init__cCsd|j|j|j|j|jfS)Nz'{DebugTensorDatum (%s) %s:%d @ %s @ %d})rdrBrCrE timestamp)rrrrrHs zDebugTensorDatum.__str__cCs|S)N)r)rrrr__repr__OszDebugTensorDatum.__repr__cCs t|jS)zqGet tensor from the dump (`Event`) file. Returns: The tensor loaded from the dump (`Event`) file. )r, file_path)rrrr get_tensorRszDebugTensorDatum.get_tensorcCs|jS)zoTimestamp of when this tensor value was dumped. Returns: (`int`) The timestamp in microseconds. )r{)rrrrr\szDebugTensorDatum.timestampcCs|jS)aSExtended timestamp, possibly with an index suffix. The index suffix, e.g., "-1", is for disambiguating multiple dumps of the same tensor with the same timestamp, which can occur if the dumping events are spaced by shorter than the temporal resolution of the timestamps. Returns: (`str`) The extended timestamp. )rx)rrrrextended_timestampfs z#DebugTensorDatum.extended_timestampcCs|jS)z]Name of the debug op. Returns: (`str`) debug op name (e.g., `DebugIdentity`). )r|)rrrrrEtszDebugTensorDatum.debug_opcCs|jS)z\Name of the device that the tensor belongs to. Returns: (`str`) device name. )ru)rrrrrd~szDebugTensorDatum.device_namecCs|jS)zName of the node from which the tensor value was dumped. Returns: (`str`) name of the node watched by the debug op. )r~)rrrrrBszDebugTensorDatum.node_namecCs|jS)zOutput slot index from which the tensor value was dumped. Returns: (`int`) output slot index watched by the debug op. )r})rrrrrCszDebugTensorDatum.output_slotcCst|j|jS)zName of the tensor watched by the debug op. Returns: (`str`) `Tensor` name, in the form of `node_name`:`output_slot` )rDrBrC)rrrr tensor_nameszDebugTensorDatum.tensor_namecCst|j|j|jS)zWatch key identities a debug watch on a tensor. Returns: (`str`) A watch key, in the form of `tensor_name`:`debug_op`. )rFrBrCrE)rrrr watch_keys zDebugTensorDatum.watch_keycCs|jS)z=Path to the file which stores the value of the dumped tensor.)r)rrrrrszDebugTensorDatum.file_pathcCs|jS)zSize of the dump file. Unit: byte. Returns: If the dump file exists, size of the dump file, in bytes. If the dump file does not exist, None. )r)rrrrdump_size_bytess z DebugTensorDatum.dump_size_bytesN)rrr r!rrrrr"rrrErdrBrCrrrrrrrrrm s 0  rmc@s eZdZdS)'WatchKeyDoesNotExistInDebugDumpDirErrorN)rrr rrrrrsrc@seZdZdZd`ddZddZdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ eddZeddZeddZeddZed d!Zd"d#Zd$d%Zd&d'Zd(d)Zdad+d,Zd-d.Zd/d0Zd1d2Zed3d4Zed5d6Zd7d8Zdbd9d:Zdcd;d<Z ddd>d?Z!ded@dAZ"dBdCZ#dfdDdEZ$dgdFdGZ%dHdIZ&dhdJdKZ'dLdMZ(didNdOZ)djdPdQZ*dkdRdSZ+dldTdUZ,dmdVdWZ-dndXdYZ.dodZd[Z/dpd\d]Z0d^d_Z1dS)q DebugDumpDirzData set from a debug-dump directory on filesystem. An instance of `DebugDumpDir` contains all `DebugTensorDatum` instances in a tfdbg dump root directory. NTcCsPt|std|g|_||_||||||d|_ dS)a`DebugDumpDir` constructor. Args: dump_root: (`str`) path to the dump root directory. partition_graphs: A repeated field of GraphDefs representing the partition graphs executed by the TensorFlow runtime. validate: (`bool`) whether the dump files are to be validated against the partition graphs. Raises: IOError: If dump_root does not exist as a directory. ValueError: If more than one core metadata file is found under the dump root directory. z%Dump root directory %s does not existN) rZ IsDirectoryIOError_core_metadata _dump_root_load_core_metadata_load_fetches_info_load_feeds_info_load_all_device_dumps _python_graph)rrpartition_graphsvalidaterrrrs   zDebugDumpDir.__init__cCsttj|jttd}g|_i|_i|_ i|_ i|_ i|_ i|_ i|_i|_x,|D]$}t|}|j||||qXW||||x|jD]}||qWdS)z#Load the dump data for all devices.*N)rrgrhrcrr:rb _device_names_t0s_dump_tensor_data_dump_graph_file_paths_debug_watches_watch_key_to_devices_watch_key_to_datum_watch_key_to_rel_time_watch_key_to_dump_size_bytesrlappend_load_device_dumps_load_partition_graphs _calculate_t0_create_tensor_watch_maps)rrrZ device_dirsrkrdrrrrs&    z#DebugDumpDir._load_all_device_dumpscCsg|j|<tdd|j|<x~t|D]p\}}}xd|D]\}t|r\tj |||j |<q:| ||}|j| ||j||j |j|jq:Wq*Wt|j|ddd|j|<|j|r|j|dj|j|<n d|j|<dS)aeLoad `DebugTensorDatum` instances from the dump root of a given device. Populates a map {device_name: a list of `DebugTensorDatum`}, where the list is sorted by ascending timestamp. This sorting order reflects the order in which the TensorFlow executor processed the nodes of the graph. It is (one of many possible) topological sort of the nodes. This is useful for displaying tensors in the debugger frontend as well as for the use case in which the user wants to find a "culprit tensor", i.e., the first tensor in the graph that exhibits certain problematic properties, i.e., all zero values, or bad numerical values such as nan and inf. In addition, creates a map from node name to debug watches. In this Map, the key is the watched node name; the value is a dictionary. Of this dictionary, the key is the watched_output_slot. This method attempts to load the debug watches from the tensor dump files first, before loading the full set of debug watches from the partition graphs as done later. This is necessary because sometimes the partition graphs may not be available, e.g., when the run errors out. Args: device_name: (`str`) name of the device. device_root: (`str`) dump root directory of the given device. Raises: ValueError: If GraphDef for the device is not available. cSs ttS)N) collections defaultdictsetrrrr(z1DebugDumpDir._load_device_dumps..cSs|jS)N)r)xrrrr6r)keyrN)rrrrrZWalkr=rgrhrcr_dump_file_name_to_datumrrBrCaddrEsortedrr)rrdZ device_rootrootrNfilesr+rMrrrrs"    zDebugDumpDir._load_device_dumpscCs*dd|jD}|r t|nd|_dS)z1Calculate the first timestamp across all devices.cSsg|]}|dk r|qS)Nr)r]t0rrrr_?sz.DebugDumpDir._calculate_t0..N)rvaluesmin_t0)rZt0srrrr=szDebugDumpDir._calculate_t0c Cslttj|jttd}xJ|D]B}t|d,}t }| | |j t|WdQRXq"WdS)Nrr#)rrgrhrcrr:CORE_METADATA_TAGrr%rr$r&r'rrrY)rZcore_metadata_filesZcore_metadata_filer+r*rrrrBs z DebugDumpDir._load_core_metadatacCsDttj|jttd}g|_x|D]}|jt |q(WdS)Nr) rrgrhrcrr:r>_run_fetches_inforr8)rZfetches_info_filesZfetches_info_filerrrrLs  zDebugDumpDir._load_fetches_infocCsDttj|jttd}g|_x|D]}|jt |q(WdS)Nr) rrgrhrcrr:r@_run_feed_keys_inforr8)rZfeeds_info_filesZfeeds_info_filerrrrTs  zDebugDumpDir._load_feeds_infocCs&tjtj||j|}t|j|S)a)Obtain a DebugTensorDatum from the directory and file name. Args: dir_name: (`str`) Name of the directory in which the dump file resides. file_name: (`str`) Base name of the dump file. Returns: (`DebugTensorDatum`) The `DebugTensorDatum` loaded from the dump file. )rgrhrcrelpathrrm)rdir_namer<rrrrr\s z%DebugDumpDir._dump_file_name_to_datumcCsi|j|<i|j|<i|j|<x|j|D]}|j|jkrJ|h|j|j<n|j|j||j|j|kr|g|j||j<|j|jg|j||j<|j g|j||j<q*|j||j ||j||j |j|j|j||j |j q*WdS)a;Create maps from tensor watch keys to datum and to timestamps. Create a map from watch key (tensor name + debug op) to `DebugTensorDatum` item. Also make a map from watch key to relative timestamp. "relative" means (absolute timestamp - t0). Args: device_name: (str) name of the device. N) rrrrrrrrrrr)rrdrMrrrrls    z&DebugDumpDir._create_tensor_watch_mapscCsB||_i|_|jr>x*|jD]}ttt|j|j|j<qWdS)aJProvide Python `Graph` object to the wrapper. Unlike the partition graphs, which are protobuf `GraphDef` objects, `Graph` is a Python object and carries additional information such as the traceback of the construction of the nodes in the graph. Args: python_graph: (ops.Graph) The Python Graph object. N)r_node_tracebackZget_operationstuplemap tracebackname)r python_graphoprrrset_python_graphs zDebugDumpDir.set_python_graphcCs|jS)zGet the Python graph. Returns: If the Python graph has been set, returns a `tf.Graph` object. Otherwise, returns None. )r)rrrrrs zDebugDumpDir.python_graphcCs|j}t|dkr|dS|S)a^Metadata about the `Session.run()` call from the core runtime. Of the three counters available in the return value, `global_step` is supplied by the caller of the debugged `Session.run()`, while `session_run_index` and `executor_step_index` are determined by the state of the core runtime, automatically. For the same fetch list, feed keys and debug tensor watch options, the same executor will be used and `executor_step_index` should increase by one at a time. However, runs with different fetch lists, feed keys and debug_tensor watch options that all share the same `Session` object can lead to gaps in `session_run_index`. Returns: If core metadata are loaded, a `namedtuple` with the fields: `global_step`: A global step count supplied by the caller of `Session.run()`. It is optional to the caller. If the caller did not supply this parameter, its value will be -1. `session_run_index`: A sorted index for Run() calls to the underlying TensorFlow `Session` object. `executor_step_index`: A counter for invocations of a given runtime executor. The same executor is re-used for the same fetched tensors, target nodes, input feed keys and debug tensor watch options. `input_names`: Names of the input (feed) Tensors. `output_names`: Names of the output (fetched) Tensors. `target_nodes`: Names of the target nodes. If the core metadata have not been loaded, `None`. If more than one core metadata files exist, return a list of the `nametuple` described above. r-r)rrj)rrrrr core_metadataszDebugDumpDir.core_metadatacCs\t|dkr"|j|dS|j}g}x|D]}||q6Wt|dddSdS)zRetrieve dumped tensor data.r-rcSs|jS)N)r)rrrrrrz1DebugDumpDir.dumped_tensor_data..)rN)rjdevicesrrextendr)rZall_devices_datadataZ device_datarrrdumped_tensor_datas  zDebugDumpDir.dumped_tensor_datacCs|jS)zAbsolute timestamp of the first dumped tensor across all devices. Returns: (`int`) absolute timestamp of the first dumped tensor, in microseconds. )r)rrrrrszDebugDumpDir.t0cstfddjDS)zTotal number of dumped tensors in the dump root directory. Returns: (`int`) The total number of dumped tensors in the dump root directory. c3s|]}tj|VqdS)N)rjr)r]rd)rrr sz$DebugDumpDir.size..)sumr)rr)rrsizes zDebugDumpDir.sizecCsi|_i|_g}xX|jD]N}d}||jkr:t|j|}ntd||||}|r|||fqWxP|D]H\}}t j ||d}||j|j <| ||rp|j |j krp||j qpWdS)a&Load and process partition graphs. Load the graphs; parse the input and control input structure; obtain the device and op type of each node; remove the Copy and debug ops inserted by the debugger. The gathered information can be used to validate the tensor dumps. Args: client_partition_graphs: A repeated field of GraphDefs representing the partition graphs executed by the TensorFlow runtime, from the Python client. These partition graphs are used only if partition graphs cannot be loaded from the dump directory on the file system. validate: (`bool`) Whether the dump files are to be validated against the partition graphs. Raises: ValueError: If the partition GraphDef of one or more devices fail to be loaded. NzFailed to load partition graphs for device %s from disk. As a fallback, the client graphs will be used. This may cause mismatches in device names.)rd) _debug_graphs _node_devicesrrr5loggingwarn_find_partition_graphrrZ DebugGraphrd_collect_node_devicesr_validate_dump_with_graphs)rZclient_partition_graphsrZ!partition_graphs_and_device_namesrdZpartition_graphZmaybe_device_name debug_graphrrrrs.     z#DebugDumpDir._load_partition_graphscCs@|dkr dSx*|D]"}x|jD]}|j|kr|SqWqWdSdS)N)nodedevice)rrrdr4Znode_defrrrrs    z"DebugDumpDir._find_partition_graphcCsLxF|jD]<}||jkr4|j||j||j|<q|j||j|<qWdS)N)Z node_devicesrunion)rrrBrrrr&s    z"DebugDumpDir._collect_node_devicescCs|jstd||j|}x0|j|D]"}|j|jkr(td|j|fq(Wi}x|jD]}g||<|j|}xr|D]j}t|}t|} ||j |krv| |j ||krv|j |dkrv|| f||krv|| || fqvWqZWxt |j|D]\} }|j}|j} |j||||j| ddsFtd||jt||f|j|} xP| D]H} || }|| f|krV|| dkr|dd=n|||| f=qVWqWdS) a Validate the dumped tensor data against the partition graphs. Only the watched nodes are validated by this method, because tfdbg allows clients to watch only a subset of the nodes. Args: device_name: (`str`) device name. Raises: LookupError: If the partition graphs have not been loaded yet. ValueError: If dumps contain node names not found in partition graph. Or if the temporal order of the dump's timestamps violate the input relations on the partition graphs. z(No partition graphs loaded for device %sz=Node name '%s' is not found in partition graphs of device %s.)ZEnterZ NextIterationr-)start_izdCausality violated in timing relations of debug dumps: %s (%d): these input(s) are not satisfied: %sMergeN)r LookupErrorrrB node_inputsrwr get_node_nameZget_output_slotr node_op_typesgetr enumeraterC_satisfied_at_timestamprreprnode_recipients node_op_typeindex)rrdrrMZpending_inputsrinputsinpZinp_nodeZinp_output_slotiZslotZ recipientsZ recipientZrecipient_pending_inputsrrrr.sH            z'DebugDumpDir._validate_dump_with_graphsrcCsj|sdSxZ|j||dD]D}|j|kr,P|j|kr|j|jf|kr||j|jf|sdSqW| S)aGDetermine whether pending inputs are satisfied at given timestamp. Note: This method mutates the input argument "pending". Args: device_name: (str) device name. pending: A list of 2-tuple (node_name, output_slot): the dependencies to check. timestamp: (int) the timestamp in question. start_i: (int) the index in self._dump_tensor_data to start searching for the timestamp. Returns: (bool) Whether all the dependencies in pending are satisfied at the timestamp. If pending is empty to begin with, return True. TN)rrrBrCremove)rrdpendingrrrMrrrrts  z$DebugDumpDir._satisfied_at_timestampcCs t|jS)z/Test whether partition graphs have been loaded.)boolr)rrrrloaded_partition_graphssz$DebugDumpDir.loaded_partition_graphscs"jstdfddjDS)zGet the partition graphs. Returns: Partition graphs as a list of GraphDef. Raises: LookupError: If no partition graphs have been loaded. z%No partition graphs have been loaded.csg|]}j|jqSr)rZdebug_graph_def)r]r)rrrr_sz1DebugDumpDir.partition_graphs..)rr)rr)rrrs  zDebugDumpDir.partition_graphscCs(i}x|jD]}|j|j||<q W|S)aReconstruct partition graphs with the debugger-inserted ops stripped. The reconstructed partition graphs are identical to the original (i.e., non-debugger-decorated) partition graphs except in the following respects: 1) The exact names of the runtime-inserted internal nodes may differ. These include _Send, _Recv, _HostSend, _HostRecv, _Retval ops. 2) As a consequence of 1, the nodes that receive input directly from such send- and recv-type ops will have different input names. 3) The parallel_iteration attribute of while-loop Enter ops are set to 1. Returns: A dict mapping device names (`str`s) to reconstructed `tf.compat.v1.GraphDef`s. )rZnon_debug_graph_def)rZnon_debug_graphsrrrr(reconstructed_non_debug_partition_graphss z5DebugDumpDir.reconstructed_non_debug_partition_graphscCs|j}t|dkr|dS|S)axGet a str representation of the fetches used in the Session.run() call. Returns: If the information is available from one `Session.run` call, a `str` obtained from `repr(fetches)`. If the information is available from multiple `Session.run` calls, a `list` of `str` from `repr(fetches)`. If the information is not available, `None`. r-r)rrj)rrrrrrun_fetches_infos zDebugDumpDir.run_fetches_infocCs|j}t|dkr|dS|S)aGet a str representation of the feed_dict used in the Session.run() call. Returns: If the information is available from one `Session.run` call, a `str` obtained from `repr(feed_dict)`. If the information is available from multiple `Session.run` calls, a `list` of `str` obtained from `repr(feed_dict)`. If the information is not available, `None`. r-r)rrj)rrrrrrun_feed_keys_infos zDebugDumpDir.run_feed_keys_infocCsr|dkrj||jkrRt|j|dkr6t|j|dStdt|j||fqntdt|j|fn|SdS)aEInfer the device name given node name. If device_name is provided (i.e., not None), it'll be simply returned right away. Args: device_name: (str or None) name of the device. If None, will try to infer the device name by looking at the available nodes. node_name: (str) name of the node. Returns: (str) Inferred name of the device, if available. Raises: ValueError: If the node name does not exist on any of the available devices or if there are multiple devices that contain the node with the given name. Nr-rzWThere are multiple (%d) devices with nodes named '%s' but device_name is not specified.z/None of the %d device(s) has a node named '%s'.)rrjlistrwr)rrdrBrrr_infer_device_names zDebugDumpDir._infer_device_namecCsn|jstd|dkrDg}x$|jD]}||j|jq"W|S||jkrZtd||j|jSdS)aoGet a list of all nodes from the partition graphs. Args: device_name: (`str`) name of device. If None, all nodes from all available devices will be included. Returns: All nodes' names, as a list of str. Raises: LookupError: If no partition graphs have been loaded. ValueError: If specified node name does not exist. z%No partition graphs have been loaded.NzInvalid device name: %s)rrrrkeysrw)rrdnodesrrrrs   zDebugDumpDir.nodescCs*|jstd|||}|j|j|S)ajGet the attributes of a node. Args: node_name: Name of the node in question. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: Attributes of the node. Raises: LookupError: If no partition graphs have been loaded. z%No partition graphs have been loaded.)rrrnode_attributes)rrBrdrrrrs zDebugDumpDir.node_attributesFcCsB|jstd|||}|r.|j|j|S|j|j|SdS)aDGet the inputs of given node according to partition graphs. Args: node_name: Name of the node. is_control: (`bool`) Whether control inputs, rather than non-control inputs, are to be returned. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: (`list` of `str`) inputs to the node, as a list of node names. Raises: LookupError: If node inputs and control inputs have not been loaded from partition graphs yet. z5Node inputs are not loaded from partition graphs yet.N)rrrnode_ctrl_inputsr)rrB is_controlrdrrrr's zDebugDumpDir.node_inputscCsz|jstd|||}|j|jg}|r>||j|j|rT||j|jtj|| |d}| || S)aGet the transitive inputs of given node according to partition graphs. Args: node_name: Name of the node. include_control: Include control inputs (True by default). include_reversed_ref: Whether a ref input, say from A to B, is to be also considered as an input from B to A. The rationale is that ref inputs generally let the recipient (e.g., B in this case) mutate the value of the source (e.g., A in this case). So the reverse direction of the ref edge reflects the direction of information flow. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: (`list` of `str`) all transitive inputs to the node, as a list of node names. Raises: LookupError: If node inputs and control inputs have not been loaded from partition graphs yet. z5Node inputs are not loaded from partition graphs yet.)skip_node_names) rrrrrrnode_reversed_ref_inputsrDFSGraphTracer_get_merge_node_namestracer)rrBinclude_controlinclude_reversed_refrd input_liststracerrrrtransitive_inputsBs  zDebugDumpDir.transitive_inputscs^||jkrtd|t|ds&i|_||jkrT|j|fddjD|j|<|j|S)z3Lazily get a list of Merge nodes on a given device.zInvalid device name: %s_merge_node_namescsg|]}j|dkr|qS)r)r)r]r)rrrr_xsz6DebugDumpDir._get_merge_node_names..)rrwhasattrrrr)rrdr)rrrns     z"DebugDumpDir._get_merge_node_namescCs|||}|||}||kr4td||||f|j|jg}|j|} |r\|| j|rl|| jtj|| ||d} y| |Wnztj k r |g| } dg| } g} | d}x8tt| t| D]"\}}||kr| ||d8}qW| SXdS)aFind a path between a source node and a destination node. Limitation: the source and destination are required to be on the same device, i.e., this method does not yet take into account Send/Recv nodes across devices. TODO(cais): Make this method work across device edges by tracing Send/Recv nodes. Args: src_node_name: (`str`) name of the source node or name of an output tensor of the node. dst_node_name: (`str`) name of the destination node or name of an output tensor of the node. include_control: (`bool`) whrther control edges are considered in the graph tracing. include_reversed_ref: Whether a ref input, say from A to B, is to be also considered as an input from B to A. The rationale is that ref inputs generally let the recipient (e.g., B in this case) mutate the value of the source (e.g., A in this case). So the reverse direction of the ref edge reflects the direction of information flow. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: A path from the src_node_name to dst_node_name, as a `list` of `str`, if it exists. The list includes src_node_name as the first item and dst_node_name as the last. If such a path does not exist, `None`. Raises: ValueError: If the source and destination nodes are not on the same device. zFSource (%s) and destination (%s) are not on the same device: %s vs. %s)rZdestination_node_namerrnr-N)rrwrrrrrrrrrZGraphTracingReachedDestinationr depth_listzipreversed)rZ src_node_nameZ dst_node_namerrrdZsrc_device_nameZdst_device_namerrrrrrhZ curr_depthrdepthrrrfind_some_path|s:(        zDebugDumpDir.find_some_pathcCs@|jstd|||}|j|}|r2|j|S|j|SdS)a^Get recipient of the given node's output according to partition graphs. Args: node_name: (`str`) name of the node. is_control: (`bool`) whether control outputs, rather than non-control outputs, are to be returned. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: (`list` of `str`) all inputs to the node, as a list of node names. Raises: LookupError: If node inputs and control inputs have not been loaded from partition graphs yet. z9Node recipients are not loaded from partition graphs yet.N)rrrZnode_ctrl_recipientsr)rrBrrdrrrrrs   zDebugDumpDir.node_recipientscCs|jS)z^Get the list of device names. Returns: (`list` of `str`) names of the devices. )r)rrrrrszDebugDumpDir.devicescCsV|jstd|dk r,||jkr,td|x$|jD]\}}||jkr8dSq8WdS)aTest if a node exists in the partition graphs. Args: node_name: (`str`) name of the node to be checked. device_name: optional device name. If None, will search for the node on all available devices. Otherwise, search for the node only on the given device. Returns: A boolean indicating whether the node exists. Raises: LookupError: If no partition graphs have been loaded yet. ValueError: If device_name is specified but cannot be found. z5Nodes have not been loaded from partition graphs yet.Nz/The specified device_name '%s' cannot be found.TF)rrrwitemsr)rrBrdrNrrrr node_existss  zDebugDumpDir.node_existscCsJ|jstd||jkr$td|t|j|}t|dkrF|dS|S)aGet the names of the devices that has nodes of the specified name. Args: node_name: (`str`) name of the node. Returns: (`str` or `list` of `str`) name of the device(s) on which the node of the given name is found. Returns a `str` if there is only one such device, otherwise return a `list` of `str`. Raises: LookupError: If node inputs and control inputs have not been loaded from partition graphs yet. ValueError: If the node does not exist in partition graphs. z6Node devices are not loaded from partition graphs yet.z-Node '%s' does not exist in partition graphs.r-r)rrrrwrrj)rrBrrrr node_device s zDebugDumpDir.node_devicecCs*|jstd|||}|j|j|S)aGet the op type of given node. Args: node_name: (`str`) name of the node. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: (`str`) op type of the node. Raises: LookupError: If node op types have not been loaded from partition graphs yet. z7Node op types are not loaded from partition graphs yet.)rrrr)rrBrdrrrr's  zDebugDumpDir.node_op_typecCsy|||}Wntk r$gSX||j|kr8gSg}xH|j||D]6}|j|||}x|D]}|t|||qhWqLW|S)a-Get all tensor watch keys of given node according to partition graphs. Args: node_name: (`str`) name of the node. device_name: (`str`) name of the device. If there is only one device or if node_name exists on only one device, this argument is optional. Returns: (`list` of `str`) all debug tensor watch keys. Returns an empty list if the node name does not correspond to any debug watch keys. Raises: `LookupError`: If debug watch information has not been loaded from partition graphs yet. )rrwrrrF)rrBrdZ watch_keysZ watched_slotZ debug_opsrErrrdebug_watch_keys=s zDebugDumpDir.debug_watch_keyscsz|dkrRfddjD}|s&gSt|dkr<|d}qhtdt|fn|jkrhtd|j|gS)adGet all `DebugTensorDatum` instances corresponding to a debug watch key. Args: debug_watch_key: (`str`) debug watch key. device_name: (`str`) name of the device. If there is only one device or if the specified debug_watch_key exists on only one device, this argument is optional. Returns: A list of `DebugTensorDatum` instances that correspond to the debug watch key. If the watch key does not exist, returns an empty list. Raises: ValueError: If there are multiple devices that have the debug_watch_key, but device_name is not specified. Ncsg|]}j|kr|qSr)r)r]r)debug_watch_keyrrrr_rsz2DebugDumpDir.watch_key_to_data..r-rz[The debug watch key '%s' exists on multiple (%d) devices, but device name is not specified.z=There is no device named '%s' consisting of debug watch keys.)rrjrwZ_debug_key_to_datumr)rr rdZmatching_device_namesr)r rrwatch_key_to_data_s   zDebugDumpDir.watch_key_to_datacCs|rt|}g}xx|dkr"|jn |j|fD]Z}xT|j|D]F}|rT||jrTq>|||r>|||dkr>t||kr>|Sq>Wq.W|S)a$Find dumped tensor data by a certain predicate. Args: predicate: A callable that takes two input arguments: ```python def predicate(debug_tensor_datum, tensor): # returns a bool ``` where `debug_tensor_datum` is an instance of `DebugTensorDatum`, which carries the metadata, such as the `Tensor`'s node name, output slot timestamp, debug op name, etc.; and `tensor` is the dumped tensor value as a `numpy.ndarray`. first_n: (`int`) return only the first n `DebugTensotDatum` instances (in time order) for which the predicate returns True. To return all the `DebugTensotDatum` instances, let first_n be <= 0. device_name: optional device name. exclude_node_names: Optional regular expression to exclude nodes with names matching the regular expression. Returns: A list of all `DebugTensorDatum` objects in this `DebugDumpDir` object for which predicate returns True, sorted in ascending order of the timestamp. Nr)recompilermatchrBrrrj)r predicateZfirst_nrdZexclude_node_namesZ matched_datarrMrrrrzs   zDebugDumpDir.findcCsN|||}t|||}||j|kr6td||fdd|j||DS)aGet the file paths from a debug-dumped tensor. Args: node_name: (`str`) name of the node that the tensor is produced by. output_slot: (`int`) output slot index of tensor. debug_op: (`str`) name of the debug op. device_name: (`str`) name of the device. If there is only one device or if the specified debug_watch_key exists on only one device, this argument is optional. Returns: List of file path(s) loaded. This is a list because each debugged tensor may be dumped multiple times. Raises: WatchKeyDoesNotExistInDebugDumpDirError: If the tensor does not exist in the debug-dump data. z.)rrFrr)rrBrCrErdrrrrget_tensor_file_pathss   z"DebugDumpDir.get_tensor_file_pathsc Cs^t|||}y$|||}dd|j||DSttfk rXtd||fYnXdS)aGet the tensor value from for a debug-dumped tensor. The tensor may be dumped multiple times in the dump root directory, so a list of tensors (`numpy.ndarray`) is returned. Args: node_name: (`str`) name of the node that the tensor is produced by. output_slot: (`int`) output slot index of tensor. debug_op: (`str`) name of the debug op. device_name: (`str`) name of the device. If there is only one device or if the specified debug_watch_key exists on only one device, this argument is optional. Returns: List of tensors (`numpy.ndarray`) loaded from the debug-dump file(s). Raises: WatchKeyDoesNotExistInDebugDumpDirError: If the tensor does not exist in the debug-dump data. cSsg|] }|qSr)r)r]rMrrrr_sz,DebugDumpDir.get_tensors..zs     zDebugDumpDir.node_traceback)NT)r)N)N)FN)TFN)TFN)FN)N)N)N)N)rNN)N)N)N)N)2rrr r!rrrrrrrrrrr"rrrrrrrrrrrrrrrrrrrrrrrrrr rr r rzrrrrrrrrrrsh 6 ! " 2 F   "    ( I    " ' 1  $ % r)4r!rrrVrgr r numpyrHZtensorflow.core.frameworkrrZtensorflow.core.utilrZtensorflow.python.debug.librZtensorflow.python.frameworkrZtensorflow.python.platformrrrZtensorflow.python.utilr r:rr;rbZHASH_TAGr>r@rrr,r(r5r8r=r?rArDrFrO namedtuplerXrYrerlrmrwrrrrrrsX        &   :