B һdޑ@s:dZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl m Z ddl mZdd lmZdd lmZdd lmZd d ZGdddZGdddZGdddZGdddZGdddZGdddZGdddZGdddZGdddejejd Z Gd!d"d"Z!Gd#d$d$e Z"dS)%anFramework of debug wrapper sessions. A debug wrapper session is a wrapper around a TensorFlow Python Session. The wrapper preserves the Session interface, most importantly the run() method, while providing abilities to: a) Intercept a run() call to a wrapped session and insert debug tensor watches according to externally-specified debug URLs. b) Release control to an external (i.e., non-Session) object before and after the run() call, so that the external object can perform actions such as launching a UI to let users inspect the intermediate tensors and partition graphs from the run() call. c) (To be implemented in a future CL) Enter an instruction loop to let an external object (e.g., remote client) launch run() and cont() calls remotely. *** The lifetime of a debug wrapper session: *** 1) The wrapper session is created by calling the constructor with a wrapped (normal) session as the argument: wrapper = FooDebugWrapperSession(sess) wherein FooDebugWrapperSession is a concrete subclass implementing the abstract BaseDebugWrapperSession class below. 2) Near the end of the constructor call, the on_session_init() callback is invoked, with a OnSessionInitRequest object as the argument. The object carries the wrapped (normal) session object. 3) The callback handles the request and returns a OnSessionInitResponse object with an action field, directing the wrapper session what to do next. If the action field in the OnSessionInitResponse is PROCEED, the constructor returns. Control is released back to the caller of the constructor, which can invoke run() method of wrapper session with the same syntax as a non-wrapped session, e.g.,: wrapper.run(fetches, feed_dict=feeds, options=run_options) Below, A1 - A2 is the lifetime of a wrapper run() call if the action is PROCEED: A1) Right at the start of each run() call, the on_run_start() callback is invoked, with an OnRunStartRequest object carrying information such as the fetches, the feed dict, the run options and run metadata used in this run call, along with a count of how many run calls has occurred on this wrapper session. The callback then returns an OnRunStartResponse object, of which the action field directs what the wrapper session actually will do of the run() call. If the action is DEBUG_RUN, a debugged (tensor-watched) run will ensue, with the debug URLs supplied in the debug_urls field of the response. These can be file:// or grpc:// URLs, for example. If the action is NON_DEBUG_RUN, a non-debug (normal) run will ensue. A2) Right before the run() returns, the on_run_end() callback is invoked, with an OnRunEndRequest object as the argument, which carries information including the actual action performed in the wrapper run() call and the run_metadata from the run() call. However, if the action field in OnSessionInitResponse is REMOTE_INSTR_LOOP, the constructor will automatically invoke an instruction loop that gives the control to a remote caller. In the remote instruction loop, the following steps will happen: B1) Callback on_instr_start() is invoked. The callback will return an OnInstrStartResponse object with an action field which can order one of the following actions: i) a run() call with fetches, feeds and debug_urls specified. ii) exit the instruction loop. B2) The wrapper session carries out the action specified above. B3) If still in the instruction loop, the wrapper session invokes the on_instr_end() callback. After the on_instr_end() callback returns, jump back to B1. TODO(cais): Implemented the instruction loop in B1 - B3. N) config_pb2)session) debug_utils)errors)ops) tf_logging)monitored_session)nest)collections_abccCs"t||std|t|fdS)zCheck if an object is of the expected type. Args: obj: The object being checked. expected_types: (`type` or an iterable of `type`s) The expected `type`(s) of obj. Raises: TypeError: If obj is not an instance of expected_type. zExpected type %s; got type %sN) isinstance TypeErrortype)objZexpected_typesr\/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/debug/wrappers/framework.py _check_typeqs rc@seZdZdZddZdS)OnSessionInitRequestz{Request to an on-session-init callback. This callback is invoked during the __init__ call to a debug-wrapper session. cCst|tjtjf||_dS)zEConstructor. Args: sess: A tensorflow Session object. N)rr BaseSessionrMonitoredSession)selfsessrrr__init__szOnSessionInitRequest.__init__N)__name__ __module__ __qualname____doc__rrrrrrsrc@seZdZdZdZdZdS)OnSessionInitActionz=Enum-like values for possible action to take on session init.proceedZremote_instr_loopN)rrrrPROCEEDREMOTE_INSTR_LOOPrrrrrsrc@seZdZdZddZdS)OnSessionInitResponsez*Response from an on-session-init callback.cCst|t||_dS)zkConstructor. Args: action: (`OnSessionInitAction`) Debugger action to take on session init. N)rstraction)rr"rrrrs zOnSessionInitResponse.__init__N)rrrrrrrrrr sr c@seZdZdZdddZdS)OnRunStartRequestzRequest to an on-run-start callback. This callback is invoked during a run() call of the debug-wrapper session, immediately after the run() call counter is incremented. FcCs(||_||_||_||_||_||_dS)aeConstructor of `OnRunStartRequest`. Args: fetches: Fetch targets of the run() call. feed_dict: The feed dictionary to the run() call. run_options: RunOptions input to the run() call. run_metadata: RunMetadata input to the run() call. The above four arguments are identical to the input arguments to the run() method of a non-wrapped TensorFlow session. run_call_count: 1-based count of how many run calls (including this one) has been invoked. is_callable_runner: (bool) whether a runner returned by Session.make_callable is being run. N)fetches feed_dict run_options run_metadatarun_call_countis_callable_runner)rr$r%r&r'r(r)rrrrs zOnRunStartRequest.__init__N)F)rrrrrrrrrr#sr#c@seZdZdZdZdZdZdS)OnRunStartActionzFEnum-like values for possible action to take on start of a run() call.Z debug_runZ profile_runZ non_debug_runN)rrrr DEBUG_RUN PROFILE_RUN NON_DEBUG_RUNrrrrr*sr*c@seZdZdZdddZdS)OnRunStartResponsezRequest from an on-run-start callback. The caller of the callback can use this response object to specify what action the debug-wrapper session actually takes on the run() call. DebugIdentityNFcCsBt|t||_t|t||_||_||_||_||_||_ dS)aConstructor of `OnRunStartResponse`. Args: action: (`OnRunStartAction`) the action actually taken by the wrapped session for the run() call. debug_urls: (`list` of `str`) debug_urls used in watching the tensors during the run() call. debug_ops: (`str` or `list` of `str`) Debug op(s) to be used by the debugger. node_name_regex_allowlist: Regular-expression allowlist for node name. op_type_regex_allowlist: Regular-expression allowlist for op type. tensor_dtype_regex_allowlist: Regular-expression allowlist for tensor dtype. tolerate_debug_op_creation_failures: Whether debug op creation failures are to be tolerated. N) rr!r"list debug_urls debug_opsnode_name_regex_allowlistop_type_regex_allowlisttensor_dtype_regex_allowlist#tolerate_debug_op_creation_failures)rr"r1r2r3r4r5r6rrrrs  zOnRunStartResponse.__init__)r/NNNF)rrrrrrrrrr.s r.c@seZdZdZdddZdS)OnRunEndRequestzpRequest to an on-run-end callback. The callback is invoked immediately before the wrapped run() call ends. NcCs:t|t||_|dk r$t|tj||_||_||_dS)aConstructor for `OnRunEndRequest`. Args: performed_action: (`OnRunStartAction`) Actually-performed action by the debug-wrapper session. run_metadata: run_metadata output from the run() call (if any). client_graph_def: (GraphDef) GraphDef from the client side, i.e., from the python front end of TensorFlow. Can be obtained with session.graph.as_graph_def(). tf_error: (errors.OpError subtypes) TensorFlow OpError that occurred during the run (if any). N)rr!performed_actionr RunMetadatar'client_graph_deftf_error)rr8r'r:r;rrrrs  zOnRunEndRequest.__init__)NNN)rrrrrrrrrr7 sr7c@seZdZdZddZdS)OnRunEndResponsez%Response from an on-run-end callback.cCsdS)Nr)rrrrr1szOnRunEndResponse.__init__N)rrrrrrrrrr<.sr<c@s2eZdZdZdAddZeddZedd Zed d Zed d Z dBddZ ddZ ddZ ddZ ddZdCddZdDddZddZddZdEd d!Zd"d#Zed$d%Zd&d'Zd(d)ZdFd+d,Zd-d.Zejd/d0Zejd1d2Zejd3d4Zd5d6Zd7d8Z d9d:Z!d;d<Z"d=d>Z#d?d@Z$dS)GBaseDebugWrapperSessionzBase class of debug-wrapper session classes. Concrete classes that inherit from this class need to implement the abstract methods such as on_session_init, on_run_start and on_run_end. NFcCst|tjtjf||_|r&t|nd|_||_ d|_ | t |j}t|t |jtjkr`n$|jtjkrvtdntd|jd|_i|_dS)aConstructor of `BaseDebugWrapperSession`. Args: sess: An (unwrapped) TensorFlow session instance. It should be a subtype of `BaseSession` or `tf.MonitoredSession`. thread_name_filter: Regular-expression filter (allowlist) for name(s) of thread(s) on which the wrapper session will be active. This regular expression is used in a start-anchored fashion on the thread name, i.e., by applying the `match` method of the compiled pattern. The default `None` means that the wrapper session will be active on all threads. E.g., r"MainThread$", r"QueueRunnerThread.*". pass_through_operrors: If True, all captured OpErrors will be propagated. By default this captures all OpErrors. Raises: ValueError: On invalid `OnSessionInitAction` value. NotImplementedError: If a non-DirectSession sess object is received. Nrz?OnSessionInitAction REMOTE_INSTR_LOOP has not been implemented.z%Invalid OnSessionInitAction value: %s)rrrrr_sessrecompile_thread_name_filter_pattern_pass_through_operrors_run_call_counton_session_initrr r"rrrNotImplementedError ValueError _default_session_context_manager_cached_callables_from_options)rrthread_name_filterpass_through_operrorsresponserrrr>s     z BaseDebugWrapperSession.__init__cCs|jjS)N)r>graph)rrrrrLvszBaseDebugWrapperSession.graphcCs|jjS)N)r> graph_def)rrrrrMzsz!BaseDebugWrapperSession.graph_defcCs|jjS)N)r>sess_str)rrrrrN~sz BaseDebugWrapperSession.sess_strcCs|jS)N)r>)rrrrrszBaseDebugWrapperSession.sessionc s|r|rtd|r&|s|r&tdn|r:|s2|r:td|fdd|}|rdtd|sp|r|r|||S|r|j||S|jj||||dS|t |||||j t |d} t | t | jtjkr|| |||||||\} } n| jtjkr$|| |||||||\} } nh| jtjkr~|rB||} n0|r^|j|} | |} n|jj||||d} t| j} ntd | j|| } t | t| S) abWrapper around Session.run() that inserts tensor watch options. Args: fetches: Same as the `fetches` arg to regular `Session.run()`. feed_dict: Same as the `feed_dict` arg to regular `Session.run()`. options: Same as the `options` arg to regular `Session.run()`. run_metadata: Same as the `run_metadata` arg to regular `Session.run()`. callable_runner: A `callable` returned by `Session.make_callable()`. If not `None`, `fetches` and `feed_dict` must both be `None`. Mutually exclusive with `callable_options`. callable_runner_args: An optional list of arguments to `callable_runner` or for `callable_options`. callable_options: An instance of `config_pb2.CallableOptions`, to be used with `Session._make_callable_from_options()`. Mutually exclusive with `callable_runner`. Returns: Simply forwards the output of the wrapped `Session.run()` call. Raises: ValueError: On invalid `OnRunStartAction` value. Or if `callable_runner` is not `None` and either or both of `fetches` and `feed_dict` is `None`. zcallable_runner and callable_options are mutually exclusive, but are both specified in this call to BaseDebugWrapperSession.run().zZcallable_runner and fetches/feed_dict are mutually exclusive, but are used simultaneously.z[callable_options and fetches/feed_dict are mutually exclusive, but are used simultaneously.csHt|sdSt|tjr*t|Sx|D]}|s0dSq0WdS)z3Check whether a possibly nested structure is empty.FT)r Z is_nestedr r Mappingr0values)xitem)is_emptyrrrSs   z-BaseDebugWrapperSession.run..is_emptyzpDue to empty fetches, tfdbg Session wrapper is letting a Session.run pass through without any debugging actions.)r%optionsr')r)z"Invalid OnRunStartAction value: %s)rFincrement_run_call_countrinfo_is_disabled_threadr>_make_callable_from_optionsrun on_run_startr#rCboolrr.r"r*r+_run_with_debuggingr,_run_with_profilingr-r7 on_run_endr<)rr$r%rTr'callable_runnercallable_runner_argscallable_optionsZ empty_fetchesrun_start_respretvalsZ run_end_reqcallable_objectZ run_end_respr)rSrrYsp                zBaseDebugWrapperSession.runc  Cs@d} |r4t|} | |jkr@t} | || j} n |p>t} |pJt}| rt|j| |j |j |j |j |j |jdd} yn|r||| |d} nT|r| |jkr|j| }n|j| }||j| <||d|i} n|jj||| |d} Wn:tjk r }z|jr||} |} Wdd}~XYnX| t|j||jj| dfS)z3Perform a session.run() or callable with debugging.N)r2r3r4r5r6)rTr'r')r%rTr')r'r:r;)idrHrCallableOptionsCopyFromr& RunOptionsr9_decorate_run_options_for_debugr1r2r3r4r5r6r>rXrYrZOpErrorrBr7r"rL as_graph_def)rrbr$r%rTr'r_r`radecorated_run_optionscallable_options_idnew_callable_optionsr;rcrdZop_errorrrrr\s^           z+BaseDebugWrapperSession._run_with_debuggingc Csd} |r4t|} | |jkr@t} | || j} n |p>t} || |pTt}|rl||| |d} n4|r|j | } | |d|i} n|j j ||| |d} | t |j ||j jdfS)z3Perform a session.run() or callable with profiling.N)rTr'r')r%rTr')r'r:)rerHrrfrgr&rh!_decorate_run_options_for_profiler9r>rXrYr7r"rLrj)rrbr$r%rTr'r_r`rarkrlrmrcrdrrrr]Js6       z+BaseDebugWrapperSession._run_with_profilingcCs"tjp d}|jo |j| S)N) threadingcurrent_threadnamerAmatch)rZ thread_namerrrrWwsz+BaseDebugWrapperSession._is_disabled_threadcCs|tj|j|jS)N)rrZ StepContextr>rY)rZstep_fnrrr run_step_fn|sz#BaseDebugWrapperSession.run_step_fncCs tddS)z>Sets up the feeds and fetches for partial runs in the session.z@partial_run_setup is not implemented for debug-wrapper sessions.N)rE)rr$Zfeedsrrrpartial_run_setupsz)BaseDebugWrapperSession.partial_run_setupcCs tddS)Nz:partial_run is not implemented for debug-wrapper sessions.)rE)rhandler$r%rrr partial_runsz#BaseDebugWrapperSession.partial_runcOs|jj||S)N)r> list_devices)rargskwargsrrrrxsz$BaseDebugWrapperSession.list_devicescOs|jj||S)N)r>reset)rryrzrrrr{szBaseDebugWrapperSession.resetcs$jj||ddfdd}|S)NT) feed_listaccept_optionscs&jdd|dd|dd|dS)NrTr')r%rTr'r_r`)rYget)Z runner_argsrz)runnerrrrwrapped_runners   z=BaseDebugWrapperSession.make_callable..wrapped_runner)r> make_callable)rr$r|r}rr)rrrrs z%BaseDebugWrapperSession.make_callablecsfdd}|S)Ncsjd|dd|dS)Nr')r'rar`)rYr~)Z feed_valuesrz)rarrrrs zKBaseDebugWrapperSession._make_callable_from_options..wrapped_runnerr)rrarr)rarrrXsz3BaseDebugWrapperSession._make_callable_from_optionscCs|jS)N)rC)rrrrr(sz&BaseDebugWrapperSession.run_call_countcCs|jd7_dS)N)rC)rrrrrUsz0BaseDebugWrapperSession.increment_run_call_countcCsdS)aIndicates whether disk usage is reset after each Session.run. Subclasses that clean up the disk usage after every run should override this protected method. Returns: (`bool`) Whether the disk usage amount is reset to zero after each Session.run. Fr)rrrr_is_disk_usage_reset_each_runs z5BaseDebugWrapperSession._is_disk_usage_reset_each_runr/c Cs8d|_tj||jj|||||||jdkp.|d dS)aModify a RunOptions object for debug tensor watching. Specifies request for outputting partition graphs. Adds debug_tensor_watch_opts with proper debug URLs. Args: run_options: (RunOptions) the modified RunOptions object. debug_urls: (list of str) debug URLs to be entered in run_options. debug_tensor_watch_opts. debug_ops: (str or list of str) debug op(s) to be used by the debugger. node_name_regex_allowlist: Regular-expression allowlist for node name. op_type_regex_allowlist: Regular-expression allowlist for op type. tensor_dtype_regex_allowlist: Regular-expression allowlist for tensor dtype. tolerate_debug_op_creation_failures: Whether debug op creation failures are to be tolerated. Tr)r1r2r3r4r5r6Zreset_disk_byte_usageN)Zoutput_partition_graphsrZ watch_graphr>rLrCr)rr&r1r2r3r4r5r6rrrris z7BaseDebugWrapperSession._decorate_run_options_for_debugcCstjj|_dS)zModify a RunOptions object for profiling TensorFlow graph execution. Args: run_options: (RunOptions) the modified RunOptions object. N)rrhZ FULL_TRACEZ trace_level)rr&rrrrnsz9BaseDebugWrapperSession._decorate_run_options_for_profilecCsdS)agCallback invoked during construction of the debug-wrapper session. This is a blocking callback. The invocation happens right before the constructor ends. Args: request: (`OnSessionInitRequest`) callback request carrying information such as the session being wrapped. Returns: An instance of `OnSessionInitResponse`. Nr)rrequestrrrrDs z'BaseDebugWrapperSession.on_session_initcCsdS)aZCallback invoked on run() calls to the debug-wrapper session. This is a blocking callback. The invocation happens after the wrapper's run() call is entered, after an increment of run call counter. Args: request: (`OnRunStartRequest`) callback request object carrying information about the run call such as the fetches, feed dict, run options, run metadata, and how many `run()` calls to this wrapper session have occurred. Returns: An instance of `OnRunStartResponse`, carrying information to debug URLs used to watch the tensors. Nr)rrrrrrZsz$BaseDebugWrapperSession.on_run_startcCsdS)aCallback invoked on run() calls to the debug-wrapper session. This is a blocking callback. The invocation happens right before the wrapper exits its run() call. Args: request: (`OnRunEndRequest`) callback request object carrying information such as the actual action performed by the session wrapper for the run() call. Returns: An instance of `OnRunStartResponse`. Nr)rrrrrr^ sz"BaseDebugWrapperSession.on_run_endcCs t|S)N)rZdefault_session)rrrr as_defaultsz"BaseDebugWrapperSession.as_defaultcCs|jdkr||_|jS)N)rGr __enter__)rrrrr s  z!BaseDebugWrapperSession.__enter__cCs|j|||dS)N)rG__exit__)rZ exec_typeZ exec_valueZexec_tbrrrr%sz BaseDebugWrapperSession.__exit__cCst|jdr|jdS)N__del__)hasattrr>r)rrrrr)s zBaseDebugWrapperSession.__del__cCs|jdS)N)r>close)rrrrr-szBaseDebugWrapperSession.closecCs(t|jdr|jStd|jdS)N should_stopzxThe wrapped session %r does not have a method called 'should_stop'. Do you intend to wrap a tf.MonitoredSession instead?)rr>rrF)rrrrr3s   z#BaseDebugWrapperSession.should_stop)NF)NNNNNN)N)N)NF)r/NNNF)%rrrrrpropertyrLrMrNrrYr\r]rWrtrurwrxr{rrXr(rUrrirnabcabstractmethodrDrZr^rrrrrrrrrrr=7sT 7     sJ-     ! r=) metaclassc@s"eZdZdZdddZddZdS) WatchOptionsz#Type for return values of watch_fn.NFcCs0|r ||_ndg|_||_||_||_||_dS)a(Constructor of WatchOptions: Debug watch options. Used as return values of `watch_fn`s. Args: debug_ops: (`str` or `list of str`) Debug ops to be used. node_name_regex_allowlist: Regular-expression allowlist for node_name, e.g., `"(weight_[0-9]+|bias_.*)"` op_type_regex_allowlist: Regular-expression allowlist for the op type of nodes, e.g., `"(Variable|Add)"`. If both `node_name_regex_allowlist` and `op_type_regex_allowlist` are set, the two filtering operations will occur in a logical `AND` relation. In other words, a node will be included if and only if it hits both allowlists. tensor_dtype_regex_allowlist: Regular-expression allowlist for Tensor data type, e.g., `"^int.*"`. This allowlist operates in logical `AND` relations to the two allowlists above. tolerate_debug_op_creation_failures: (`bool`) whether debug op creation failures (e.g., due to dtype incompatibility) are to be tolerated by not throwing exceptions. r/N)r2r3r4r5r6)rr2r3r4r5r6rrrr?szWatchOptions.__init__cCsd|j|j|j|j|jfS)NzWatchOptions(debug_ops=%r, node_name_regex_allowlist=%r, op_type_regex_allowlist=%r, tensor_dtype_regex_allowlist=%r, tolerate_debug_op_creation_failures=%r))r2r3r4r5r6)rrrr__repr__eszWatchOptions.__repr__)NNNNF)rrrrrrrrrrr<s !rc@sHeZdZdZdddZddZejdd Zd d Z d d Z ddZ dS)!NonInteractiveDebugWrapperSessionzFBase class for non-interactive (i.e., non-CLI) debug wrapper sessions.NFcCs:tj||||dd|_|dk r6t|s0td||_dS)aAConstructor of NonInteractiveDebugWrapperSession. Args: sess: The TensorFlow `Session` object being wrapped. watch_fn: (`Callable`) A Callable that maps the fetches and feeds of a debugged `Session.run()` call to `WatchOptions.` * Args: * `fetches`: the fetches to the `Session.run()` call. * `feeds`: the feeds to the `Session.run()` call. * Returns: (`tf_debug.WatchOptions`) An object containing debug options including the debug ops to use, the node names, op types and/or tensor data types to watch, etc. See the documentation of `tf_debug.WatchOptions` for more details. thread_name_filter: Regular-expression white list for threads on which the wrapper session will be active. See doc of `BaseDebugWrapperSession` for more details. pass_through_operrors: If true, all captured OpErrors will be propagated. By default this captures all OpErrors. Raises: TypeError: If a non-None `watch_fn` is specified and it is not callable. )rIrJNzwatch_fn is not callable)r=r _watch_fncallabler )rrZwatch_fnrIrJrrrrqsz*NonInteractiveDebugWrapperSession.__init__cCs ttjS)z0See doc of BaseDebugWrapperSession.on_run_start.)r rr)rrrrrrDsz1NonInteractiveDebugWrapperSession.on_session_initcCsdS)aAbstract method to be implemented by concrete subclasses. This method prepares the run-specific debug URL(s). Args: fetches: Same as the `fetches` argument to `Session.run()` feed_dict: Same as the `feed_dict` argument to `Session.run()` Returns: debug_urls: (`str` or `list` of `str`) Debug URLs to be used in this `Session.run()` call. Nr)rr$r%rrrprepare_run_debug_urlss z8NonInteractiveDebugWrapperSession.prepare_run_debug_urlsc Cs6||j|j\}}ttj||j|j|j|j |j dS)z0See doc of BaseDebugWrapperSession.on_run_start.)r2r3r4r5r6) _prepare_run_watch_configr$r%r.r*r+r2r3r4r5r6)rrr1Z watch_optsrrrrZsz.NonInteractiveDebugWrapperSession.on_run_startcCsD|||}|jdkrt}n|||}t|tr_s8          0"  2