B ٻdf@s4dZddlZddlmZddlmZddlmZddlmZ ddl m Z ddl m Z e d Gd d d ejZe d Gd ddejZe dGdddejZe dGddde jZe dGddde jZe dGddde jZe de ddGddde jZe dGd d!d!e jZdS)"z%API for specifying `tf.data` options.N)logging)dataset_options_pb2) model_pb2)options) deprecation) tf_exportz#data.experimental.AutotuneAlgorithmc@s<eZdZdZdZdZdZdZdZe ddZ e d d Z d S) AutotuneAlgorithmaRepresents the type of autotuning algorithm to use. DEFAULT: The default behavior is implementation specific and may change over time. HILL_CLIMB: In each optimization step, this algorithm chooses the optimial parameter and increases its value by 1. GRADIENT_DESCENT: In each optimization step, this algorithm updates the parameter values in the optimal direction. MAX_PARALLELISM: Similar to HILL_CLIMB but uses a relaxed stopping condition, allowing the optimization to oversubscribe the CPU. STAGE_BASED: In each optimization step, this algorithm chooses the worst bottleneck parameter and increases its value by 1. rcCsp||jkrtjjS||jkr$tjjS||jkr6tjjS||jkrHtjjS||jkrZtjjStd|jddS)NzkInvalid `obj.` Supported values include `DEFAULT`, `HILL_CLIMB` `GRADIENT_DESCENT`, and `STAGE_BASED`. Got .) DEFAULTrr HILL_CLIMBGRADIENT_DESCENTMAX_PARALLELISM STAGE_BASED ValueErrorname)clsobjrT/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/options.py _to_proto5s     zAutotuneAlgorithm._to_protocCsn|tjjkr|jS|tjjkr$|jS|tjjkr6|jS|tjjkrH|jS|tjjkrZ|jStd|ddS)NzjInvalid `pb.` Supported values include `DEFAULT`, `HILL_CLIMB`, `GRADIENT_DESCENT` and `STAGE_BASED`. Got r )rrrrrrrr)rpbrrr _from_protoEs     zAutotuneAlgorithm._from_protoN) __name__ __module__ __qualname____doc__rrrrr classmethodrrrrrrrs rz!data.experimental.AutoShardPolicyc@s<eZdZdZdZdZdZdZdZe ddZ e d d Z d S) AutoShardPolicyaTRepresents the type of auto-sharding to use. OFF: No sharding will be performed. AUTO: Attempts FILE-based sharding, falling back to DATA-based sharding. FILE: Shards by input files (i.e. each worker will get a set of files to process). When this option is selected, make sure that there is at least as many files as workers. If there are fewer input files than workers, a runtime error will be raised. DATA: Shards by elements produced by the dataset. Each worker will process the whole dataset and discard the portion that is not for itself. Note that for this mode to correctly partitions the dataset elements, the dataset needs to produce elements in a deterministic order. HINT: Looks for the presence of `shard(SHARD_HINT, ...)` which is treated as a placeholder to replace with `shard(num_workers, worker_index)`. rr r r cCsp||jkrtjjS||jkr$tjjS||jkr6tjjS||jkrHtjjS||jkrZtjjStd|jddS)zConvert enum to proto.zVInvalid `obj.` Supported values include `OFF`, `FILE`, `DATA`,`AUTO`, and `HINT`. Got r N) OFFrr!FILEDATAAUTOHINTrr)rrrrrrts     zAutoShardPolicy._to_protocCsn|tjjkr|jS|tjjkr$|jS|tjjkr6|jS|tjjkrH|jS|tjjkrZ|jStd|ddS)zConvert proto to enum.zUInvalid `pb.` Supported values include `OFF`, `FILE`, `DATA`,`AUTO`, and `HINT`. Got r N)rr!r#r$r%r&r'r)rrrrrrs     zAutoShardPolicy._from_protoN) rrrrr#r&r$r%r'r rrrrrrr!Vs r!z%data.experimental.ExternalStatePolicyc@s4eZdZdZdZdZdZeddZeddZ d S) ExternalStatePolicyzRepresents how to handle external state during serialization. See the `tf.data.Options.experimental_external_state_policy` documentation for more information. rr r cCsL||jkrtjjS||jkr$tjjS||jkr6tjjStd|j ddS)zConvert enum to proto.zZInvalid `obj.` Supported values include `POLICY_IGNORE`,`POLICY_FAIL`, `POLICY_WARN`. Got r N) IGNORErr( POLICY_IGNOREFAIL POLICY_FAILWARN POLICY_WARNrr)rrrrrrs   zExternalStatePolicy._to_protocCsJ|tjjkr|jS|tjjkr$|jS|tjjkr6|jStd|ddS)zConvert proto to enum.zYInvalid `pb.` Supported values include `POLICY_IGNORE`,`POLICY_FAIL`, `POLICY_WARN`. Got r N) rr(r*r)r,r+r.r-r)rrrrrrs   zExternalStatePolicy._from_protoN) rrrrr-r)r+r rrrrrrr(s  r(z!data.experimental.AutotuneOptionsc@sheZdZdZejdeddZejdeddZ ejdeddZ ejd e d dZ d d Z d dZddZdS)AutotuneOptionszRepresents options for autotuning dataset performance. ```python options = tf.data.Options() options.autotune.enabled = False dataset = dataset.with_options(options) ``` enabledzKWhether to automatically tune performance knobs. If None, defaults to True.)rty docstring cpu_budgetzWhen autotuning is enabled (through `autotune`), determines the CPU budget to use. Values greater than the number of schedulable CPU cores are allowed but may result in CPU contention. If None, defaults to the number of schedulable CPU cores. ram_budgetzWhen autotuning is enabled (through `autotune`), determines the RAM budget to use. Values greater than the available RAM in bytes may result in OOM. If None, defaults to half of the available RAM in bytes.autotune_algorithmzQWhen autotuning is enabled (through `autotune`), determines the algorithm to use.cCsZt}|jdk r|j|_|jdk r,|j|_|jdk r>|j|_|jdk rVt|j|_|S)N)rr/r0r3r4r5rr)selfrrrrrs     zAutotuneOptions._to_protocCsb|ddk r|j|_|ddk r,|j|_|ddk rB|j|_|ddk r^t|j|_dS)NZoptional_enabledZoptional_cpu_budgetZoptional_ram_budgetZoptional_autotune_algorithm) WhichOneofr0r3r4rrr5)r6rrrrrszAutotuneOptions._from_protocCst|d|dS)zFChange the mutability value to `mutable` on this options and children._mutableN)object __setattr__)r6mutablerrr _set_mutableszAutotuneOptions._set_mutableN)rrrr options_lib create_optionboolr0intr3r4rr5rrr<rrrrr/s(   r/z#data.experimental.DistributeOptionsc@sFeZdZdZejdeddddZejdedd Z d d Z d d Z dS)DistributeOptionsaRepresents options for distributed data processing. You can set the distribution options of a dataset through the `experimental_distribute` property of `tf.data.Options`; the property is an instance of `tf.data.experimental.DistributeOptions`. ```python options = tf.data.Options() options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF dataset = dataset.with_options(options) ``` auto_shard_policyzcThe type of sharding to use. See `tf.data.experimental.AutoShardPolicy` for additional information.cCstjS)N)r!r&rrrrzDistributeOptions.)rr1r2default_factory num_deviceszoThe number of devices attached to this input pipeline. This will be automatically set by `MultiDeviceIterator`.)rr1r2cCs,t}t|j|_|jdk r(|j|_|S)N)rrAr!rrBrF)r6rrrrr!s  zDistributeOptions._to_protocCs(t|j|_|ddk r$|j|_dS)NZoptional_num_devices)r!rrBr7rF)r6rrrrr(szDistributeOptions._from_protoN) rrrrr=r>r!rBr@rFrrrrrrrAs  rAz%data.experimental.OptimizationOptionsc@seZdZdZejdeddZejdeddZejdeddZ ejd ed dZ ejd ed dZ ejd eddZ ejdeddZ ejdeddZejdeddZejdeddZejdeddZddZddZddZdS) OptimizationOptionsaRepresents options for dataset optimizations. You can set the optimization options of a dataset through the `experimental_optimization` property of `tf.data.Options`; the property is an instance of `tf.data.experimental.OptimizationOptions`. ```python options = tf.data.Options() options.experimental_optimization.noop_elimination = True options.experimental_optimization.apply_default_optimizations = False dataset = dataset.with_options(options) ``` apply_default_optimizationszWhether to apply default graph optimizations. If False, only graph optimizations that have been explicitly enabled will be applied.)rr1r2 filter_fusionzCWhether to fuse filter transformations. If None, defaults to False.filter_parallelizationzTWhether to parallelize stateless filter transformations. If None, defaults to False.inject_prefetchzWhether to inject prefetch transformation as the last transformation when the last transformation is a synchronous transformation. If None, defaults to True.map_and_batch_fusionzIWhether to fuse map and batch transformations. If None, defaults to True.map_and_filter_fusionzKWhether to fuse map and filter transformations. If None, defaults to False. map_fusionz@Whether to fuse map transformations. If None, defaults to False.map_parallelizationzPWhether to parallelize stateless map transformations. If None, defaults to True.noop_eliminationzFWhether to eliminate no-op transformations. If None, defaults to True.parallel_batchzLWhether to parallelize copying of batch elements. If None, defaults to True.shuffle_and_repeat_fusionzNWhether to fuse shuffle and repeat transformations. If None, defaults to True.cCst}|jdk r|j|_|jdk r,|j|_|jdk r>|j|_|jdk rP|j|_|jdk rb|j|_|jdk rt|j|_|jdk r|j|_|j dk r|j |_ |j dk r|j |_ |j dk r|j |_ |j dk r|j |_ |S)N) rrGrHrIrJrKrLrMrNrOrPrQrR)r6rrrrrs0           zOptimizationOptions._to_protocCs|ddk r|j|_|ddk r,|j|_|ddk rB|j|_|ddk rX|j|_|ddk rn|j|_|ddk r|j|_|ddk r|j|_|ddk r|j|_|d dk r|j |_ |d dk r|j |_ |d dk r|j |_ dS) NZ$optional_apply_default_optimizationsZoptional_filter_fusionZoptional_filter_parallelizationZoptional_inject_prefetchZoptional_map_and_batch_fusionZoptional_map_and_filter_fusionZoptional_map_fusionZoptional_map_parallelizationZoptional_noop_eliminationZoptional_parallel_batchZ"optional_shuffle_and_repeat_fusion) r7rHrIrJrKrLrMrNrOrPrQrR)r6rrrrrs,zOptimizationOptions._from_protocCst|d|dS)zFChange the mutability value to `mutable` on this options and children.r8N)r9r:)r6r;rrrr<sz OptimizationOptions._set_mutableN)rrrrr=r>r?rHrIrJrKrLrMrNrOrPrQrRrrr<rrrrrG.s`rGz"data.experimental.ThreadingOptionszdata.ThreadingOptionsc@s@eZdZdZejdeddZejdeddZddZ d d Z d S) ThreadingOptionsamRepresents options for dataset threading. You can set the threading options of a dataset through the `experimental_threading` property of `tf.data.Options`; the property is an instance of `tf.data.ThreadingOptions`. ```python options = tf.data.Options() options.threading.private_threadpool_size = 10 dataset = dataset.with_options(options) ``` max_intra_op_parallelismz@If set, it overrides the maximum degree of intra-op parallelism.)rr1r2private_threadpool_sizezIf set, the dataset will use a private threadpool of the given size. The value 0 can be used to indicate that the threadpool size should be determined at runtime based on the number of available CPU cores.cCs0t}|jdk r|j|_|jdk r,|j|_|S)N)rrSrTrU)r6rrrrrs   zThreadingOptions._to_protocCs0|ddk r|j|_|ddk r,|j|_dS)NZ!optional_max_intra_op_parallelismZ optional_private_threadpool_size)r7rTrU)r6rrrrrszThreadingOptions._from_protoN) rrrrr=r>r@rTrUrrrrrrrSsrSz data.OptionscseZdZdZejdededZejdeddZ ejded dZ ejd e d e dZ ejd e d dZejdededZejdeddZejdeddZejdededZfddZfddZddZddZddZd d!ZZS)"OptionsalRepresents options for `tf.data.Dataset`. A `tf.data.Options` object can be, for instance, used to control which static optimizations to apply to the input pipeline graph or whether to use performance modeling to dynamically tune the parallelism of operations such as `tf.data.Dataset.map` or `tf.data.Dataset.interleave`. The options are set for the entire dataset and are carried over to datasets created through tf.data transformations. The options can be set by constructing an `Options` object and using the `tf.data.Dataset.with_options(options)` transformation, which returns a dataset with the options set. >>> dataset = tf.data.Dataset.range(42) >>> options = tf.data.Options() >>> options.deterministic = False >>> dataset = dataset.with_options(options) >>> print(dataset.options().deterministic) False Note: A known limitation of the `tf.data.Options` implementation is that the options are not preserved across tf.function boundaries. In particular, to set options for a dataset that is iterated within a tf.function, the options need to be set within the same tf.function. autotunezpThe autotuning options associated with the dataset. See `tf.data.experimental.AutotuneOptions` for more details.)rr1r2rE deterministiczZWhether the outputs need to be produced in deterministic order. If None, defaults to True.)rr1r2experimental_deterministicz(DEPRECATED. Use `deterministic` instead.experimental_distributez}The distribution strategy options associated with the dataset. See `tf.data.experimental.DistributeOptions` for more details."experimental_external_state_policyaNThis option can be used to override the default policy for how to handle external state when serializing a dataset or checkpointing its iterator. There are three settings available - IGNORE: External state is ignored without a warning; WARN: External state is ignored and a warning is logged; FAIL: External state results in an error.experimental_optimizationzvThe optimization options associated with the dataset. See `tf.data.experimental.OptimizationOptions` for more details.experimental_slacka+Whether to introduce 'slack' in the last `prefetch` of the input pipeline, if it exists. This may reduce CPU contention with accelerator host-side activity at the start of a step. The slack frequency is determined by the number of devices attached to this input pipeline. If None, defaults to False.experimental_threadingz$DEPRECATED. Use `threading` instead. threadingzcThe threading options associated with the dataset. See `tf.data.ThreadingOptions` for more details.cs>|dkrtdt|dS|dkr.t|dStt||S)Nr^zLoptions.experimental_threading is deprecated. Use options.threading instead.r_rYrX)rwarninggetattrsuperrV__getattribute__)r6r) __class__rrrcJs    zOptions.__getattribute__cs\|dkr(tdtt|d|dS|dkrFtt|d|dStt|||dS)Nr^zLoptions.experimental_threading is deprecated. Use options.threading instead.r_rYrX)rr`rbrVr:)r6rvalue)rdrrr:Vs zOptions.__setattr__cCst}|jdk r|j|_|j|j|j|j|j dk rVt |j |_ |j |j |jdk rz|j|_|j|j|S)N)rrVrXautotune_optionsZCopyFromrWrdistribute_optionsrZr[r(external_state_policyoptimization_optionsr\r]slackthreading_optionsr_)r6rrrrrds    zOptions._to_protocCs|ddk r|j|_|j|j|j|j|ddk rNt|j|_ |j |j |ddk rr|j |_ |j|jdS)NZoptional_deterministicZoptional_external_state_policyZoptional_slack)r7rXrWrrfrZrgr(rhr[r\rirjr]r_rk)r6rrrrrts zOptions._from_protocCsBt|d||j||j||j||j|dS)zFChange the mutability value to `mutable` on this options and children.r8N)r9r:rWr<rZr\r_)r6r;rrrr<s    zOptions._set_mutablecCs t||S)aMerges itself with the given `tf.data.Options`. If this object and the `options` to merge set an option differently, a warning is generated and this object's value is updated with the `options` object's value. Args: options: The `tf.data.Options` to merge with. Returns: New `tf.data.Options` object which is the result of merging self with the input `tf.data.Options`. )r=Z merge_options)r6rrrrmergesz Options.merge)rrrrr=r>r/rWr?rXrYrArZr(r[rGr\r]rSr^r_rcr:rrr<rl __classcell__rr)rdrrVs^  rV)renumZabslrZtensorflow.core.frameworkrrZtensorflow.python.data.utilrr=Ztensorflow.python.utilrZ tensorflow.python.util.tf_exportrEnumrIntEnumr!r(Z OptionsBaser/rArGZdeprecated_endpointsrSrVrrrrs2       9 B % D )  +