B ٻdDP@sdZddlZddlZddlZddlZddlmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd lm Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZedgdeddddZedgdeddddZedgdeddddZedgdeddd d!Z dBd"d#Z!d$d%Z"d&d'Z#d(d)Z$d*d+Z%d,d-Z&d.d/Z'd0d1Z(d2d3Z)d4d5Z*d6d7Z+d8d9Z,dCd;d<Z-Gd=d>d>ej.Z/e 0d?Gd@dAdAe j1Z2e 3e4de2j5dS)Dz;Utilities for describing the structure of a `tf.data` type.N)nest)composite_tensor)ops) sparse_tensor) tensor_shape) tensor_spec) type_spec)resource_variable_ops)tensor_array_ops) ragged_tensor) tf_logging) deprecation)collections_abc) tf_exportz!data.experimental.TensorStructure)Zv1zUse `tf.TensorSpec` instead.cCs t||S)N)r TensorSpec)dtypeshaperW/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/data/util/structure.py_TensorStructure'srz'data.experimental.SparseTensorStructurez"Use `tf.SparseTensorSpec` instead.cCs t||S)N)rSparseTensorSpec)rrrrr_SparseTensorStructure-srz&data.experimental.TensorArrayStructurez!Use `tf.TensorArraySpec` instead.cCst||||S)N)r TensorArraySpec)rZ element_shape dynamic_size infer_shaperrr_TensorArrayStructure3srz'data.experimental.RaggedTensorStructurez"Use `tf.RaggedTensorSpec` instead.cCst|||S)N)r RaggedTensorSpec)rrZ ragged_rankrrr_RaggedTensorStructure:src Csg}|dkr*t|}dgt|}|}nt|}t||}|}tdlddlm}xVtt ||D]B\}\}} y| dkrt |dd} Wn,t k r| tj |d|dYqpXt| tjr| tj|qpt| tjr | tj|d|dqpt| tj|jfr,| |qpt| trF| tqpt| tjrt| tj |d || jd qpt|tjr| |qpt | d d} | tj |d|| d qpWWdQRXt!||S) aNormalizes a nested structure of element components. * Components matching `SparseTensorSpec` are converted to `SparseTensor`. * Components matching `RaggedTensorSpec` are converted to `RaggedTensor`. * Components matching `VariableSpec` are converted to `Tensor`. * Components matching `DatasetSpec` or `TensorArraySpec` are passed through. * `CompositeTensor` components are passed through. * All other components are converted to `Tensor`. Args: element: A nested structure of individual components. element_signature: (Optional.) A nested structure of `tf.DType` objects corresponding to each component of `element`. If specified, it will be used to set the exact type of output tensor when converting input components which are not tensors themselves (e.g. numpy arrays, native python types, etc.) Returns: A nested structure of `Tensor`, `Variable`, `Dataset`, `SparseTensor`, `RaggedTensor`, or `TensorArray` objects. Nnormalize_elementr) dataset_opsF) use_fallbackz component_%d)nameZ component_)r!rr)"rflattenlenZ flatten_up_torZ name_scopeZtensorflow.python.data.opsr enumerateziptype_spec_from_value TypeErrorappendconvert_to_tensor isinstancerr SparseTensor from_valuer rZ"convert_to_tensor_or_ragged_tensorr rZ DatasetSpecNoneTensorSpec NoneTensorr Z VariableSpecrrCompositeTensorgetattrpack_sequence_as) elementZelement_signatureZnormalized_components componentsZflattened_signatureZpack_asritspecrrrrrCsJ          $rc Cst|}t|}t|}g}xt|||D]\}}} t| tjrR|| q0t| tj rr|t ||q0t| t j r|t ||q0t| tjr|tj|dd|t|dt|ddq0td| jq0Wt||S)aReturns a `Structure` that represents the given legacy structure. This method provides a way to convert from the existing `Dataset` and `Iterator` structure-related properties to a `Structure` object. A "legacy" structure is represented by the `tf.data.Dataset.output_types`, `tf.data.Dataset.output_shapes`, and `tf.data.Dataset.output_classes` properties. TODO(b/110122868): Remove this function once `Structure` is used throughout `tf.data`. Args: output_types: A nested structure of `tf.DType` objects corresponding to each component of a structured value. output_shapes: A nested structure of `tf.TensorShape` objects corresponding to each component a structured value. output_classes: A nested structure of Python `type` objects corresponding to each component of a structured value. Returns: A `Structure`. Raises: TypeError: If a structure cannot be built from the arguments, because one of the component classes in `output_classes` is not supported. Nr)rrzCould not build a structure for output class {}. Make sure any component class in `output_classes` inherits from one of the following classes: `tf.TypeSpec`, `tf.sparse.SparseTensor`, `tf.Tensor`, `tf.TensorArray`.)rr"r%r*rZTypeSpecr( issubclassrr+rrZTensorrrr Z TensorArrayrrZdimension_valuer'format__name__r1) Z output_typesZ output_shapesZoutput_classesZ flat_typesZ flat_shapesZ flat_classesflat_retZ flat_typeZ flat_shapeZ flat_classrrrconvert_legacy_structures,          r=c Cst|}dd|D}t|t|kr@tdt|t|d}g}x>t||D]0\}}||||} |||| ||7}qTWt||S)a]Returns an element constructed from the given spec and tensor list. Args: decode_fn: Method that constructs an element component from the element spec component and a tensor list. element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. tensor_list: A list of tensors to use for constructing the value. Returns: An element constructed from the given spec and tensor list. Raises: ValueError: If the number of tensors needed to construct an element for the given spec does not match the given number of tensors. cSsg|]}t|jqSr)r#_flat_tensor_specs).0r6rrr sz,_from_tensor_list_helper..zExpected {} tensors but got {}.r) rr"sumr# ValueErrorr:r%r(r1) Z decode_fn element_spec tensor_listZ flat_specsZflat_spec_lengthsr4r<Zcomponent_specZnum_flat_valuesvaluerrr_from_tensor_list_helpers  rFcCstdd||S)aReturns an element constructed from the given spec and tensor list. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. tensor_list: A list of tensors to use for constructing the value. Returns: An element constructed from the given spec and tensor list. Raises: ValueError: If the number of tensors needed to construct an element for the given spec does not match the given number of tensors. cSs ||S)N)Z_from_compatible_tensor_list)r6rErrrz-from_compatible_tensor_list..)rF)rCrDrrrfrom_compatible_tensor_listsrIcCstdd||S)a(Returns an element constructed from the given spec and tensor list. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. tensor_list: A list of tensors to use for constructing the value. Returns: An element constructed from the given spec and tensor list. Raises: ValueError: If the number of tensors needed to construct an element for the given spec does not match the given number of tensors or the given spec is not compatible with the tensor list. cSs ||S)N)Z_from_tensor_list)r6rErrrrGrHz"from_tensor_list..)rF)rCrDrrrfrom_tensor_listsrJcCs ttjddt|DS)a Returns a list `tf.TypeSpec`s for the element tensor representation. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. Returns: A list `tf.TypeSpec`s for the element tensor representation. css|] }|jVqdS)N)r>)r?r6rrr !sz(get_flat_tensor_specs..)list itertoolschain from_iterablerr")rCrrrget_flat_tensor_specss rPcCsddt|DS)aReturns a list `tf.TensorShapes`s for the element tensor representation. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. Returns: A list `tf.TensorShapes`s for the element tensor representation. cSsg|] }|jqSr)r)r?r6rrrr@.sz*get_flat_tensor_shapes..)rP)rCrrrget_flat_tensor_shapes$s rQcCsddt|DS)aReturns a list `tf.DType`s for the element tensor representation. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. Returns: A list `tf.DType`s for the element tensor representation. cSsg|] }|jqSr)r)r?r6rrrr@;sz)get_flat_tensor_types..)rP)rCrrrget_flat_tensor_types1s rRcs8t||fdd}t|tt|t|gS)aReturns a tensor list representation of the element. Args: encode_fn: Method that constructs a tensor list representation from the given element spec and element. element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. element: The element to convert to tensor list representation. Returns: A tensor list representation of `element`. Raises: ValueError: If `element_spec` and `element` do not have the same number of elements or if the two structures are not nested in the same way. TypeError: If `element_spec` and `element` differ in the type of sequence in any of their substructures. cs|\}}|||S)Nr)staterEr6 component) encode_fnrr reduce_fnTsz)_to_tensor_list_helper..reduce_fn)rassert_same_structure functoolsreducer%r")rUrCr2rVr)rUr_to_tensor_list_helper>s  rZcCstdd||S)aReturns a tensor list representation of the element. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. element: The element to convert to tensor list representation. Returns: A tensor list representation of `element`. Raises: ValueError: If `element_spec` and `element` do not have the same number of elements or if the two structures are not nested in the same way or the rank of any of the tensors in the tensor list representation is 0. TypeError: If `element_spec` and `element` differ in the type of sequence in any of their substructures. cSs|||S)N)_to_batched_tensor_list)rSr6rTrrrrGrsz(to_batched_tensor_list..)rZ)rCr2rrrto_batched_tensor_list\sr\cCstdd||S)aCReturns a tensor list representation of the element. Args: element_spec: A nested structure of `tf.TypeSpec` objects representing to element type specification. element: The element to convert to tensor list representation. Returns: A tensor list representation of `element`. Raises: ValueError: If `element_spec` and `element` do not have the same number of elements or if the two structures are not nested in the same way. TypeError: If `element_spec` and `element` differ in the type of sequence in any of their substructures. cSs|||S)N)_to_tensor_list)rSr6rTrrrrGrHz to_tensor_list..)rZ)rCr2rrrto_tensor_listvsr^cCsxyt||Wn&tk r$dStk r6dSXx:tt|t|D] \}}||rl||sPdSqPWdS)aIndicates whether two type specifications are compatible. Two type specifications are compatible if they have the same nested structure and the their individual components are pair-wise compatible. Args: spec1: A `tf.TypeSpec` object to compare. spec2: A `tf.TypeSpec` object to compare. Returns: `True` if the two type specifications are compatible and `False` otherwise. FT)rrWr'rBr%r"Zis_compatible_with)Zspec1Zspec2s1s2rrrare_compatibles raTc st}|dk r|SttjrZttjr<fdd}nt}|ddDStt rt drtj tj rt ddj Drttjrtj}nt}|d dDSt d dDSt jd rtjd }tfd d|DS|rvy$t}t|}|dk r0|SWnBttfk rt}ztd dtj|fWdd}~XYnXtdtjdS)aCreates a type specification for the given value. Args: element: The element to create the type specification for. use_fallback: Whether to fall back to converting the element to a tensor in order to compute its `TypeSpec`. Returns: A nested structure of `TypeSpec`s that represents the type specification of `element`. Raises: TypeError: If a `TypeSpec` cannot be built for `element`, because its type is not supported. Ncstj|S)N)typedefault_factory)items)r2rrrGrHz&type_spec_from_value..cSsg|]\}}|t|fqSr)r&)r?kvrrrr@sz(type_spec_from_value.._fieldscss|]}t|tVqdS)N)r*str)r?frrrrKsz'type_spec_from_value..cSsg|] }t|qSr)r&)r?rfrrrr@scSsg|] }t|qSr)r&)r?rfrrrr@sZ__attrs_attrs__csg|]}tt|jqSr)r&r0r!)r?a)r2rrr@sz"Failed to convert %r to tensor: %sz0Could not build a `TypeSpec` for {} with type {})rZ_type_spec_from_valuer*rMapping collections defaultdictrbrdtuplehasattrrgSequenceallwraptZ ObjectProxy __wrapped__ __class__r0rr)r&rBr'loggingZvlogr;r:)r2r r6ZctorZ element_typeattrsZtensorer)r2rr&sB          (r&c@seZdZdZeddZdS)r.z1Composite tensor representation for `None` value.cCstS)N)r-)selfrrr _type_specszNoneTensor._type_specN)r; __module__ __qualname____doc__propertyrzrrrrr.sr.ztf.NoneTensorSpecc@seZdZdZeddZddZeddZdd Zd d Z d d Z e ddZ ddZ ddZddZddZddZddZddZdS)r-z$Type specification for `None` value.cCstS)N)r.)ryrrr value_typeszNoneTensorSpec.value_typecCsdS)Nrr)ryrrr _serializeszNoneTensorSpec._serializecCsgS)Nr)ryrrr_component_specsszNoneTensorSpec._component_specscCsgS)Nr)ryrErrr_to_components szNoneTensorSpec._to_componentscCsdS)Nr)ryr3rrr_from_componentsszNoneTensorSpec._from_componentscCsgS)Nr)ryrErrrr]szNoneTensorSpec._to_tensor_listcCstS)N)r-)rErrrr,szNoneTensorSpec.from_valuecCstS)N)r-)ryZ batch_sizerrr_batchszNoneTensorSpec._batchcCstS)N)r-)ryrrr_unbatchszNoneTensorSpec._unbatchcCsgS)Nr)ryrErrrr[sz&NoneTensorSpec._to_batched_tensor_listcCs|S)Nr)ryrrr_to_legacy_output_types!sz&NoneTensorSpec._to_legacy_output_typescCs|S)Nr)ryrrr_to_legacy_output_shapes$sz'NoneTensorSpec._to_legacy_output_shapescCs|S)Nr)ryrrr_to_legacy_output_classes'sz(NoneTensorSpec._to_legacy_output_classescCs$t|t|k r td|||S)Nz/No `TypeSpec` is compatible with both {} and {})rbrBr:)ryotherrrrmost_specific_compatible_shape*s z-NoneTensorSpec.most_specific_compatible_shapeN)r;r{r|r}r~rrrrrr] staticmethodr,rrr[rrrrrrrrr-s   r-)N)T)6r}rmrXrMrsZtensorflow.python.data.utilrZtensorflow.python.frameworkrrrrrrZtensorflow.python.opsr r Ztensorflow.python.ops.raggedr Ztensorflow.python.platformr rvZtensorflow.python.utilr Ztensorflow.python.util.compatrZ tensorflow.python.util.tf_exportr deprecatedrrrrrr=rFrIrJrPrQrRrZr\r^rar&r/r.registerZBatchableTypeSpecr-Z'register_type_spec_from_value_converterrbr,rrrrsX                   C;#   H  4