B ٻdf@s dZddlZddlmZddlmZddlmZddlmZddlm Z ddl m Z dd l m Z dd l m Z dd l mZdd l mZdd lmZddlmZddlmZddlmZddlmZdZddZd3ddZd4ddZddZGdddejZedgdGdd d ejZ edgdGd!d"d"ej!Z"Gd#d$d$ejZ#Gd%d&d&ej$Z%ed'gdGd(d)d)ej&Z'ed'gdGd*d+d+ej!Z(Gd,d-d-ejZ)ed.gdGd/d0d0ejZ*ed.gdGd1d2d2ej!Z+e,re*Z-e'Z.e Z/n e+Z-e(Z.e"Z/dS)5z$Python wrappers for reader Datasets.N)tf2) dataset_ops)from_tensor_slices_op)structured_function)convert)dtypes)ops) tensor_shape) tensor_spec) type_spec) array_ops)gen_dataset_ops)gen_experimental_dataset_ops)nest) tf_exporticCst|tjrt|S|S)zHConvert pathlib-like objects to str (__fspath__ compatibility, PEP 519).) isinstanceosPathLikefspath)pathrT/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/data/ops/readers.py_normalise_fspath%srcCst|tjr\t|}|tjkr0td|dt|}|t gstd|dn\t t |}tj|tjd}|jtjkrtd|jdtj|dgdd }tj|d |d }|S) a8Creates (or validates) a dataset of filenames. Args: filenames: Either a list or dataset of filenames. If it is a list, it is convert to a dataset. If it is a dataset, its type and shape is validated. name: (Optional.) A name for the tf.data operation. Returns: A dataset of filenames. zNThe `filenames` argument must contain `tf.string` elements. Got a dataset of `z ` elements.zvThe `filenames` argument must contain `tf.string` elements of shape [] (i.e. scalars). Got a dataset of element shape .)Z dtype_hintzAThe `filenames` argument must contain `tf.string` elements. Got `Zflat_filenames)nameT)Zis_filesr)rr DatasetV2Zget_legacy_output_typesrstring TypeErrorZget_legacy_output_shapesZis_compatible_withr Z TensorShaperZ map_structurerrconvert_to_tensordtyper ZreshaperZTensorSliceDataset) filenamesrZ element_typeZ element_shaperrr%_create_or_validate_filenames_dataset*s$       r"c sXfdd}|dkr"|j||dS|tjkr<|j|||dSt|||dddd|dSdS) aCreates a dataset that reads the given files using the given reader. Args: dataset_creator: A function that takes in a single file name and returns a dataset. filenames: A `tf.data.Dataset` containing one or more filenames. num_parallel_reads: The number of parallel reads we should do. name: (Optional.) A name for the tf.data operation. Returns: A `Dataset` that reads data from `filenames`. cstj|tjdd}|S)Nfilename)r)rrrr)r#)dataset_creatorrr read_one_file_sz-_create_dataset_reader..read_one_fileN)r)Znum_parallel_callsrF) cycle_length block_lengthsloppybuffer_output_elementsprefetch_input_elementsr)Zflat_maprZAUTOTUNEZ interleaveParallelInterleaveDataset)r$r!num_parallel_readsrr%r)r$r_create_dataset_readerNs   r.cCs t|tjr|St|SdS)z0Returns the type of `value` if it is a TypeSpec.N)rr ZTypeSpecZ value_typetype)valuerrr _get_typets r1cs.eZdZdZdfdd ZeddZZS)_TextLineDatasetz;A `Dataset` comprising records from one or more text files.Ncsf||_tjd|dtjd|_tjd|td|_||_t j |j|j|j|j d}t t||dS)aCreates a `TextLineDataset`. Args: filenames: A `tf.string` tensor containing one or more filenames. compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. buffer_size: (Optional.) A `tf.int64` scalar denoting the number of bytes to buffer. A value of 0 results in the default buffering values chosen based on the compression type. name: (Optional.) A name for the tf.data operation. compression_type)argument_defaultargument_dtype buffer_size)r5)metadataN) _filenamesroptional_param_to_tensorrr_compression_type!_DEFAULT_READER_BUFFER_SIZE_BYTES _buffer_size_namer Ztext_line_dataset _metadataSerializeToStringsuperr2__init__)selfr!r3r7rvariant_tensor) __class__rrrBs"  z_TextLineDataset.__init__cCstgtjS)N)r TensorSpecrr)rCrrr element_specsz_TextLineDataset.element_spec)NNN)__name__ __module__ __qualname____doc__rBpropertyrG __classcell__rr)rErr2}s r2zdata.TextLineDataset)Zv1cs.eZdZdZdfdd ZeddZZS)TextLineDatasetV2aCreates a `Dataset` comprising lines from one or more text files. The `tf.data.TextLineDataset` loads text from text files and creates a dataset where each line of the files becomes an element of the dataset. For example, suppose we have 2 files "text_lines0.txt" and "text_lines1.txt" with the following lines: >>> with open('/tmp/text_lines0.txt', 'w') as f: ... f.write('the cow\n') ... f.write('jumped over\n') ... f.write('the moon\n') >>> with open('/tmp/text_lines1.txt', 'w') as f: ... f.write('jack and jill\n') ... f.write('went up\n') ... f.write('the hill\n') We can construct a TextLineDataset from them as follows: >>> dataset = tf.data.TextLineDataset(['/tmp/text_lines0.txt', ... '/tmp/text_lines1.txt']) The elements of the dataset are expected to be: >>> for element in dataset.as_numpy_iterator(): ... print(element) b'the cow' b'jumped over' b'the moon' b'jack and jill' b'went up' b'the hill' Ncs\t|d}||_|_|_fdd}t|||d|_|jj}tt| |dS)aCreates a `TextLineDataset`. The elements of the dataset will be the lines of the input files, using the newline character '\n' to denote line splits. The newline characters will be stripped off of each element. Args: filenames: A `tf.data.Dataset` whose elements are `tf.string` scalars, a `tf.string` tensor, or a value that can be converted to a `tf.string` tensor (such as a list of Python strings). compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. buffer_size: (Optional.) A `tf.int64` scalar denoting the number of bytes to buffer. A value of 0 results in the default buffering values chosen based on the compression type. num_parallel_reads: (Optional.) A `tf.int64` scalar representing the number of files to read in parallel. If greater than one, the records of files read in parallel are outputted in an interleaved order. If your input pipeline is I/O bottlenecked, consider setting this parameter to a value greater than one to parallelize the I/O. If `None`, files will be read sequentially. name: (Optional.) A name for the tf.data operation. )rcst|dS)N)r)r2)r#)r7r3rrr creator_fnsz.TextLineDatasetV2.__init__..creator_fnN) r"r9r;r=r._impl_variant_tensorrArNrB)rCr!r3r7r-rrOrD)rE)r7r3rrrBs zTextLineDatasetV2.__init__cCstgtjS)N)r rFrr)rCrrrrGszTextLineDatasetV2.element_spec)NNNN)rHrIrJrKrBrLrGrMrr)rErrNs "'rNcsFeZdZdZdfdd Zejje_eddZejddZZ S) TextLineDatasetV1z9A `Dataset` comprising lines from one or more text files.Ncs$t|||||}tt||dS)N)rNrArRrB)rCr!r3r7r-rwrapped)rErrrBszTextLineDatasetV1.__init__cCs|jjS)N)_datasetr9)rCrrrr9 szTextLineDatasetV1._filenamescCs ||j_dS)N)rTr9)rCr0rrrr9s)NNNN) rHrIrJrKrBrNrLr9setterrMrr)rErrRs  rRcs.eZdZdZdfdd ZeddZZS)_TFRecordDatasetz?A `Dataset` comprising records from one or more TFRecord files.Ncsf||_tjd|dtjd|_tjd|td|_||_t j |j|j|j|j d}t t||dS)aCreates a `TFRecordDataset`. Args: filenames: A `tf.string` tensor containing one or more filenames. compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. buffer_size: (Optional.) A `tf.int64` scalar representing the number of bytes in the read buffer. 0 means no buffering. name: (Optional.) A name for the tf.data operation. r3r4)r5r6r7)r5)r8N)r9rr:rrr;r<r=r>r Ztf_record_datasetr?r@rArVrB)rCr!r3r7rrD)rErrrBs   z_TFRecordDataset.__init__cCstgtjS)N)r rFrr)rCrrrrG9sz_TFRecordDataset.element_spec)NNN)rHrIrJrKrBrLrGrMrr)rErrVs rVcs>eZdZdZd fdd ZddZeddZd d ZZ S) r,zHA `Dataset` that maps a function over its input and flattens the result.Nc s||_tj|||d|_t|jjtjsDt dt |jjd|jjj |_ t j |tjdd|_t j |tjdd|_tjd|d|d |_tjd |d|d |_|d krd |_n|rd |_nd|_||_tj|jj|jjj|j|j|j|jf|jj|jd|j} tt| || d S)z=See `tf.data.experimental.parallel_interleave()` for details.)Zdatasetzged_opsZ%legacy_parallel_interleave_dataset_v2rQfunctionZcaptured_inputsZ _common_argsrAr,rB) rCZ input_datasetZmap_funcr'r(r)r*r+rrD)rErrrBAsJ    z"ParallelInterleaveDataset.__init__cCs|jgS)N)r])rCrrr _functionstsz$ParallelInterleaveDataset._functionscCs|jS)N)r^)rCrrrrGwsz&ParallelInterleaveDataset.element_speccCsdS)Nz*tf.data.experimental.parallel_interleave()r)rCrrrr\{sz.ParallelInterleaveDataset._transformation_name)N) rHrIrJrKrBrbrLrGr\rMrr)rErr,>s  + r,zdata.TFRecordDatasetcs6eZdZdZd fdd ZddZeddZZS) TFRecordDatasetV2aA `Dataset` comprising records from one or more TFRecord files. This dataset loads TFRecords from the files as bytes, exactly as they were written.`TFRecordDataset` does not do any parsing or decoding on its own. Parsing and decoding can be done by applying `Dataset.map` transformations after the `TFRecordDataset`. A minimal example is given below: >>> import tempfile >>> example_path = os.path.join(tempfile.gettempdir(), "example.tfrecords") >>> np.random.seed(0) >>> # Write the records to a file. ... with tf.io.TFRecordWriter(example_path) as file_writer: ... for _ in range(4): ... x, y = np.random.random(), np.random.random() ... ... record_bytes = tf.train.Example(features=tf.train.Features(feature={ ... "x": tf.train.Feature(float_list=tf.train.FloatList(value=[x])), ... "y": tf.train.Feature(float_list=tf.train.FloatList(value=[y])), ... })).SerializeToString() ... file_writer.write(record_bytes) >>> # Read the data back out. >>> def decode_fn(record_bytes): ... return tf.io.parse_single_example( ... # Data ... record_bytes, ... ... # Schema ... {"x": tf.io.FixedLenFeature([], dtype=tf.float32), ... "y": tf.io.FixedLenFeature([], dtype=tf.float32)} ... ) >>> for batch in tf.data.TFRecordDataset([example_path]).map(decode_fn): ... print("x = {x:.4f}, y = {y:.4f}".format(**batch)) x = 0.5488, y = 0.7152 x = 0.6028, y = 0.5449 x = 0.4237, y = 0.6459 x = 0.4376, y = 0.8918 Ncsbt|d}||_|_|_||_fdd}t|||d|_|jj}tt | |dS)aCreates a `TFRecordDataset` to read one or more TFRecord files. Each element of the dataset will contain a single TFRecord. Args: filenames: A `tf.string` tensor or `tf.data.Dataset` containing one or more filenames. compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. buffer_size: (Optional.) A `tf.int64` scalar representing the number of bytes in the read buffer. If your input pipeline is I/O bottlenecked, consider setting this parameter to a value 1-100 MBs. If `None`, a sensible default for both local and remote file systems is used. num_parallel_reads: (Optional.) A `tf.int64` scalar representing the number of files to read in parallel. If greater than one, the records of files read in parallel are outputted in an interleaved order. If your input pipeline is I/O bottlenecked, consider setting this parameter to a value greater than one to parallelize the I/O. If `None`, files will be read sequentially. name: (Optional.) A name for the tf.data operation. Raises: TypeError: If any argument does not have the expected type. ValueError: If any argument does not have the expected shape. )rcst|dS)N)r)rV)r#)r7r3rrrrOsz.TFRecordDatasetV2.__init__..creator_fnN) r"r9r;r=Z_num_parallel_readsr.rPrQrArcrB)rCr!r3r7r-rrOrD)rE)r7r3rrrBs zTFRecordDatasetV2.__init__cCs |jS)N)rP_inputs)rCrrrrdszTFRecordDatasetV2._inputscCstgtjS)N)r rFrr)rCrrrrGszTFRecordDatasetV2.element_spec)NNNN) rHrIrJrKrBrdrLrGrMrr)rErrcs+*rccsFeZdZdZdfdd Zejje_eddZejddZZ S) TFRecordDatasetV1z?A `Dataset` comprising records from one or more TFRecord files.Ncs&t|||||d}tt||dS)N)r)rcrArerB)rCr!r3r7r-rrS)rErrrBszTFRecordDatasetV1.__init__cCs|jjS)N)rTr9)rCrrrr9szTFRecordDatasetV1._filenamescCs ||j_dS)N)rTr9)rCr0rrrr9s)NNNN) rHrIrJrKrBrcrLr9rUrMrr)rErres  recs.eZdZdZdfdd ZeddZZS)_FixedLengthRecordDatasetzBA `Dataset` of fixed-length records from one or more binary files.Nc s||_tj|tjdd|_td||_td||_ td|t |_ tjd|dtj d|_ ||_tj|j|j|j|j |j |j |jd }tt||d S) a Creates a `FixedLengthRecordDataset`. Args: filenames: A `tf.string` tensor containing one or more filenames. record_bytes: A `tf.int64` scalar representing the number of bytes in each record. header_bytes: (Optional.) A `tf.int64` scalar representing the number of bytes to skip at the start of a file. footer_bytes: (Optional.) A `tf.int64` scalar representing the number of bytes to ignore at the end of a file. buffer_size: (Optional.) A `tf.int64` scalar representing the number of bytes to buffer when reading. compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. name: (Optional.) A name for the tf.data operation. record_bytes)r r header_bytes footer_bytesr7r3r4)r5r6)r8N)r9rrrr_ _record_bytesrr: _header_bytes _footer_bytesr<r=rr;r>r Zfixed_length_record_dataset_v2r?r@rArfrB) rCr!rgrhrir7r3rrD)rErrrBs0    z"_FixedLengthRecordDataset.__init__cCstgtjS)N)r rFrr)rCrrrrG1sz&_FixedLengthRecordDataset.element_spec)NNNNN)rHrIrJrKrBrLrGrMrr)rErrfs+rfzdata.FixedLengthRecordDatasetcs.eZdZdZdfdd ZeddZZS)FixedLengthRecordDatasetV2aA `Dataset` of fixed-length records from one or more binary files. The `tf.data.FixedLengthRecordDataset` reads fixed length records from binary files and creates a dataset where each record becomes an element of the dataset. The binary files can have a fixed length header and a fixed length footer, which will both be skipped. For example, suppose we have 2 files "fixed_length0.bin" and "fixed_length1.bin" with the following content: >>> with open('/tmp/fixed_length0.bin', 'wb') as f: ... f.write(b'HEADER012345FOOTER') >>> with open('/tmp/fixed_length1.bin', 'wb') as f: ... f.write(b'HEADER6789abFOOTER') We can construct a `FixedLengthRecordDataset` from them as follows: >>> dataset1 = tf.data.FixedLengthRecordDataset( ... filenames=['/tmp/fixed_length0.bin', '/tmp/fixed_length1.bin'], ... record_bytes=2, header_bytes=6, footer_bytes=6) The elements of the dataset are: >>> for element in dataset1.as_numpy_iterator(): ... print(element) b'01' b'23' b'45' b'67' b'89' b'ab' Nc stt|d}||_|_|_|_|_|_fdd} t| ||d|_|jj } t t | | dS)aCreates a `FixedLengthRecordDataset`. Args: filenames: A `tf.string` tensor or `tf.data.Dataset` containing one or more filenames. record_bytes: A `tf.int64` scalar representing the number of bytes in each record. header_bytes: (Optional.) A `tf.int64` scalar representing the number of bytes to skip at the start of a file. footer_bytes: (Optional.) A `tf.int64` scalar representing the number of bytes to ignore at the end of a file. buffer_size: (Optional.) A `tf.int64` scalar representing the number of bytes to buffer when reading. compression_type: (Optional.) A `tf.string` scalar evaluating to one of `""` (no compression), `"ZLIB"`, or `"GZIP"`. num_parallel_reads: (Optional.) A `tf.int64` scalar representing the number of files to read in parallel. If greater than one, the records of files read in parallel are outputted in an interleaved order. If your input pipeline is I/O bottlenecked, consider setting this parameter to a value greater than one to parallelize the I/O. If `None`, files will be read sequentially. name: (Optional.) A name for the tf.data operation. )rc st|dS)N)r)rf)r#)r7r3rirhrrgrrrOsz7FixedLengthRecordDatasetV2.__init__..creator_fnN) r"r9rjrkrlr=r;r.rPrQrArmrB) rCr!rgrhrir7r3r-rrOrD)rE)r7r3rirhrrgrrBYs  z#FixedLengthRecordDatasetV2.__init__cCstgtjS)N)r rFrr)rCrrrrGsz'FixedLengthRecordDatasetV2.element_spec)NNNNNN)rHrIrJrKrBrLrGrMrr)rErrm6s!0rmcsFeZdZdZdfdd Zejje_eddZejddZZ S) FixedLengthRecordDatasetV1zBA `Dataset` of fixed-length records from one or more binary files.Nc s,t||||||||d} tt|| dS)N)r)rmrArnrB) rCr!rgrhrir7r3r-rrS)rErrrBs z#FixedLengthRecordDatasetV1.__init__cCs|jjS)N)rTr9)rCrrrr9sz%FixedLengthRecordDatasetV1._filenamescCs ||j_dS)N)rTr9)rCr0rrrr9s)NNNNNN) rHrIrJrKrBrmrLr9rUrMrr)rErrns  rn)N)NN)0rKrZtensorflow.pythonrZtensorflow.python.data.opsrrrZtensorflow.python.data.utilrZtensorflow.python.frameworkrrr r r Ztensorflow.python.opsr r rr`Ztensorflow.python.utilrZ tensorflow.python.util.tf_exportrr<rr"r.r1Z DatasetSourcer2rNZDatasetV1AdapterrRrVZ UnaryDatasetr,rrcrerfrmrnenabledZFixedLengthRecordDatasetZTFRecordDatasetZTextLineDatasetrrrrsZ                & # + T (A c : _ "