B ӻd[k @sBdZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlmZddlmZddlmZddlmZdd lm Z dd l!m"Z"e j#dd krdDd d Z$n ddl%m$Z$ddZ&dEddZ'e"ddFddZ(ddZ)dGddZ*dHd d!Z+dId"d#Z,Gd$d%d%e-Z.d&d'Z/e"d(Gd)d*d*e-Z0d+d,Z1ia2da3e4Z5da6e7Z8da9e:Z;d-d.Zd3d4Z?d5d6Z@e"d7Gd8d9d9e-ZAe"d:Gd;d<dZCd?d@ZDe"dAGdBdCdCeAZEdS)Kz(Utilities for file download and caching.)abstractmethod)closingN)ops)urlopen) tf_inspect)Progbar)path_to_string) keras_exportc CsNddd}t||}t|d&}x|||dD]}||q.WWdQRXdS)aReplacement for `urlretrieve` for Python 2. Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy `urllib` module, known to have issues with proxy management. Args: url: url to retrieve. filename: where to store the retrieved data locally. reporthook: a hook function that will be called once on establishment of the network connection and once after each block read thereafter. The hook will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. data: `data` argument passed to `urlopen`. Ncssh|d}d}|dk r&t|}d}x8||}|d7}|dk rR|||||r^|Vq,Pq,WdS)NzContent-Lengthr)infogetintstripread)response chunk_size reporthook content_type total_sizecountchunkrZ/var/www/html/venv/lib/python3.7/site-packages/tensorflow/python/keras/utils/data_utils.py chunk_readAs   zurlretrieve..chunk_readwb)r)r N)ropenwrite)urlfilenamerdatarrfdrrrr urlretrieve1s    r$)r$cCsJttttttf}t|tjt j f|r*dSt |pHt|t pHt|tjS)z'Check if `x` is a Keras generator type.F)strlisttupledictset frozenset isinstancerZTensornpZndarrayr isgeneratorSequencetypingIterator)xZbuiltin_iteratorsrrris_generator_or_sequenceYs   r2.autoc Cs|dkr dS|dkrddg}t|tr,|g}t|}t|}x|D]}|dkrZtj}tj}|dkrntj}tj}||rB||b}y| |WnLtj t t fk rt j|rt j|rt |n t|YnXWdQRXdSqBWdS)a6Extracts an archive if it matches tar, tar.gz, tar.bz, or zip formats. Args: file_path: path to the archive file path: path to extract the archive file archive_format: Archive format to try for extracting the file. Options are 'auto', 'tar', 'zip', and None. 'tar' includes tar, tar.gz, and tar.bz files. The default 'auto' is ['tar', 'zip']. None or an empty list will return no matches found. Returns: True if a match was found and an archive extraction was completed, False otherwise. NFr4tarzipT)r+r%rtarfiler is_tarfilezipfileZipFile is_zipfile extractallTarError RuntimeErrorKeyboardInterruptospathexistsisfileremoveshutilrmtree) file_pathrAarchive_formatZ archive_typeZopen_fnZ is_match_fnarchiverrr_extract_archivecs6       rJzkeras.utils.get_fileFdatasetsc  s4| dkrtjtjdd} |dk r6|dkr6|}d}tj| } t| tjs^tjdd} tj| |} t| t|}|rtj| |} | d} ntj| |} d}tj| r|dk rt | ||dst d |d |d d }nd }|rt d |Gdddt fdd}d}yyt || |Wnzt jjk rn}zt|||j|jWdd}~XYn>t jjk r}zt|||j|jWdd}~XYnXWnBttfk r}ztj| rt| Wdd}~XYnXd_|rtj| st| | dd| S|r0t| | || S)aDownloads a file from a URL if it not already in the cache. By default the file at the url `origin` is downloaded to the cache_dir `~/.keras`, placed in the cache_subdir `datasets`, and given the filename `fname`. The final location of a file `example.txt` would therefore be `~/.keras/datasets/example.txt`. Files in tar, tar.gz, tar.bz, and zip formats can also be extracted. Passing a hash will verify the file after download. The command line programs `shasum` and `sha256sum` can compute the hash. Example: ```python path_to_downloaded_file = tf.keras.utils.get_file( "flower_photos", "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz", untar=True) ``` Args: fname: Name of the file. If an absolute path `/path/to/file.txt` is specified the file will be saved at that location. origin: Original URL of the file. untar: Deprecated in favor of `extract` argument. boolean, whether the file should be decompressed md5_hash: Deprecated in favor of `file_hash` argument. md5 hash of the file for verification file_hash: The expected hash string of the file after download. The sha256 and md5 hash algorithms are both supported. cache_subdir: Subdirectory under the Keras cache dir where the file is saved. If an absolute path `/path/to/folder` is specified the file will be saved at that location. hash_algorithm: Select the hash algorithm to verify the file. options are `'md5'`, `'sha256'`, and `'auto'`. The default 'auto' detects the hash algorithm in use. extract: True tries extracting the file as an Archive, like tar or zip. archive_format: Archive format to try for extracting the file. Options are `'auto'`, `'tar'`, `'zip'`, and `None`. `'tar'` includes tar, tar.gz, and tar.bz files. The default `'auto'` corresponds to `['tar', 'zip']`. None or an empty list will return no matches found. cache_dir: Location to store cached files, when None it defaults to the default directory `~/.keras/`. Returns: Path to the downloaded file N~z.kerasmd5z/tmpz.tar.gzF) algorithmzNA local file was found, but it seems to be incomplete or outdated because the z0 file hash does not match the original value of z! so we will re-download the data.TzDownloading data fromc@seZdZdZdS)z!get_file..ProgressTrackerN)__name__ __module__ __qualname__progbarrrrrProgressTrackersrScs6jdkr"|dkrd}t|_nj||dS)Nr )rRrupdate)r block_sizer)rSrr dl_progresss   zget_file..dl_progressz!URL fetch failure on {}: {} -- {}r5)rH)r@rAjoin expanduseraccessW_OK_makedirs_exist_okrrB validate_fileprintobjectr$urlliberror HTTPError ExceptionformatcodemsgURLErrorerrnoreasonr?rDrRrJ)fnameoriginZuntarZmd5_hash file_hashZ cache_subdirZhash_algorithmextractrH cache_dirZ datadir_basedatadirZ untar_fpathfpathdownloadrV error_msger)rSrget_files`;     &,  rscCstj|dddS)NT)exist_ok)r@makedirs)rnrrrr[sr[cCs<|dkrtS|dkr4|dk r4t|dkr4tStS)z+Returns hash algorithm as hashlib function.sha256r4N@)hashlibrvlenrM)rNrkrrr_resolve_hashers rzrvc s^t|trt|}n|}t|d.x&tfdddD]}||q:WWdQRX|S)aCalculates a file sha256 or md5 hash. Example: ```python _hash_file('/path/to/file.zip') 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' ``` Args: fpath: path to the file being validated algorithm: hash algorithm, one of `'auto'`, `'sha256'`, or `'md5'`. The default `'auto'` detects the hash algorithm in use. chunk_size: Bytes to read at a time, important for large files. Returns: The file hash rbcs S)N)rr)r fpath_filerrAz_hash_file..rN)r+r%rzriterrT hexdigest)rorNrhasherrr)rr}r _hash_file(s   rcCs.t||}tt|||t|kr&dSdSdS)aValidates a file against a sha256 or md5 hash. Args: fpath: path to the file being validated file_hash: The expected hash string of the file. The sha256 and md5 hash algorithms are both supported. algorithm: Hash algorithm, one of 'auto', 'sha256', or 'md5'. The default 'auto' detects the hash algorithm in use. chunk_size: Bytes to read at a time, important for large files. Returns: Whether the file is valid TFN)rzr%r)rorkrNrrrrrr\Gs r\c@s0eZdZdZddZddZddZdd Zd S) ThreadsafeIterzEWrap an iterator with a lock and propagate exceptions to all threads.cCs||_t|_d|_dS)N)it threadingLocklock _exception)selfrrrr__init__`s zThreadsafeIter.__init__cCs|S)Nr)rrrr__iter__mszThreadsafeIter.__iter__cCs|S)N)__next__)rrrrnextpszThreadsafeIter.nextc CsX|jH|jr|jy t|jStk rH}z ||_Wdd}~XYnXWdQRXdS)N)rrrrrb)rrrrrrrss zThreadsafeIter.__next__N)rOrPrQ__doc__rrrrrrrrr]s  rcstfdd}|S)Ncst||S)N)r)akw)frrgszthreadsafe_generator..g) functoolswraps)rrr)rrthreadsafe_generatorsrzkeras.utils.Sequencec@s8eZdZdZeddZeddZddZdd Zd S) r.a1Base object for fitting to a sequence of data, such as a dataset. Every `Sequence` must implement the `__getitem__` and the `__len__` methods. If you want to modify your dataset between epochs you may implement `on_epoch_end`. The method `__getitem__` should return a complete batch. Notes: `Sequence` are a safer way to do multiprocessing. This structure guarantees that the network will only train once on each sample per epoch which is not the case with generators. Examples: ```python from skimage.io import imread from skimage.transform import resize import numpy as np import math # Here, `x_set` is list of path to the images # and `y_set` are the associated classes. class CIFAR10Sequence(Sequence): def __init__(self, x_set, y_set, batch_size): self.x, self.y = x_set, y_set self.batch_size = batch_size def __len__(self): return math.ceil(len(self.x) / self.batch_size) def __getitem__(self, idx): batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size] batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size] return np.array([ resize(imread(file_name), (200, 200)) for file_name in batch_x]), np.array(batch_y) ``` cCstdS)zGets batch at position `index`. Args: index: position of the batch in the Sequence. Returns: A batch N)NotImplementedError)rindexrrr __getitem__s zSequence.__getitem__cCstdS)zbNumber of batch in the Sequence. Returns: The number of batches in the Sequence. N)r)rrrr__len__szSequence.__len__cCsdS)z-Method called at the end of every epoch. Nr)rrrr on_epoch_endszSequence.on_epoch_endc#s.x(fddttDD] }|VqWdS)z2Create a generator that iterate over the Sequence.c3s|]}|VqdS)Nr).0i)rrr sz$Sequence.__iter__..N)rangery)ritemr)rrrs zSequence.__iter__N) rOrPrQrrrrrrrrrrr.s - r.ccsxx|D] }|VqWqWdS)zIterates indefinitely over a Sequence. Args: seq: `Sequence` instance. Yields: Batches of data from the `Sequence`. Nr)seqrrrriter_sequence_infinites  rcstfdd}|S)Nc s,ttd}a||}|a|SQRXdS)NT)_FORCE_THREADPOOL_LOCK_FORCE_THREADPOOL)argskwargsZold_force_threadpoolout)rrrwrappeds   z.dont_use_multiprocessing_pool..wrapped)rr)rrr)rrdont_use_multiprocessing_poolsrcCs|rtrtjjStjS)N)rmultiprocessingdummyZPool)use_multiprocessingrrrget_pool_classsrcCstdkrtatS)z,Lazily create the queue to track worker ids.N)_WORKER_ID_QUEUErQueuerrrrget_worker_id_queuesrcCs|adS)N)_SHARED_SEQUENCES)seqsrrr init_poolsrcCs t||S)aQGet the value from the Sequence `uid` at index `i`. To allow multiple Sequences to be used at the same time, we use `uid` to get a specific one. A single Sequence would cause the validation to overwrite the training Sequence. Args: uid: int, Sequence identifier i: index Returns: The value at index `i`. )r)uidrrrr get_indexsrzkeras.utils.SequenceEnqueuerc@sjeZdZdZdddZddZdd d Zd d ZdddZddZ e ddZ e ddZ e ddZ d S)SequenceEnqueueraBase class to enqueue inputs. The task of an Enqueuer is to use parallelism to speed up preprocessing. This is done with processes or threads. Example: ```python enqueuer = SequenceEnqueuer(...) enqueuer.start() datas = enqueuer.get() for data in datas: # Use the inputs; training, evaluating, predicting. # ... stop sometime. enqueuer.stop() ``` The `enqueuer.get()` should be an infinite stream of datas. Fc Cs||_||_tdkr>ytddaWntk r<daYnXtttrXt|_td7an*t tj |_tj d7_ WdQRXd|_ d|_ d|_ d|_d|_dS)Nrrr )sequencer_SEQUENCE_COUNTERrValueOSErrorr+rrZget_lockvalueworkers executor_fnqueue run_thread stop_signal)rrrrrrr<s$    zSequenceEnqueuer.__init__cCs|jdk o|j S)N)ris_set)rrrr is_runningZszSequenceEnqueuer.is_runningr csd|jr||_nfdd|_|_t||_t|_tj |j d|_ d|j _ |j dS)zStarts the handler's workers. Args: workers: Number of workers. max_queue_size: queue size (when full, workers could block on `put()`) cs tdS)NF)r)_)rrrr~irz(SequenceEnqueuer.start..)targetTN)r_get_executor_initrrrrrEventrThread_runrdaemonstart)rrZmax_queue_sizer)rrr]s  zSequenceEnqueuer.startcCs|jt|j<dS)z&Sends current Iterable to all workers.N)rrr)rrrr_send_sequenceqszSequenceEnqueuer._send_sequenceNc CsX|j|jj&|jjd|j_|jjWdQRX|j |dt |j <dS)zStops running threads and wait for them to exit, if necessary. Should be called by the same thread which called `start()`. Args: timeout: maximum time to wait on `thread.join()` rN) rr)rmutexclearunfinished_tasksnot_fullnotifyrrWrr)rtimeoutrrrstopvs    zSequenceEnqueuer.stopcCs|r|dS)N)rr)rrrr__del__szSequenceEnqueuer.__del__cCstdS)z?Submits request to the executor and queue the `Future` objects.N)r)rrrrrszSequenceEnqueuer._runcCstdS)zGets the Pool initializer for multiprocessing. Args: workers: Number of workers. Returns: Function, a Function to initialize the pool N)r)rrrrrrs z#SequenceEnqueuer._get_executor_initcCstdS)zCreates a generator to extract data from the queue. Skip the data if it is `None`. # Returns Generator yielding tuples `(inputs, targets)` or `(inputs, targets, sample_weights)`. N)r)rrrrrs zSequenceEnqueuer.get)F)r r)N)rOrPrQrrrrrrrrrrrrrrrr&s     rzkeras.utils.OrderedEnqueuercsBeZdZdZd fdd ZddZddZd d Zd d ZZ S)OrderedEnqueueraBuilds a Enqueuer from a Sequence. Args: sequence: A `tf.keras.utils.data_utils.Sequence` object. use_multiprocessing: use multiprocessing if True, otherwise threading shuffle: whether to shuffle the data at the beginning of each epoch Fcstt|||||_dS)N)superrrshuffle)rrrr) __class__rrrszOrderedEnqueuer.__init__csfdd}|S)zGets the Pool initializer for multiprocessing. Args: workers: Number of workers. Returns: Function, a Function to initialize the pool cs(tdt|dtfd}t||S)NT) initializerinitargs)rinit_pool_generatorr _DATA_POOLSadd)rpool)rrrpool_fns  z3OrderedEnqueuer._get_executor_init..pool_fnr)rrrr)rrrs z"OrderedEnqueuer._get_executor_initcCs.x(td|jjdks"|jrdSqWdS)zWait for the queue to be empty.g?rN)timesleeprrrr)rrrr _wait_queues zOrderedEnqueuer._wait_queuec Csttt|j}|x|jr,t|t|t V}x8|D]0}|j rTdS|j j |t|j|fddqBW||j rdSWdQRX|j|qWdS)z?Submits request to the executor and queue the `Future` objects.NT)block)r&rryrrrrandomrrrrrrput apply_asyncrrrr)rrexecutorrrrrrs      zOrderedEnqueuer._runc csx|ry8|jjddd}|r2|j|dk r@|VWqtjk rXYqtk r}z||Wdd}~XYqXqWdS)zCreates a generator to extract data from the queue. Skip the data if it is `None`. Yields: The next element in the queue, i.e. a tuple `(inputs, targets)` or `(inputs, targets, sample_weights)`. T)rrN)rrr task_doneEmptyrbr)rinputsrrrrrrs   zOrderedEnqueuer.get)FF) rOrPrQrrrrrr __classcell__rr)rrrs rcCsR|at}d|j|_|dk r4tj||j|dk rN|j |jddddS)aInitializer function for pool workers. Args: gens: State which should be made available to worker processes. random_seed: An optional value with which to seed child processes. id_queue: A multiprocessing Queue of worker ids. This is used to indicate that a worker process was created by Keras and can be terminated using the cleanup_all_keras_forkpools utility. zKeras_worker_{}NTg?)rr) rrcurrent_processrcnamer,rseedidentr)Zgens random_seedZid_queueZ worker_procrrrrs rcCs tt|S)aKGets the next value from the generator `uid`. To allow multiple generators to be used at the same time, we use `uid` to get a specific one. A single generator would cause the validation to overwrite the training generator. Args: uid: int, generator identifier Returns: The next value of generator `uid`. )rr)rrrr next_samples rzkeras.utils.GeneratorEnqueuercs:eZdZdZd fdd ZddZdd Zd d ZZS) GeneratorEnqueueraBuilds a queue out of a data generator. The provided generator can be finite in which case the class will throw a `StopIteration` exception. Args: generator: a generator function which yields data use_multiprocessing: use multiprocessing if True, otherwise threading random_seed: Initial seed for workers, will be incremented by one for each worker. FNcstt|||||_dS)N)rrrr)r generatorrr)rrrr:szGeneratorEnqueuer.__init__csfdd}|S)zGets the Pool initializer for multiprocessing. Args: workers: Number of works. Returns: A Function to initialize the pool cs*tdt|jtfd}t||S)NT)rr)rrrrrr)rr)rrrrrIs  z5GeneratorEnqueuer._get_executor_init..pool_fnr)rrrr)rrrr@s z$GeneratorEnqueuer._get_executor_initc CsV|t|t6}x.|jr(dS|jj|t |j fddqWWdQRXdS)z?Submits request to the executor and queue the `Future` objects.NT)r) rrrrrrrrrrr)rrrrrrQs zGeneratorEnqueuer._runc csyusz)GeneratorEnqueuer.get..zgenerator already executingzyYour generator is NOT thread-safe. Keras requires a thread-safe generator when `use_multiprocessing=False, workers > 1`. ) rrrr StopIterationqsizeappendwaitrbrr%r>)rrZ last_onesrrrrrrr\s,       zGeneratorEnqueuer.get)FN) rOrPrQrrrrrrrr)rrr,s   r)NN)r3r4)FNNrKr4Fr4N)N)rvr{)r4r{)NN)Frabcr contextlibrrrxrZmultiprocessing.dummyr@rrrEsysr7rrr/r_weakrefr9numpyr,Ztensorflow.python.frameworkrZsix.moves.urllib.requestrZtensorflow.python.keras.utilsrZ+tensorflow.python.keras.utils.generic_utilsrZ&tensorflow.python.keras.utils.io_utilsrZ tensorflow.python.util.tf_exportr version_infor$urllib.requestr2rJrsr[rzrr\r^rrr.rrrWeakSetrrr)Z _WORKER_IDSrRLockrrrrrrrrrrrrrrrs         %  1 z  " N Y