B @ d,@sddlmZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZmZGdd d ZGd d d Ze eefZe eZGd d d ZdS))aliases)sha256)dumps)AnyDictIteratorListOptionalTupleUnion)TOO_BIG_SEQUENCE) iana_nameis_multi_byte_encoding unicode_rangec@seZdZd8eeeedeedddZe edddZ e edd d Z e ed d d Z ed ddZed ddZdddddZe ed ddZe eed ddZe ed ddZe ed ddZe eed ddZe ed ddZe ed d d!Ze ed d"d#Ze ed d$d%Ze ed d&d'Ze ed d(d)Ze edd d*d+Ze ed d,d-Ze eed d.d/Ze eed d0d1Z d9eed3d4d5Z!e ed d6d7Z"dS): CharsetMatchNCoherenceMatches)payloadguessed_encodingmean_mess_ratiohas_sig_or_bom languagesdecoded_payloadcCsF||_||_||_||_||_d|_g|_d|_d|_d|_ ||_ dS)Ng) _payload _encoding_mean_mess_ratio _languages_has_sig_or_bom_unicode_ranges_leavesZ_mean_coherence_ratio_output_payload_output_encoding_string)selfrrrrrrr$I/tmp/pip-install-587nrtaf/charset-normalizer/charset_normalizer/models.py__init__ s zCharsetMatch.__init__)otherreturncCs>t|ts&tdt|jt|j|j|jko<|j|jkS)Nz&__eq__ cannot be invoked on {} and {}.) isinstancer TypeErrorformatstr __class__encoding fingerprint)r#r'r$r$r%__eq__$s  zCharsetMatch.__eq__cCsvt|tstt|j|j}t|j|j}|dkrj|dkrj|dkr^|j|jkr^|j|jkS|j|jkS|j|jkS)zQ Implemented to make sorted available upon CharsetMatches items. g{Gz?g{Gz?g)r)r ValueErrorabschaos coherencemulti_byte_usage)r#r'Zchaos_differenceZcoherence_differencer$r$r%__lt__-s   zCharsetMatch.__lt__)r(cCsdtt|t|jS)Ng?)lenr,raw)r#r$r$r%r5@szCharsetMatch.multi_byte_usagecCs"|jdkrt|j|jd|_|jS)Nstrict)r"r,rr)r#r$r$r%__str__Ds zCharsetMatch.__str__cCsd|j|jS)Nz)r+r.r/)r#r$r$r%__repr__JszCharsetMatch.__repr__cCs8t|tr||kr"td|jd|_|j|dS)Nz;Unable to add instance <{}> as a submatch of a CharsetMatch)r)rr1r+r-r"rappend)r#r'r$r$r% add_submatchMs  zCharsetMatch.add_submatchcCs|jS)N)r)r#r$r$r%r.XszCharsetMatch.encodingcCsHg}x>tD]2\}}|j|kr,||q|j|kr||qW|S)z Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855. )ritemsr.r<)r#Z also_known_asupr$r$r%encoding_aliases\s   zCharsetMatch.encoding_aliasescCs|jS)N)r)r#r$r$r%bomiszCharsetMatch.bomcCs|jS)N)r)r#r$r$r%byte_order_markmszCharsetMatch.byte_order_markcCsdd|jDS)z Return the complete list of possible languages found in decoded sequence. Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'. cSsg|] }|dqS)rr$).0er$r$r% wsz*CharsetMatch.languages..)r)r#r$r$r%rqszCharsetMatch.languagescCsp|jsbd|jkrdSddlm}m}t|jr8||jn||j}t|dksVd|krZdS|dS|jddS)z Most probable language found in decoded sequence. If none were detected or inferred, the property will return "Unknown". asciiZEnglishr)encoding_languagesmb_encoding_languagesz Latin BasedUnknown)rcould_be_from_charsetZcharset_normalizer.cdrHrIrr.r7)r#rHrIrr$r$r%languageys  zCharsetMatch.languagecCs|jS)N)r)r#r$r$r%r3szCharsetMatch.chaoscCs|js dS|jddS)Ngrr )r)r#r$r$r%r4szCharsetMatch.coherencecCst|jdddS)Nd)ndigits)roundr3)r#r$r$r% percent_chaosszCharsetMatch.percent_chaoscCst|jdddS)NrMrN)rO)rPr4)r#r$r$r%percent_coherenceszCharsetMatch.percent_coherencecCs|jS)z+ Original untouched bytes. )r)r#r$r$r%r8szCharsetMatch.rawcCs|jS)N)r)r#r$r$r%submatchszCharsetMatch.submatchcCst|jdkS)Nr)r7r)r#r$r$r% has_submatchszCharsetMatch.has_submatchcCs@|jdk r|jSddt|D}ttdd|D|_|jS)NcSsg|] }t|qSr$)r)rDcharr$r$r%rFsz*CharsetMatch.alphabets..cSsh|] }|r|qSr$r$)rDrr$r$r% sz)CharsetMatch.alphabets..)rr,sortedlist)r#Zdetected_rangesr$r$r% alphabetss  zCharsetMatch.alphabetscCs|jgdd|jDS)z The complete list of encoding that output the exact SAME str result and therefore could be the originating encoding. This list does include the encoding available in property 'encoding'. cSsg|] }|jqSr$)r.)rDmr$r$r%rFsz6CharsetMatch.could_be_from_charset..)rr)r#r$r$r%rKsz"CharsetMatch.could_be_from_charsetutf_8)r.r(cCs2|jdks|j|kr,||_t||d|_|jS)z Method to get re-encoded bytes payload using given target encoding. Default to UTF-8. Any errors will be simply ignored by the encoder NOT replaced. Nreplace)r!r,encoder )r#r.r$r$r%outputszCharsetMatch.outputcCst|S)zw Retrieve the unique SHA256 computed using the transformed (re-encoded) payload. Not the original one. )rr_ hexdigest)r#r$r$r%r/szCharsetMatch.fingerprint)N)r\)#__name__ __module__ __qualname__bytesr,floatboolr r&objectr0r6propertyr5r:r;r=r.rrArBrCrrLr3r4rQrRr8rSrTrZrKr_r/r$r$r$r%r sR     rc@seZdZdZdeeedddZeedddZ e e e fed d d Z e dd d ZedddZedd ddZeddddZeddddZdS)CharsetMatchesz Container with every CharsetMatch items ordered by default from most probable to the less one. Act like a list(iterable) but does not implements all related methods. N)resultscCs|r t|ng|_dS)N)rX_results)r#rjr$r$r%r&szCharsetMatches.__init__)r(ccs|jEdHdS)N)rk)r#r$r$r%__iter__szCharsetMatches.__iter__)itemr(cCsNt|tr|j|St|trFt|d}x|jD]}||jkr0|Sq0WtdS)z Retrieve a single item either by its position or encoding name (alias may be used here). Raise KeyError upon invalid index or encoding not present in results. FN)r)intrkr,rrKKeyError)r#rmresultr$r$r% __getitem__s      zCharsetMatches.__getitem__cCs t|jS)N)r7rk)r#r$r$r%__len__szCharsetMatches.__len__cCst|jdkS)Nr)r7rk)r#r$r$r%__bool__szCharsetMatches.__bool__cCs~t|tstdt|jt|jtkrbx4|j D]*}|j |j kr4|j |j kr4| |dSq4W|j |t|j |_ dS)z~ Insert a single match. Will be inserted accordingly to preserve sort. Can be inserted as a submatch. z-Cannot append instance '{}' to CharsetMatchesN)r)rr1r+r,r-r7r8r rkr/r3r=r<rX)r#rmmatchr$r$r%r<s    zCharsetMatches.appendrcCs|js dS|jdS)zQ Simply return the first match. Strict equivalent to matches[0]. Nr)rk)r#r$r$r%bestszCharsetMatches.bestcCs|S)zP Redundant method, call the method best(). Kept for BC reasons. )ru)r#r$r$r%firstszCharsetMatches.first)N)rarbrc__doc__r rrr&rrlr rnr,rqrrrfrsr<rurvr$r$r$r%risric @sjeZdZeeeeeeeeeeeeeeeed ddZe e ee fdddZ edddZ d S) CliDetectionResult) pathr.rAalternative_encodingsrLrZrr3r4 unicode_path is_preferredc CsF||_| |_||_||_||_||_||_||_||_| |_ | |_ dS)N) ryr{r.rArzrLrZrr3r4r|) r#ryr.rArzrLrZrr3r4r{r|r$r$r%r&&szCliDetectionResult.__init__)r(c Cs2|j|j|j|j|j|j|j|j|j|j |j d S)N) ryr.rArzrLrZrr3r4r{r|) ryr.rArzrLrZrr3r4r{r|)r#r$r$r%__dict__@szCliDetectionResult.__dict__cCst|jdddS)NT) ensure_asciiindent)rr})r#r$r$r%to_jsonPszCliDetectionResult.to_jsonN)rarbrcr,r rrfrer&rhrrr}rr$r$r$r%rx%srxN)Zencodings.aliasesrhashlibrjsonrtypingrrrrr r r Zconstantr utilsrrrrrir,reZCoherenceMatchrrxr$r$r$r%s   $ UC