^\kdc@sdZddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZmZddlmZmZddlmZmZmZddlmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&d gZ'd Z(idgd6Z)d e*fdYZ+de*fdYZ,de*fdYZ-de*fdYZ.e.e Z/dZ0dZ1dS(s!Routines related to PyPI, indexesiN(tlogger(tInftnormalize_nametsplitextt is_prerelease(tDistributionNotFoundtBestVersionAlreadyInstalledtInstallationErrortInvalidWheelFilenametUnsupportedWheel(turlparset url2pathname(t PipSessiont url_to_patht path_to_url(tWheelt wheel_ext(tsupported_tagstsupported_tags_noarcht get_platform(thtml5libtrequestst pkg_resources(tSSLErrort PackageFinderslast.pypi.python.orgthttpsthttpcBseZdZeggeeeddZdZdZdZ dZ dZ dZ dZ ejd Zejd ejZejd Zd Zd ZdZdZdZdZRS(sThis finds packages. This is meant to match easy_install's technique for looking for packages, by reading pages and looking for appropriate links c Cs||_||_g|_t|_t|_||_td|D|_td|D|_ |j|j O_||_ t |_ t |_ ||_||_t |_| pt|_dS(Ncss|]}t|VqdS(N(R(t.0tn((s0/var/www/testScraper/venv/build/pip/pip/index.pys 4scss|]}t|VqdS(N(R(RR((s0/var/www/testScraper/venv/build/pip/pip/index.pys 8s(t find_linkst index_urlstdependency_linkst PageCachetcachetsett logged_linkst use_wheeltallow_externaltallow_unverifiedtallow_all_externaltFalsetneed_warn_externaltneed_warn_unverifiedtallow_all_prereleasestprocess_dependency_linkst_have_warned_dependency_linksR tsession( tselfRRR$R%R&R'R+R,R.((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__init__&s"            cCsE|jrA|js.tjddt|_n|jj|ndS(Ns1.6spDependency Links processing has been deprecated with an accelerated time schedule and will be removed in pip 1.6(R,R-Rt deprecatedtTrueRtextend(R/tlinks((s0/var/www/testScraper/venv/build/pip/pip/index.pytadd_dependency_linksSs   c sGggfd}x|D]}tjj|}|jd}||jk}|sg|r,|rv|}n t|}|rtjj|rtjj|}xztj|D]}|tjj ||qWq9|r tjj|r j |q9tjj |r9||q9q%j |q%WfS(st Sort locations into "files" (archives) and "urls", and return a pair of lists (files,urls) csLt|}tj|dtddkr;j|n j|dS(Ntstrictis text/html(Rt mimetypest guess_typeR(tappend(tpathturl(tfilesturls(s0/var/www/testScraper/venv/build/pip/pip/index.pyt sort_pathks sfile:( tosR:texistst startswithRR tisdirtrealpathtlistdirtjoinR9tisfile( R/t locationsR>R;t is_local_patht is_file_urlt is_find_linkR:titem((R<R=s0/var/www/testScraper/venv/build/pip/pip/index.pyt_sort_locationsbs*     cCs|\}}}|jrtt}|tkr9d}nW|jtkrt|j}|jsyt d|jn|j }n| }||fS|SdS(sL Function used to generate link sort key for link tuples. The greater the return value, the more preferred it is. If not finding wheels, then sorted by version only. If finding wheels, then the sort order is by version, then: 1. existing installs 2. wheels ordered via Wheel.support_index_min() 3. source archives Note: it was considered to embed this logic into the Link comparison operators, but then different sdist links with the same version, would have to be considered equal isB%s is not a supported wheel for this platform. It can't be sorted.N( R$tlenRtINSTALLED_VERSIONtextRRtfilenamet supportedR tsupport_index_min(R/t link_tupletparsed_versiontlinkt_t support_numtpritwheel((s0/var/www/testScraper/venv/build/pip/pip/index.pyt_link_sort_keys       cCst|d|jdtS(s Bring the latest version (and wheels) to the front, but maintain the existing ordering as secondary. See the docstring for `_link_sort_key` for details. This function is isolated for easier unit testing. tkeytreverse(tsortedRZR2(R/tapplicable_versions((s0/var/www/testScraper/venv/build/pip/pip/index.pyt_sort_versionsscsfd}|jd}|jrt||jddt}|j||}|dkr|jt|jddt|p|jqndk rg|jD]}||^q|j}nt|j}xH|j D]=}dk r|dk rt j |j |g|}qqW|j |\} } |j |j\} } | j| g| D]}t|dt^qn}|jg| D]}t|^qtjd|x|D]} tjd| tjt| }|jtkrt|j}t|dkrT| |j|d|jf}tjd|qt|dkr| |jdj ||jf}tjd |q| |jf}tjd |qqWg}|j|jg|jD]}t|d dt^q|jjg}xy|j||D]e}tjd |j tjd 7_z)|j|j|j|jjWdtjd 8_Xq(Wt|jg|jD]}t|^q|jj}|rtjddj g|D]\}}}|j ^qnt|jg| D]}t|^q |jj}| r| r| r| rtj d||j!rtjd|jn|j"rtjd|jnt#d|ng}|j$dk r|j$j%t&|j$j'fg}n|rX|j(dttjddj g|D]\}}}t)|j ^q,n|||||}g}x|D]\}}}||j*krtjd||dj g|j*j+D]}dj |^qfq{nKt,|r/|j-p|j. r/|t&k r/tjd||fq{q/n|j/|||fq{W|j0|}t1g|D]!\}}}|t&krb|^qb}| r|r|ddt&krtjd|j$j'n%tjd|j$j'|dd fdS|s~tj d|dj g|D]\}}}|^qf|j!rKtjdn|j"rktjd|jnt#d|n|ddt&krtjd|j$j'dj g|dD]\}}}|^qpdft2nt|dkr@tjd |dd dj g|D]\}}}|^qfn|dd}|j3dk r~|j3 r~tjd!|jn|j4dk r|j4 rtjd"|jn|j5rtj6d#d$|jn|S(%Ncs2tj|}|jds.|d}n|S(Nt/(t posixpathREtendswith(R;tloc(turl_name(s0/var/www/testScraper/venv/build/pip/pip/index.pytmkurl_pypi_urls ittrusteds#URLs to search for versions for %s:s* %sisS%s uses an insecure transport scheme (%s). Consider using %s if %s has it availables, sc%s uses an insecure transport scheme (%s). Consider using one of %s if %s has any of them availables*%s uses an insecure transport scheme (%s).s-fsAnalyzing links from page %sisdependency_links found: %ss<Could not find any downloads that satisfy the requirement %ssMSome externally hosted files were ignored (use --allow-external %s to allow).sWSome insecure and unverifiable files were ignored (use --allow-unverified %s to allow).s$No distributions at all found for %sR\sLocal files found: %ss-Ignoring link %s, version %s doesn't match %st,tsCIgnoring link %s, version %s is a pre-release (use --pre to allow).sLExisting installed version (%s) is most up-to-date and satisfies requirementsUExisting installed version (%s) satisfies requirement (most up-to-date version is %s)sNCould not find a version that satisfies the requirement %s (from versions: %s)sJSome externally hosted files were ignored (use --allow-external to allow).s,No distributions matching the version for %ss=Installed version (%s) is most up-to-date (past versions: %s)tnones)Using version %s (newest of versions: %s)s2%s an externally hosted file and may be unreliables,%s is potentially insecure and unverifiable.s1.7sa%s discovered using a deprecated method of parsing, in the future it will no longer be discovered(7RdtNoneRtLinkR2t _get_paget_find_url_nameRtlisttabsolute_versionsRaRER;RLRR3RtdebugR tstrtschemetINSECURE_SCHEMESRMtnetloctwarnt_package_versionstnametlowert _get_pagestindentR4tinfotfatalR)R*Rt satisfied_byRTRNtversiontsortR treqtspecsRR+t prereleasesR9R_tboolRtinternalt verifiablet_deprecated_regexR1(R/RtupgradeRetmain_index_urltpageR;RGR~tfile_locationst url_locationst _flocationst _ulocationstlocationtparsedtsecure_schemestctxtfound_versionst page_versionstdependency_versionsRUt file_versionstinstalled_versiont all_versionsR^RTtstexisting_applicabletselected_version((Rds0/var/www/testScraper/venv/build/pip/pip/index.pytfind_requirements   7 )# %&         ;) 4< 1    !B <  4    3    C  >     cCs|jjds$|jd7_n|j||}|dkrWtjd|dSt|j}xY|jD]N}t j |j j d}|t|krptj d||f|SqpWdS(sFinds the true URL name of a package, when the given name isn't quite correct. This is usually used to implement case-insensitivity.R`sCannot fetch index base URL %sNs!Real name of requirement %s is %s(R;RbRlRjRR|RRdR4RatbasenameR:trstriptnotify(R/t index_urlRdRRt norm_nameRUtbase((s0/var/www/testScraper/venv/build/pip/pip/index.pyRmbs c cs7t|}t}x|r2|jd}||kr?qn|j||j||}|dkrpqn|Vx|jD]}t|jj }||j kr|j rt |_ tjd|qn|jdk r|j r||jkrtjd|t |_qn|j|qWqWdS(s Yields (page, page_url) from the given locations, skipping locations that have errors, and adding download/homepage links is@Not searching %s for files because external urls are disallowed.s_Not searching %s for urls, it is an untrusted link and cannot produce safe or verifiable files.N(RnR"tpoptaddRlRjt rel_linksRRwRxR%R'R2R)RRpRfR&R*R9( R/RGRt all_locationstseenRRRUt normalized((s0/var/www/testScraper/venv/build/pip/pip/index.pyRyus6            s #egg=([^&]*)s([a-z0-9_.]+)-([a-z0-9_.-]+)s-py([123]\.?[0-9]?)$cCsqgg}}t}xP|D]H}||kr|j||jrU|j|qe|j|qqW||S(sgReturns elements of links in order, non-egg links first, egg links second, while eliminating duplicates(R"Rt egg_fragmentR9(R/R4teggstno_eggsRRU((s0/var/www/testScraper/venv/build/pip/pip/index.pyt _sort_linkss      ccs@x9|j|D](}x|j||D] }|Vq)WqWdS(N(Rt_link_package_versions(R/R4t search_nameRUtv((s0/var/www/testScraper/venv/build/pip/pip/index.pyRvscCsd}|jr|tfS|S(Ns.tar.gzs.tar.bz2s.tars.tgzs.zip(s.tar.gzs.tar.bz2s.tars.tgzs.zip(R$R(R/t extensions((s0/var/www/testScraper/venv/build/pip/pip/index.pyt_known_extensionss  c Cst}d}|jr$|j}nD|j\}}|ss||jkrotjd||jj|ngS|jdr|d }d|}n||j kr||jkrtjd||f|jj|ngSd|j kr:|dkr:||jkr6tjd||jj|ngS|t krhyt |j }Wn#tk r~tjd|gSX|jj|jkrtjd ||fgS|jstjd |gSt|d d}|jd  r\|jd  r\|dk r\tj|jjjdr\|jdts\tjd|gSn|j}n|s|j|||}n|dkrtjd ||fgS|jdk r |j r t|j|jkr |j r tjd|t|_gS|j dk r[|j r[t|j|j!kr[tjd|t|_"gS|j#j$|} | r|| j% }| j&d} | t'jd krtjd|gSntjd||ft(j)|||fgS(s Return an iterable of triples (pkg_resources_version_key, link, python_version) that can be extracted from the given link. Meant to be overridden by subclasses, not called by clients. sSkipping link %s; not a files.taris,Skipping link %s; unknown archive format: %stmacosx10s.zipsSkipping link %s; macosx10 ones1Skipping %s because the wheel filename is invalids-Skipping link %s; wrong project name (not %s)s9Skipping %s because it is not compatible with this Pythont comes_fromtwintmacosxspypi.python.orgttagssOSkipping %s because it is a pypi-hosted binary Wheel on an unsupported platforms,Skipping %s because it is externally hosted.s<Skipping %s because it is an insecure and unverifiable file.iis/Skipping %s because Python version is incorrectsFound link %s, version: %sN(*RRjRRR#RRpRRbRR:RRRPRRwRxRQtgetattrRAR R;RtRR~t_egg_info_matchesRRR%R'R2R)RR&R*t_py_version_retsearchtstarttgrouptsysRt parse_version( R/RURtplatformR~tegg_infoRORYRtmatcht py_version((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs                    cCs|jj|}|s-tjd|dS|jdj}|jdd}|jd}|j|r|jdt |SdSdS(Ns%Could not parse version from link: %siRVt-( t _egg_info_reRRRpRjRRxtreplaceRARM(R/RRRURRwtlook_for((s0/var/www/testScraper/venv/build/pip/pip/index.pyR&scCs"tj||d|jd|jS(NR!R.(tHTMLPagetget_pageR!R.(R/RUR((s0/var/www/testScraper/venv/build/pip/pip/index.pyRl5s N(t__name__t __module__t__doc__R2R(RjR0R5RLRZR_RRmRytretcompilet_egg_fragment_retIRRRRvRRRRl(((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs( *  (    )   l R cBsVeZdZdZdZdZdZdZedZ dZ dZ RS( sCache of HTML pagesicCsi|_i|_i|_dS(N(t _failurest_pagest _archives(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyR0As  cCs|jj|d|jkS(Ni(Rtgett failure_limit(R/R;((s0/var/www/testScraper/venv/build/pip/pip/index.pyttoo_many_failuresFscCs|jj|S(N(RR(R/R;((s0/var/www/testScraper/venv/build/pip/pip/index.pyRIscCs|jj|tS(N(RRR((R/R;((s0/var/www/testScraper/venv/build/pip/pip/index.pyt is_archiveLscCs||j|\s*home\s*pages\s*download\s+urls(href=(?:"([^"]*)"|'([^']*)'|([^>\s\n]*))cCsC||_tj|jdt|_||_||_||_dS(NtnamespaceHTMLElements(tcontentRtparseR(RR;theadersRf(R/RR;RRf((s0/var/www/testScraper/venv/build/pip/pip/index.pyR0bs    cCs|jS(N(R;(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__str__isc Cs%|dkrt}n|j}|jddd}|j|rJdSddlm}xT|jD]I}|jj |rd|t |dkrdt j dt dSqdW|dk r|j|} | dk r| Sny|r|dk r |j|r dSn|j} xdd d d d gD]}} | j| r)|j|d |} | jj drlPqt j d|| f|dk r|j|ndSq)q)Wnt j d|tj|\}} }}}}|dkrEtjjt|rE|jds|d7}ntj|d}t j d|n|j|didd6}|j|jjdd} | jj dst j d|| f|dk r|j|ndS||j|j|jd|j} Wntj k rI}|j!j"dkrdnd}|j#||||d|d|ntj$k r~}|j#||d||d|ntj%k r|j#||d|d|ntt&k r}d |}|j#||||d|ddd!t j'n-X|dk r|j(||jg| n| SdS("Nt#iii(t VcsSupports+:s&Cannot look at %(scheme)s URL %(link)ss.tars.tar.gzs.tar.bz2s.tgzs.zipR.s text/htmls,Skipping page %s because of Content-Type: %ssGetting page %stfileR`s index.htmls# file: URL is directory, getting %sRtAccepts Content-TypetunknownRfiiR!Rsconnection error: %ss timed outs6There was a problem confirming the ssl certificate: %stmeth()RjR R;tsplitRtpip.vcsRtschemesRxRARMRRptlocalsRRRPRbt_get_content_typeRR R?R:RBR turljoinRtraise_for_statusRttextRfRt HTTPErrortresponset status_codet _handle_failtConnectionErrortTimeoutRRR(tclsRURR!t skip_archivesR.R;RRrtinstRPtbad_extt content_typeRtR:tparamstquerytfragmenttresptexcRtreason((s0/var/www/testScraper/venv/build/pip/pip/index.pyRls   +       !$    (%  icCsb|dkrtj}n|d|||d|j|f|dk r^|j||ndS(NsCould not fetch URL %s: %ss7Will skip URL %s when looking for download links for %s(RjRR{R;R(RRURR;R!RR((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs   cCsx|dkrt}ntj|\}}}}}|d krFdS|j|dt}|j|jjddS( s;Get the Content-Type of the given url, using a HEAD requestRRtftptftpsRhtallow_redirectss Content-TypeN(shttpshttpssftpsftps( RjR R turlsplittheadR2RRR(R;R.RrRtR:RRR((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs    cCst|dsd}g|jjdD]*}|jddjdkr(|^q(}|ry t|djdd}Wqttfk rd}qXn||_ n|j S(Nt _api_versions.//metaRwRhs api-versioniR( thasattrRjRtfindallRRxtintt TypeErrort ValueErrorR(R/Rtxtmetas((s0/var/www/testScraper/venv/build/pip/pip/index.pyt api_versions*   cCsgt|ds`|jjd}|dk rQ|jdrQ|jd|_q`|j|_n|jS(Nt _base_urls.//basethref(RRtfindRjRR R;(R/R((s0/var/www/testScraper/venv/build/pip/pip/index.pytbase_urls ccsx|jjdD]}|jdr|jd}|jtj|j|}d}|jr|jdkrt |jdod|jdj k}nt ||d|VqqWdS(sYields all links in the pages.//aRitrelRN( RRRt clean_linkR RRRjR RRRk(R/tanchorRR;R((s0/var/www/testScraper/venv/build/pip/pip/index.pyR4s!ccs<x|jD] }|Vq Wx|jD] }|Vq)WdS(N(texplicit_rel_linkstscraped_rel_links(R/R;((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs thomepagetdownloadccst|}x|jjdD]}|jdr|jdrt|jdj}||@r|jd}|jtj|j|}t ||dt VqqqWdS(s)Yields all links with the given relationss.//aRRRfN( R"RRRRRR RRRkR((R/trelsRt found_relsRR;((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs  c csx|j|jfD]}|j|j}|s7qn|jj|jd|j}|sdqn|jdp|jdp|jd}|sqn|jtj |j |}t ||dt dt VqWdS(NtposiiiRfR(t _homepage_ret _download_reRRt_href_retendRRR RRRkR(R2(R/tregexRt href_matchR;((s0/var/www/testScraper/venv/build/pip/pip/index.pyR&s!-s[^a-z0-9$&+,/:;=?@.#%_\\|-]cCs|jjd|S(sMakes sure a link is fully encoded. That is, if a ' ' shows up in the link, it will be rewritten to %20 (while not over-quoting % or other characters).cSsdt|jdS(Ns%%%2xi(tordR(R((s0/var/www/testScraper/venv/build/pip/pip/index.pyt<s(t _clean_retsub(R/R;((s0/var/www/testScraper/venv/build/pip/pip/index.pyR7s N(shomepagesdownload(RRRRRRRRtSRRjR0Rt classmethodR2Rt staticmethodRRtpropertyR RR4RRRR#R(((s0/var/www/testScraper/venv/build/pip/pip/index.pyRZs( [     RkcBs+eZdddedZdZdZdZdZdZ dZ dZ dZ d Z ed Zed Zed Zd ZedZedZejdZedZejdZedZedZedZedZRS(cCs1||_||_||_||_||_dS(N(R;RRRfR(R/R;RRRfR((s0/var/www/testScraper/venv/build/pip/pip/index.pyR0As     cCs.|jrd|j|jfSt|jSdS(Ns %s (from %s)(RR;Rq(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyRIs cCsd|S(Ns ((R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__repr__OscCs|j|jkS(N(R;(R/tother((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__eq__RscCs|j|jkS(N(R;(R/R*((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__ne__UscCs|j|jkS(N(R;(R/R*((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__lt__XscCs|j|jkS(N(R;(R/R*((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__le__[scCs|j|jkS(N(R;(R/R*((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__gt__^scCs|j|jkS(N(R;(R/R*((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__ge__ascCs t|jS(N(thashR;(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyt__hash__dscCs\tj|j\}}}}}tj|jdp<|}|sXtd|j|S(NR`sURL %r produced no filename(R RR;RaRRtAssertionError(R/RVRtR:Rw((s0/var/www/testScraper/venv/build/pip/pip/index.pyRPgs!cCstj|jdS(Ni(R RR;(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyRrnscCstj|jdS(Ni(R RR;(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyR:rscCsttj|jjdS(NR`(RRaRR:R(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyRvscCs|jdS(Ni(R(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pyROyscCs=tj|j\}}}}}tj||||dfS(N(R RR;t urlunsplitRj(R/RrRtR:RR((s0/var/www/testScraper/venv/build/pip/pip/index.pyturl_without_fragment}s!s #egg=([^&]*)cCs,|jj|j}|sdS|jdS(Ni(RRR;RjR(R/R((s0/var/www/testScraper/venv/build/pip/pip/index.pyRss2(sha1|sha224|sha384|sha256|sha512|md5)=([a-f0-9]+)cCs,|jj|j}|r(|jdSdS(Ni(t_hash_reRR;RRj(R/R((s0/var/www/testScraper/venv/build/pip/pip/index.pyR1s cCs,|jj|j}|r(|jdSdS(Ni(R6RR;RRj(R/R((s0/var/www/testScraper/venv/build/pip/pip/index.pyt hash_names cCs0tj|jjdddjdddS(NRiit?(RaRR;R(R/((s0/var/www/testScraper/venv/build/pip/pip/index.pytshow_urlscCs|jpt|jdd}|dk r|ry%t|jdd}t|}Wnttfk rtd}nX|dks|dkrdS|jrtSt Sn|dk rt SdS(s Returns True if this link can be verified after download, False if it cannot, and None if we cannot determine. RfR iN( RfRRRjRR RR1R2R((R/RfR ((s0/var/www/testScraper/venv/build/pip/pip/index.pyRs   N(RRRjR(R0RR)R+R,R-R.R/R0R2R(RPRrR:RROR5RRRRR6R1R7R9R(((s0/var/www/testScraper/venv/build/pip/pip/index.pyRk?s0            cCs;t|}|j}|s1t|jd}nt|S(sPGet a requirement from the URL, if possible. This looks for #egg in the URLi(RkRRRPtpackage_to_requirement(R;RUR((s0/var/www/testScraper/venv/build/pip/pip/index.pytget_requirement_from_urls   cCsatjd|}|r9|jd}|jd}n |}d}|rYd||fS|SdS(s)Translate a name like Foo-1.2 to Foo==1.3s^(.*?)-(dev|\d.*)iiRhs%s==%sN(RRR(t package_nameRRwR~((s0/var/www/testScraper/venv/build/pip/pip/index.pyR:s(2RRR?RR7Ratpip.logRtpip.utilRRRRtpip.exceptionsRRRRR tpip.backwardcompatR R t pip.downloadR R Rt pip.wheelRRtpip.pep425tagsRRRt pip._vendorRRRtpip._vendor.requests.exceptionsRt__all__tDEFAULT_MIRROR_HOSTNAMERstobjectRR RRkRNR;R:(((s0/var/www/testScraper/venv/build/pip/pip/index.pyts6     "(