Spamworldpro Mini Shell
Spamworldpro


Server : Apache/2.4.52 (Ubuntu)
System : Linux webserver 6.8.0-49-generic #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 6 17:42:15 UTC 2 x86_64
User : www-data ( 33)
PHP Version : 8.1.2-1ubuntu2.21
Disable Function : NONE
Directory :  /lib/python3.10/urllib/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Current File : //lib/python3.10/urllib/__pycache__/robotparser.cpython-310.pyc
o

�*�g�$�@s\dZddlZddlZddlZdgZe�dd�ZGdd�d�ZGdd�d�Z	Gd	d
�d
�Z
dS)a% robotparser.py

    Copyright (C) 2000  Bastian Kleineidam

    You can choose between two licenses when using this package:
    1) GNU GPLv2
    2) PSF license for Python 2.2

    The robots.txt Exclusion Protocol is implemented as specified in
    http://www.robotstxt.org/norobots-rfc.txt
�N�RobotFileParser�RequestRatezrequests secondsc@sreZdZdZddd�Zdd�Zdd�Zd	d
�Zdd�Zd
d�Z	dd�Z
dd�Zdd�Zdd�Z
dd�Zdd�ZdS)rzs This class provides a set of methods to read, parse and answer
    questions about a single robots.txt file.

    �cCs2g|_g|_d|_d|_d|_|�|�d|_dS)NFr)�entries�sitemaps�
default_entry�disallow_all�	allow_all�set_url�last_checked��self�url�r�)/usr/lib/python3.10/urllib/robotparser.py�__init__s

zRobotFileParser.__init__cCs|jS)z�Returns the time the robots.txt file was last fetched.

        This is useful for long-running web spiders that need to
        check for new robots.txt files periodically.

        )r�r
rrr�mtime%szRobotFileParser.mtimecCsddl}|��|_dS)zYSets the time the robots.txt file was last fetched to the
        current time.

        rN)�timer)r
rrrr�modified.szRobotFileParser.modifiedcCs&||_tj�|�dd�\|_|_dS)z,Sets the URL referring to a robots.txt file.��N)r�urllib�parse�urlparse�host�pathrrrrr
6s zRobotFileParser.set_urlc
Cs�z	tj�|j�}Wn@tjjyI}z2|jdvrd|_n|jdkr0|jdkr>d|_WYd}~dSWYd}~dSWYd}~dSWYd}~dSd}~ww|�	�}|�
|�d����dS)z4Reads the robots.txt URL and feeds it to the parser.)i�i�Ti�i�Nzutf-8)
r�request�urlopenr�error�	HTTPError�coderr	�readr�decode�
splitlines)r
�f�err�rawrrrr";s
����zRobotFileParser.readcCs2d|jvr|jdur||_dSdS|j�|�dS�N�*)�
useragentsrr�append)r
�entryrrr�
_add_entryHs



�zRobotFileParser._add_entrycCsJd}t�}|��|D�]
}|s(|dkrt�}d}n|dkr(|�|�t�}d}|�d�}|dkr7|d|�}|��}|s>q|�dd�}t|�dk�r|d����|d<tj	�
|d���|d<|ddkr~|dkrs|�|�t�}|j�|d�d}q|ddkr�|dkr�|j
�t|dd	��d}q|dd
kr�|dkr�|j
�t|dd��d}q|ddkr�|dkr�|d����r�t|d�|_d}q|dd
k�r|dk�r|d�d�}t|�dk�r|d�����r|d�����rtt|d�t|d��|_d}q|ddk�r|j�|d�q|dk�r#|�|�dSdS)z�Parse the input lines from a robots.txt file.

        We allow that a user-agent: line is not preceded by
        one or more blank lines.
        rr��#N�:z
user-agent�disallowF�allowTzcrawl-delayzrequest-rate�/�sitemap)�Entryrr-�find�strip�split�len�lowerrr�unquoter*r+�	rulelines�RuleLine�isdigit�int�delayr�req_rater)r
�lines�stater,�line�i�numbersrrrrQsv




���
 ���
�zRobotFileParser.parsecCs�|jrdS|jr
dS|jsdStj�tj�|��}tj�dd|j|j	|j
|jf�}tj�|�}|s3d}|j
D]}|�|�rD|�|�Sq6|jrN|j�|�SdS)z=using the parsed robots.txt decide if useragent can fetch urlFTrr3)rr	rrrrr;�
urlunparser�params�query�fragment�quoter�
applies_to�	allowancer)r
�	useragentr�
parsed_urlr,rrr�	can_fetch�s(�

�zRobotFileParser.can_fetchcC�>|��sdS|jD]}|�|�r|jSq	|jr|jjSdS�N)rrrLr@r�r
rNr,rrr�crawl_delay��


�zRobotFileParser.crawl_delaycCrQrR)rrrLrArrSrrr�request_rate�rUzRobotFileParser.request_ratecCs|jsdS|jSrR)rrrrr�	site_maps�szRobotFileParser.site_mapscCs,|j}|jdur||jg}d�tt|��S)Nz

)rr�join�map�str)r
rrrr�__str__�s
zRobotFileParser.__str__N)r)�__name__�
__module__�__qualname__�__doc__rrrr
r"r-rrPrTrVrWr[rrrrrs
		
	I

c@s(eZdZdZdd�Zdd�Zdd�ZdS)	r=zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
       (allowance==False) followed by a path.cCs<|dkr|sd}tj�tj�|��}tj�|�|_||_dS)NrT)rrrGrrKrrM)r
rrMrrrr�s

zRuleLine.__init__cCs|jdkp
|�|j�Sr()r�
startswith)r
�filenamerrrrL�szRuleLine.applies_tocCs|jrdndd|jS)N�Allow�Disallowz: )rMrrrrrr[�szRuleLine.__str__N)r\r]r^r_rrLr[rrrrr=�s
r=c@s0eZdZdZdd�Zdd�Zdd�Zdd	�Zd
S)r5z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_d|_d|_dSrR)r*r<r@rArrrrr�s
zEntry.__init__cCs�g}|jD]
}|�d|���q|jdur|�d|j���|jdur3|j}|�d|j�d|j���|�tt|j	��d�
|�S)NzUser-agent: z
Crawl-delay: zRequest-rate: r3�
)r*r+r@rA�requests�seconds�extendrYrZr<rX)r
�ret�agent�raterrrr[�s



z
Entry.__str__cCsF|�d�d��}|jD]}|dkrdS|��}||vr dSqdS)z2check if this entry applies to the specified agentr3rr)TF)r8r:r*)r
rNrirrrrL�s
�zEntry.applies_tocCs$|jD]}|�|�r|jSqdS)zZPreconditions:
        - our agent applies to this entry
        - filename is URL decodedT)r<rLrM)r
rarDrrrrM
s



�zEntry.allowanceN)r\r]r^r_rr[rLrMrrrrr5�s
r5)r_�collections�urllib.parser�urllib.request�__all__�
namedtuplerrr=r5rrrr�<module>sB

Spamworldpro Mini