Server : Apache/2.4.52 (Ubuntu) System : Linux webserver 6.8.0-49-generic #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 6 17:42:15 UTC 2 x86_64 User : www-data ( 33) PHP Version : 8.1.2-1ubuntu2.21 Disable Function : NONE Directory : /lib/python3/dist-packages/landscape/lib/__pycache__/ |
o
���c� � @ s� d dl Z d dlZd dlZd dlmZ d dlmZ d dlmZ d dl m
Z
m Z G dd� de �Z
G dd � d e
�ZG d
d � d e
�Zd d
i ddddd dddf dd�Zdd� Zd dd�Zd dd�Zd dd�Zdd � Zed krteejd d� � dS dS )!� N)� OptionParser)� DeferredList)�
deferToThread)� iteritems�
networkStringc @ s e Zd ZdS )�
FetchErrorN)�__name__�
__module__� __qualname__� r r �5/usr/lib/python3/dist-packages/landscape/lib/fetch.pyr s r c @ s$ e Zd Zdd� Zdd� Zdd� ZdS )�
HTTPCodeErrorc C � || _ || _d S �N)� http_code�body)�selfr r r r r �__init__ �
zHTTPCodeError.__init__c C �
d| j S )Nz Server returned HTTP code %d�r �r r r r �__str__ �
zHTTPCodeError.__str__c C r )Nz <HTTPCodeError http_code=%d>r r r r r �__repr__ r zHTTPCodeError.__repr__N)r r r
r r r r r r r r
s r
c @ s0 e Zd Zdd� Zdd� Zdd� Zedd� �Zd S )
� PyCurlErrorc C r r )�
error_code�_message)r r �messager r r r r zPyCurlError.__init__c C � d| j | jf S )Nz Error %d: %s�r r r r r r r # s zPyCurlError.__str__c C r )Nz <PyCurlError args=(%d, '%s')>r r r r r r & s �zPyCurlError.__repr__c C s | j S r )r r r r r r * s zPyCurlError.messageN)r r r
r r r �propertyr r r r r r s r F� � iX Tc
C s ddl } t|t�s|�d�}t�|�}
t�� }|du r | �� }|�| jt t
| ��� |rF|�| j d� |rF|�| j t
|�� |�| j|
j� |rV| �d�rV|�| jt |�� |rh|�| jdd� tt|��D �� |rq|�| jd� | rz|�| jd � |
dur�|�| jt |
�� | dur�|�| jt | �� |�| jd
� |�| j|� |�| jd � |�| j |� |�| j d � |�| j |j � |�| j d� |�| j!d � z|�"� W n | j#y� } z t$|j%d |j%d ��d}~ww |�&� }|�'| j(�}|d kr�t)||��|S )
aC Retrieve a URL and return the content.
@param url: The url to be fetched.
@param post: If true, the POST method will be used (defaults to GET).
@param data: Data to be sent to the server as the POST content.
@param headers: Dictionary of header => value entries to be used on the
request.
@param curl: A pycurl.Curl instance to use. If not provided, one will be
created.
@param cainfo: Path to the file with CA certificates.
@param insecure: If true, perform curl using insecure option which will
not attempt to verify authenticity of the peer's certificate. (Used
during autodiscovery)
@param follow: If True, follow HTTP redirects (default True).
@param user_agent: The user-agent to set in the request.
@param proxy: The proxy url to use for the request.
r Nzutf-8Tzhttps:c S s g | ]}d | �qS )z%s: %sr )�.0�pairr r r �
<listcomp>\ s zfetch.<locals>.<listcomp>F� � s gzip,deflate�� )*�pycurl�
isinstance�bytes�encode�io�BytesIO�Curl�setopt�URLr �str�POST�
POSTFIELDSIZE�len� READFUNCTION�read�
startswith�CAINFO�
HTTPHEADER�sortedr �SSL_VERIFYPEER�FOLLOWLOCATION� USERAGENT�PROXY� MAXREDIRS�CONNECTTIMEOUT�LOW_SPEED_LIMIT�LOW_SPEED_TIME�NOSIGNAL�
WRITEFUNCTION�write�DNS_CACHE_TIMEOUT�ENCODING�perform�errorr �args�getvalue�getinfo� HTTP_CODEr
)�url�post�data�headers�cainfo�curl�connect_timeout�
total_timeout�insecure�follow�
user_agent�proxyr* �output�input�er r r r r �fetch/ s\
� ��
r_ c O s t tg| �R i |��S )z]Retrieve a URL asynchronously.
@return: A C{Deferred} resulting in the URL content.
)r r_ )rL �kwargsr r r � fetch_async� s ra c K sV g }| D ] }t |fi |��}|r|�||� |r |�||� |�|� qt|ddd�S )a9
Retrieve a list of URLs asynchronously.
@param callback: Optionally, a function that will be fired one time for
each successful URL, and will be passed its content and the URL itself.
@param errback: Optionally, a function that will be fired one time for each
failing URL, and will be passed the failure and the URL itself.
@return: A C{DeferredList} whose callback chain will be fired as soon as
all downloads have terminated. If an error occurs, the errback chain
of the C{DeferredList} will be fired immediatly.
T)�fireOnOneErrback�
consumeErrors)ra � addCallback�
addErrback�appendr )�urls�callback�errbackr` �resultsrP �resultr r r �fetch_many_async� s rl c C s. | � d��d�d }|durtj�||�}|S )z�Return the last component of the given C{url}.
@param url: The URL to get the filename from.
@param directory: Optionally a path to prepend to the returned filename.
@note: Any trailing slash in the C{url} will be removed
�/���N)�rstrip�split�os�path�join)rP � directory�filenamer r r �url_to_filename� s rv c s. � fdd�}�fdd�}t | f||d�|��S )aJ
Retrieve a list of URLs and save their content as files in a directory.
@param urls: The list URLs to fetch.
@param directory: The directory to save the files to, the name of the file
will equal the last fragment of the URL.
@param logger: Optional function to be used to log errors for failed URLs.
c s, t |� d�}t|d�}|�| � |�� d S )N�rt �wb)rv �openrG �close)rR rP ru �fdrw r r rG � s
z fetch_to_files.<locals>.writec s � r
� d|t | j�f � | S )Nz Couldn't fetch file from %s (%s))r3 �value)�failurerP )�loggerr r � log_error� s
�z!fetch_to_files.<locals>.log_error)rh ri )rl )rg rt r~ r` rG r r )rt r~ r �fetch_to_files� s
r� c C sZ t � }|jddd� |jddd� |�d� |�| �\}\}tt||j|j|jd�� d S ) Nz--post�
store_true)�actionz--datar" )�defaultz--cainfo)rQ rR rT )r �
add_option�
parse_args�printr_ rQ rR rT )rL �parser�optionsrP r r r �test� s
�r� �__main__r' )NNr )rq �sysr. �optparser �twisted.internet.deferr �twisted.internet.threadsr �twisted.python.compatr r � Exceptionr r
r r_ ra rl rv r� r� r �argvr r r r �<module> s,
�R
�