Core API#

fetchez.core#

This module is the core of the Fetchez library.

It handles the initialization of fetchers, connection pooling, threading, and the base FetchModule class.

copyright:

  1. 2010-2026 Regents of the University of Colorado

license:

MIT, see LICENSE for more details.

fetchez.core.fetches_callback(r)[source]#

Default callback for fetches processes. r: [url, local-fn, data-type, fetch-status-or-error-code]

Parameters:

r (List[Any])

fetchez.core.urlencode_(opts)[source]#

Encode opts for use in a URL.

Parameters:

opts (Dict)

Return type:

str

fetchez.core.urlencode(opts, doseq=True)[source]#

Encode opts for use in a URL.

Parameters:

  • opts (Dict) – Dictionary of query parameters.

  • doseq (bool, default: True) – If True, lists in values are encoded as separate parameters (e.g., {‘a’: [1, 2]} -> ‘a=1&a=2’).

Return type:

str

fetchez.core.xml2py(node)[source]#

Parse an xml file into a python dictionary.

Return type:

Optional[Dict]

fetchez.core.get_userpass(authenticator_url)[source]#

Retrieve username and password from netrc for a given URL.

Parameters:

authenticator_url (str)

Return type:

Tuple[Optional[str], Optional[str]]

fetchez.core.get_raw_credentials(url=None, authenticator_url='https://urs.earthdata.nasa.gov')[source]#

Get raw (username, password) from .netrc or interactive prompt. Optionally validate against an HTTP url.

Parameters:

  • url (Optional[str], default: None)

  • authenticator_url (str, default: 'https://urs.earthdata.nasa.gov')

Return type:

Tuple[Optional[str], Optional[str]]

fetchez.core.get_credentials(url=None, authenticator_url='https://urs.earthdata.nasa.gov')[source]#

Wrapper for get_raw_credentials that returns a Base64 Basic Auth string.

Parameters:

  • url (Optional[str], default: None)

  • authenticator_url (str, default: 'https://urs.earthdata.nasa.gov')

Return type:

Optional[str]

class fetchez.core.iso_xml(url=None, xml=None, timeout=20, read_timeout=60)[source]#

Bases: object

Helper class for parsing ISO 19115 XML Metadata.

__init__(url=None, xml=None, timeout=20, read_timeout=60)[source]#
title()[source]#

Extract Title.

abstract()[source]#

Extract Abstract.

date()[source]#

Extract Date.

linkages()[source]#

Extract first valid download URL (specifically looking for Zips/Data).

polygon(geom=True)[source]#

Extract Bounding Box and return GeoJSON Polygon.

class fetchez.core.HttpFile(url, session=None, callback=None)[source]#

Bases: IOBase

A file-like object backed by an HTTP URL.

Translates read() calls into HTTP Range requests to fetch only needed bytes.

__init__(url, session=None, callback=None)[source]#
seek(offset, whence=0)[source]#

Change the stream position to the given byte offset.

offset

The stream position, relative to ‘whence’.

whence

The relative position to seek from.

The offset is interpreted relative to the position indicated by whence. Values for whence are:

  • os.SEEK_SET or 0 – start of stream (the default); offset should be zero or positive

  • os.SEEK_CUR or 1 – current stream position; offset may be negative

  • os.SEEK_END or 2 – end of stream; offset is usually negative

Return the new absolute position.

tell()[source]#

Return current stream position.

read(size=-1)[source]#
class fetchez.core.fetchezSession(rauth=None, rheaders=None, **kwargs)[source]#

Bases: Session

__init__(rauth=None, rheaders=None, **kwargs)[source]#
rebuild_auth(prepared_request, response)[source]#

Intercept the security sweep to preserve Earthdata credentials.

class fetchez.core.Fetch(url, callback=<function fetches_callback>, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'}, verify=True, allow_redirects=True, auth=None)[source]#

Bases: object

Fetch class to fetch ftp/http data files

Parameters:

  • url (str)

  • headers (Dict, default: {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'})

  • verify (bool, default: True)

  • allow_redirects (bool, default: True)

  • auth (Optional[Any], default: None)

__init__(url, callback=<function fetches_callback>, headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'}, verify=True, allow_redirects=True, auth=None)[source]#
Parameters:

  • url (str)

  • headers (Dict, default: {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:146.0) Gecko/20100101 Firefox/146.0'})

  • verify (bool, default: True)

  • allow_redirects (bool, default: True)

  • auth (Optional[Any], default: None)

fetch_req(method='GET', params=None, data=None, json=None, tries=5, timeout=30, read_timeout=120)[source]#

Fetch src_url and return the requests object (iterative retry).

Parameters:
Return type:

Optional[Response]

fetch_html(timeout=2)[source]#

Fetch src_url and return it as an HTML object.

fetch_xml(timeout=2, read_timeout=10)[source]#

Fetch src_url and return it as an XML object.

fetch_file(dst_fn, method='GET', params=None, datatype=None, overwrite=False, timeout=30, read_timeout=120, tries=5, check_size=True, verbose=True)[source]#

Fetch src_url and save to dst_fn with resume support.

Parameters:

dst_fn (str)

Return type:

int

fetch_ftp_file(dst_fn, params=None, datatype=None, overwrite=False)[source]#

Fetch an ftp file via ftplib with a progress bar.

fetchez.core.run_fetchez(modules, threads=3, global_hooks=None)[source]#

Run Fetchez in parallel with hooks.

  • mod.hooks: Run ONLY on entries belonging to ‘mod’.

  • global_hooks: Run on ALL entries combined.

Parameters: