9.2. Fetchers — `MDAnalysis.fetch.fetchers`

This module contains Fetcher classes which are able to retrieve files from remote servers.These classes use the third-party library pooch as a dependency.

9.2.1. Classes

class MDAnalysis.fetch.fetchers.StaticFetcher(cache_path=None, hash='sha256')[source]

Downloads files from a URL to disk and caches them to a local directory.

Parameters:

cache_path (str or pathlib.Path) – Path to the cache directory. If set to None, the default cache directory will be used as specified by DEFAULT_CACHE_NAME_DOWNLOADER. If the directory does not exist, it will attempt to be created.
hash (str) – Hash algorithm to use for verifying the integrity of downloaded files. The default is sha256. Valid options are any hash algorithm available in the hashlib module.

cache_path

Path to the cache directory.

Type:: pathlib.Path

hash

Hash algorithm used for verifying the integrity of downloaded files.

Type:: str

Notes

The download directory can be overridden by setting the environment variable MDANALYSIS_FETCHER_DATA to a valid path. This class uses pooch as a backend for downloading and caching files.

New in version 2.11.0.

append_registry(db_path, files, write_duplicate=False)[source]

Append cached files to an existing Pooch registry.

Each entry in files is resolved relative to cache_path. The file hash is computed using the fetcher’s configured hash algorithm hash and a new registry line is appended to db_path.

Parameters:

db_path (str or path-like) – Path to the registry file to update.
files (iterable of str or path-like) – File names or paths for cached files to append to the registry. Relative paths are interpreted relative to cache_path.
write_duplicate (bool) – If set to True, append_registry will write the file and its hash to the registry regardless of the existing presence of a entry in the registry. Default behavior is False.

Returns:

This method updates the registry file in place and does not return a value.

Return type:

None

Example

>>> from MDAnalysis.fetch.fetchers import StaticFetcher
>>> fetcher = StaticFetcher()
>>> file1 = fetcher.fetch(
...     file_name="1AKE.cif",
...     base_url="https://files.wwpdb.org/download/",
...     db_name="db_hash1.txt",
... )
>>> file2 = fetcher.fetch(
...     file_name="4AKE.cif",
...     base_url="https://files.wwpdb.org/download/",
...     db_name="db_hash2.txt",
... )
>>> registry = file1.parent / "db_hash1.txt"
>>> fetcher.append_registry(registry, ["4AKE.cif"])
>>> registry.read_text()
1AKE.cif sha256:01f41b1b42318a1a5df7f650dbab881677aa0e8d825f7c42dd26ae16a94c0948
4AKE.cif sha256:fcb2ff49a3e255797fee277ce28e0acace67f6e6ddf432841f8451f00cbde9e9

Notes

Existing registry entries are preserved. This method does not check for or remove duplicate file entries.

Each appended registry line has the format:

<filename> <hash_algorithm>:<digest>

New in version 2.11.0.

check_registry(db_path, files=None, ignore=None)[source]

Return paths relative to cache_path for cache files that are missing from the registry.

This method compares filenames within the registry against files found recursively under cache_path. A cache file is considered missing when it is on disk, but it is not recorded in the registry.

Parameters:

db_path (str or path-like) – Path to the registry file to read.
files (list of pathlib.Path) – Paths to additional files to check. Each path must be relative to cache_path.
ignore (list of pathlib.Path) – Files to be ignored. Each path must be relative to cache_path.

Returns:

missing_files – Cache file paths whose filenames are not present in the registry. The registry database file itself is excluded from the result.

Return type:

list of pathlib.Path

Example

>>> files = {
...     "file1.txt": "Molecular \n",
...     "file2.txt": "Dynamics. \n",
...     "file3.txt": "Analysis. \n"
... }
>>> for filename, content in files.items():
...     with open(filename, "w") as f:
...         f.write(content)
...
>>> fetcher = StaticFetcher()
>>> fetcher.write_registry(
...     "file_1_2_and_3_hash.txt",
...     files=["file1.txt"],
... )
>>> fetcher.check_registry("file_1_2_and_3_hash.txt")
[Path('./MDAnalysis_pdbs/file3.txt'), Path('./MDAnalysis_pdbs/file2.txt')]
>>> fetcher.check_registry("file_1_2_and_3_hash.txt", ignore=["file2.txt"])
[Path('./MDAnalysis_pdbs/file3.txt')]

Notes

Each line in the registry file is expected to have the format:

<filename> <hash_algorithm>:<digest>

fetch(base_url, file_name, verbose=False, db_name='hashes.txt', append_db=False, downloader='auto', **kwargs)[source]

Download one or more files from a static base URL and cache them locally.

Primarily designed to be working with FAIR databases, this method works by sending a request to a web server and caching them to a registry.The registry is in the format of a pooch registry file, and it will be created or read relative to cache_path.

Parameters:

base_url (str) – Base URL from which to download the file(s). This should be a valid URL pointing to the directory containing the files to be downloaded.
file_name (str or sequence of str) – Name of the file or files to download. The requested URL has the form {base_url}/{file_name}.
verbose (bool, optional) – If True, show download progress. The default is False.
db_name (str or None, optional) – Name of the local hash database file used to verify cached downloads. The default is "hashes.txt". If None, no registry database is read or written.
append_db (bool, optional) – If True, add downloaded files that are missing from an existing registry to that registry. If False, missing registry entries raise a ValueError. The default is False.
timeout (float, optional) – Time in seconds to wait for a response from the server before timing out. The default is DEFAULT_TIMEOUT.
retries (int, optional) – Number of times to retry a failed download. The default is DEFAULT_RETRIES.
downloader (str, optional) – Downloader backend to use. Supported values are "auto", "http", "ftp", "sftp", and "doi". The default is "auto".

Returns:

The downloaded file path for a single file, or a list of paths for multiple files.

Return type:

pathlib.Path or list of pathlib.Path

Examples

Download a single CIF file from the RCSB Protein Data Bank.

>>> StaticFetcher().fetch(file_name="1AKE.cif",
    base_url="https://files.wwpdb.org/download/")
'./MDAnalysis_pdbs/1AKE.cif'

Download multiple CIF files from the RCSB Protein Data Bank.

>>> StaticFetcher().fetch(file_name=["1AKE.cif", "4AKE.cif"],
    base_url="https://files.wwpdb.org/download/")
['./MDAnalysis_pdbs/1AKE.cif', './MDAnalysis_pdbs/4AKE.cif']

Notes

The download directory can be overridden by setting the environment variable MDANALYSIS_FETCHER_DATA to a valid path. This class uses pooch as a backend for downloading and caching files. The cache database is created on demand when db_name does not exist relative to cache_path.

New in version 2.11.0.

read_registry(db_path)[source]

Read a Pooch registry file into a dictionary.

This method returns filenames within the registry against files found recursively under cache_path. Each key in the returned dictionary corresponds to a filename in the registry relative to cache_path.

Parameters:: db_path (str or path-like) – Path to the registry file to read.
Returns:: hash_dict – Dictionary mapping each filename in the registry to its stored hash value. Hash values are expected to include the hash algorithm prefix.
Return type:: dict

Example

>>> files = {
...     "file1.txt": "Molecular \n",
...     "file2.txt": "Dynamics. \n",
... }
>>> for filename, content in files.items():
...     with open(filename, "w") as f:
...         f.write(content)
...
>>> fetcher = StaticFetcher()
>>> fetcher.write_registry(
...     "file_1_and_2_hash.txt",
...     ["file1.txt", "file2.txt"],
... )
>>> fetcher.read_registry("file_1_and_2_hash.txt")
{'file1.txt': 'sha256:2da169c5aae36a823c202da49fb11935b76277efcb5cd42a4cf238ddda2a9b20',
'file2.txt': 'sha256:3a0dbd9e2abc4a7bbae6adfe92e2858218135926dacd4a7d3fb4ca2dbdbe457a'}

Notes

Each line in the registry file is expected to have the format:

<filename> <hash_algorithm>:<digest>

New in version 2.11.0.

write_registry(db_path, files, mode='w')[source]

Write a Pooch registry file with hashes for the given files.

This method computes the hash for each file and writes it to the registry file.The registry file maps each filename to its corresponding hash value. The hash algorithm used is determined by the hash of the fetcher.

Parameters:

db_path (str or path-like) – Path to the registry file to write.
files (iterable of str or path-like) – Files to be include in the registry. Each file must be relative to cache_path.
mode (str, optional) – File opening mode used when writing the registry. Default is "w".

Returns:

This method writes the registry to disk and does not return a value.

Return type:

None

Example

>>> files = {
...     "file1.txt": "Molecular \n",
...     "file2.txt": "Dynamics. \n",
... }
>>> for filename, content in files.items():
...     with open(filename, "w") as f:
...         f.write(content)
...
>>> fetcher = StaticFetcher()
>>> fetcher.write_registry(
...     "file_1_and_2_hash.txt",
...     ["file1.txt", "file2.txt"],
... )
>>> Path("file_1_and_2_hash.txt").read_text()
file1.txt sha256:2da169c5aae36a823c202da49fb11935b76277efcb5cd42a4cf238ddda2a9b20
file2.txt sha256:3a0dbd9e2abc4a7bbae6adfe92e2858218135926dacd4a7d3fb4ca2dbdbe457a

Notes

Each registry line is written in the format:

<filename> <hash_algorithm>:<digest>

New in version 2.11.0.

9.2.2. Variables

These module-level variables affect the runtime behavior across all Fetcher classes. Changing these values affects all initialized Fetchers.

MDAnalysis.fetch.fetchers.DEFAULT_CACHE_NAME_DOWNLOADER = 'MDAnalysis_pdbs'

Name of the pooch cache directory pooch.os_cache(DEFAULT_CACHE_NAME_DOWNLOADER);

See pooch.os_cache() for further details.

New in version 2.11.0.

MDAnalysis.fetch.fetchers.DEFAULT_TIMEOUT = 10: Default time in seconds to wait for a response from the server before timing out.

New in version 2.11.0.

MDAnalysis.fetch.fetchers.DEFAULT_RETRIES = 2: Default number of attempts to retry a download if it fails.

New in version 2.11.0.

9.2. Fetchers — MDAnalysis.fetch.fetchers

9.2.1. Classes

9.2.2. Variables

9.2. Fetchers — `MDAnalysis.fetch.fetchers`