Skip to contents

Download Earth System Model (ESM) data using wget scripts

Source: R/htr_download_ESM.R
htr_download_ESM.Rd

This function downloads climate model data from remote repositories using wget scripts. It processes multiple wget scripts in parallel to efficiently download large climate datasets, typically from CMIP6 data nodes or similar repositories.

Usage

htr_download_ESM(hpc = NA, indir, outdir)

Arguments

hpc

Character string or NA. Indicates High Performance Computing mode:

  • NA: Standard processing mode

  • "array": HPC array job mode (requires file parameter)

  • "parallel": HPC parallel mode

indir

Character string. Directory containing wget script files. These are typically bash scripts with wget commands for downloading climate data from remote repositories (e.g., ESGF data nodes).

outdir

Character string. Directory where the downloaded NetCDF files will be saved. The function will change to this directory before executing wget scripts.

Value

No return value. The function downloads NetCDF files to the specified output directory as defined by the wget scripts.

Details

The function executes bash wget scripts that contain download commands for climate data files. It changes the working directory to the output directory before running each wget script to ensure files are downloaded to the correct location.

The process involves:

  1. Finding all wget script files in the input directory

  2. For each script, changing to the output directory

  3. Executing the wget script with the -s flag (silent mode)

  4. Restoring the original working directory

All wget scripts are processed in parallel using multiple workers for efficient downloading of large datasets.

Note

  • Requires wget to be installed and accessible from the system PATH

  • Wget scripts should be properly formatted bash scripts with appropriate download commands

  • The function temporarily changes working directory during execution

  • Uses parallel processing with (number of CPU cores - 2) workers

  • Ensure sufficient disk space is available for downloaded climate data

  • Network connectivity and access permissions to data repositories are required

References

ESGF Data Portal: https://esgf-node.llnl.gov/projects/esgf-llnl/ CMIP6 Data Access: https://pcmdi.llnl.gov/CMIP6/

Author

Dave Schoeman and Tin Buenafe

Examples

if (FALSE) { # \dontrun{
htr_download_ESM(
  hpc = NA,
  indir = file.path(base_dir, "data", "raw", "wget"), # input directory
  outdir = file.path(base_dir, "data", "raw", "tos") # output directory
)
} # }