Landuse data tool (V1) fails to run on Derecho

[glemieux]

Testing out a new conda environment on derecho for ESCOMP/CTSM#2331 (comment) resulted in the following (unhelpful) error during regridding:

2:52 $ python luh2.py -l ~/scratch/luh2/LUH2_v2h_Historic/states.nc -s ~/scratch/luh2/staticData_quarterdeg.nc -r ~/scratch/luh2/surfdata_4x5_16pfts_Irrig_CMIP6_simyr2000_c170824.nc -w regridder.nc -o testout.nc
Input file dataset opened: /glade/u/home/glemieux/scratch/luh2/LUH2_v2h_Historic/states.nc
PrepDataset: LUH2
LUH2 dataset lat/lon boundary variables formatted and added as new variable for xESMF
data set updated for xESMF

Input file dataset opened: /glade/u/home/glemieux/scratch/luh2/surfdata_4x5_16pfts_Irrig_CMIP6_simyr2000_c170824.nc
PrepDataset: SurfData
Surface dataset dimensions renamed for xESMF
data set updated for xESMF

Input file dataset opened: /glade/u/home/glemieux/scratch/luh2/staticData_quarterdeg.nc

Defining regridder, method:  conservative
/glade/work/glemieux/conda-envs/ctsm_pylib/lib/python3.7/site-packages/xesmf/backend.py:56: UserWarning: Latitude is outside of [-90, 90]
  warnings.warn('Latitude is outside of [-90, 90]')
regridder saved to file:  regridder.nc

Regridding
skipping variable 1/21: time
skipping variable 2/21: lat
skipping variable 3/21: lon
regridding variable 4/21: primf
Killed

The exact same conda environment works on lobata, so I suspect this is a bandwidth issue of some sort?

[glemieux]

This does not appear to be an issue on perlmutter.

It might be worth reviewing this documentation: https://ncar-hpc-docs.readthedocs.io/en/latest/pbs/checking-memory-use/?h=memory#interactive-monitoring

Issue 1. Your Derecho or Casper job shows signs of a memory issue (e.g., the job log ends with "Killed" or "Bus Error") despite qhist reporting a higher requested than used memory amount.

[glemieux]

Per discussions with @ekluzek and @ckoven a few weeks back, I ran this on a develop queue on derecho, to see if it was an issue with running on the constraints of the login node, where I had been testing earlier. Running in batch mode with one node, I was able to successfully run the script to completion.