Improve regression debug info during move #3324

charleskawczynski · 2024-09-23T15:31:00Z

It seems that too many files are moved to the reference path, and I'm trying to understand why #3319 is failing, so this PR changes the readdir to printing all of the files (readdir is not recursive) moved to the reference path when all files are moved on central.

Sbozzolo · 2024-09-23T15:53:22Z

Is this only printed for jobs with regression tests?

Sbozzolo · 2024-09-23T16:08:40Z

This is what this PR is currently printing for a job with table:

[ Info: Files in /central/scratch/esm/slurm-buildkite/climaatmos-main/21de7fe:
[ Info:    File:`aquaplanet_rhoe_equil_clearsky_tvinsol_0M_slabocean`
[ Info:    File:`aquaplanet_rhoe_equil_clearsky_tvinsol_0M_slabocean_ft64`
[ Info:    File:`box_density_current_test`
[ Info:    File:`box_hydrostatic_balance_rhoe`
[ Info:    File:`central_cloud_diag_gpu_hs_rhoe_equil_55km_nz63_0M`
[ Info:    File:`central_gpu_hs_rhoe_equil_55km_nz63_0M`
[ Info:    File:`deep_sphere_baroclinic_wave_rhoe_equilmoist`
[ Info:    File:`default_config`
[ Info:    File:`diagnostic_edmfx_aquaplanet`
[ Info:    File:`diagnostic_edmfx_aquaplanet_gpu`
[ Info:    File:`diagnostic_edmfx_bomex_box`
[ Info:    File:`diagnostic_edmfx_bomex_stretched_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf01_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf01_explicit_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf02_box`
[ Info:    File:`diagnostic_edmfx_gabls_box`
[ Info:    File:`diagnostic_edmfx_rico_box`
[ Info:    File:`diagnostic_edmfx_test_box`
[ Info:    File:`diagnostic_edmfx_trmm_box`
[ Info:    File:`diagnostic_edmfx_trmm_box_0M`
[ Info:    File:`diagnostic_edmfx_trmm_stretched_box`
[ Info:    File:`gpu_aquaplanet_dyamond`
[ Info:    File:`mpi_make_restart`
[ Info:    File:`mpi_sphere_aquaplanet_rhoe_equilmoist_clearsky`
[ Info:    File:`perf_benchmarks.json`
[ Info:    File:`plane_agnesi_mountain_test_stretched`
[ Info:    File:`plane_agnesi_mountain_test_uniform`
[ Info:    File:`plane_density_current_test`
[ Info:    File:`prep_remap`
[ Info:    File:`prognostic_edmfx_adv_test_column`
[ Info:    File:`prognostic_edmfx_aquaplanet`
[ Info:    File:`prognostic_edmfx_aquaplanet_gpu`
[ Info:    File:`prognostic_edmfx_bomex_box`
[ Info:    File:`prognostic_edmfx_bomex_column`
[ Info:    File:`prognostic_edmfx_bomex_column_implicit`
[ Info:    File:`prognostic_edmfx_bomex_fixtke_column`
[ Info:    File:`prognostic_edmfx_bomex_pigroup_column`
[ Info:    File:`prognostic_edmfx_bomex_stretched_column`
[ Info:    File:`prognostic_edmfx_dycoms_rf01_column`
[ Info:    File:`prognostic_edmfx_gabls_column`
[ Info:    File:`prognostic_edmfx_gcmdriven_column`
[ Info:    File:`prognostic_edmfx_rico_column`
[ Info:    File:`prognostic_edmfx_simpleplume_column`
[ Info:    File:`prognostic_edmfx_trmm_column`
[ Info:    File:`prognostic_edmfx_trmm_column_0M`
[ Info:    File:`rcemipii_box_diagnostic_edmfx`
[ Info:    File:`rcemipii_sphere_diagnostic_edmfx`
[ Info:    File:`ref_counter.jl`
[ Info:    File:`restart_mpi_baroclinic_wave_rhoe`
[ Info:    File:`single_column_hydrostatic_balance_ft64`
[ Info:    File:`single_column_nonorographic_gravity_wave`
[ Info:    File:`single_column_precipitation_test`
[ Info:    File:`single_column_radiative_equilibrium_allsky_idealized_clouds`
[ Info:    File:`single_column_radiative_equilibrium_clearsky`
[ Info:    File:`single_column_radiative_equilibrium_clearsky_prognostic_surface_temp`
[ Info:    File:`single_column_radiative_equilibrium_gray`
[ Info:    File:`sphere_aquaplanet_rhoe_equilmoist_allsky_gw_raw_zonallyasymmetric`
[ Info:    File:`sphere_aquaplanet_rhoe_equilmoist_allsky_gw_res`
[ Info:    File:`sphere_aquaplanet_rhoe_nonequilmoist_allsky`
[ Info:    File:`sphere_baroclinic_wave_conservation`
[ Info:    File:`sphere_baroclinic_wave_conservation_ft64`
[ Info:    File:`sphere_baroclinic_wave_conservation_source`
[ Info:    File:`sphere_baroclinic_wave_rhoe`
[ Info:    File:`sphere_baroclinic_wave_rhoe_equilmoist`
[ Info:    File:`sphere_baroclinic_wave_rhoe_gpu`
[ Info:    File:`sphere_baroclinic_wave_rhoe_topography_dcmip_rs`
[ Info:    File:`sphere_held_suarez_rhoe_equilmoist_hightop_sponge`
[ Info:    File:`sphere_held_suarez_rhoe_equilmoist_topography_dcmip`
[ Info:    File:`sphere_held_suarez_rhoe_hightop`
[ Info:    File:`sphere_held_suarez_rhoe_topography_dcmip`
[ Info:    File:`sphere_hydrostatic_balance_rhoe_ft64`
[ Info:    File:`sphere_nonorographic_gravity_wave`
[ Info:    File:`sphere_ssp_baroclinic_wave_rhoe_equilmoist_dcmip200`
[ Info:    File:`sphere_ssp_baroclinic_wave_rhoe_equilmoist_earth`
[ Info:    File:`target_gpu_implicit_baroclinic_wave`
[ Info:    File:`target_gpu_implicit_baroclinic_wave_4process`
[ Info: Comparing results against main path:/central/scratch/esm/slurm-buildkite/climaatmos-main/21de7fe
[ Info: `ds_filename_computed`: `deep_sphere_baroclinic_wave_rhoe_equilmoist/output_0000/prog_state.nc`
[ Info: `ds_filename_reference`: `/central/scratch/esm/slurm-buildkite/climaatmos-main/21de7fe/deep_sphere_baroclinic_wave_rhoe_equilmoist/output_0000/prog_state.nc`
[ Info: Regression tables for `deep_sphere_baroclinic_wave_rhoe_equilmoist`
┌────────────┬────────────┬────────────┬────────────────────────┬────────────────────────┬───────────────┐
│   Variable │ Data scale │ Data scale │                    MSE │                    MSE │           MSE │
│            │   Computed │  Reference │               Computed │              Reference │ Reduction (%) │
├────────────┼────────────┼────────────┼────────────────────────┼────────────────────────┼───────────────┤
│      c_rho │   0.622318 │   0.622318 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
│ c_rhoe_tot │    22476.6 │    22476.6 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
│    c_u_h_1 │    7.88896 │    7.88896 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
│    c_u_h_2 │   0.781445 │   0.781445 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
│ c_rhoq_tot │ 0.00352221 │ 0.00352221 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
│     f_u₃_1 │ 0.00137855 │ 0.00137855 │ 0.0000000000000000e+00 │ 0.0000000000000000e+00 │           NaN │
└────────────┴────────────┴────────────┴────────────────────────┴────────────────────────┴───────────────┘
[ Info: Callback verification, n_expected_calls: AbstractFloat[1920.0f0, 1.875f0, 1920.0f0, 5.0, 80.0f0]
[ Info: Callback verification, n_measured_calls: [[1920], [2], [0], [6], [81]]
[ Info: Plotting
[ Info: Files in /central/scratch/esm/slurm-buildkite/climaatmos-main/21de7fe:
[ Info:    File:`aquaplanet_rhoe_equil_clearsky_tvinsol_0M_slabocean`
[ Info:    File:`aquaplanet_rhoe_equil_clearsky_tvinsol_0M_slabocean_ft64`
[ Info:    File:`box_density_current_test`
[ Info:    File:`box_hydrostatic_balance_rhoe`
[ Info:    File:`central_cloud_diag_gpu_hs_rhoe_equil_55km_nz63_0M`
[ Info:    File:`central_gpu_hs_rhoe_equil_55km_nz63_0M`
[ Info:    File:`deep_sphere_baroclinic_wave_rhoe_equilmoist`
[ Info:    File:`default_config`
[ Info:    File:`diagnostic_edmfx_aquaplanet`
[ Info:    File:`diagnostic_edmfx_aquaplanet_gpu`
[ Info:    File:`diagnostic_edmfx_bomex_box`
[ Info:    File:`diagnostic_edmfx_bomex_stretched_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf01_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf01_explicit_box`
[ Info:    File:`diagnostic_edmfx_dycoms_rf02_box`
[ Info:    File:`diagnostic_edmfx_gabls_box`
[ Info:    File:`diagnostic_edmfx_rico_box`
[ Info:    File:`diagnostic_edmfx_test_box`
[ Info:    File:`diagnostic_edmfx_trmm_box`
[ Info:    File:`diagnostic_edmfx_trmm_box_0M`
[ Info:    File:`diagnostic_edmfx_trmm_stretched_box`
[ Info:    File:`gpu_aquaplanet_dyamond`
[ Info:    File:`mpi_make_restart`
[ Info:    File:`mpi_sphere_aquaplanet_rhoe_equilmoist_clearsky`
[ Info:    File:`perf_benchmarks.json`
[ Info:    File:`plane_agnesi_mountain_test_stretched`
[ Info:    File:`plane_agnesi_mountain_test_uniform`
[ Info:    File:`plane_density_current_test`
[ Info:    File:`prep_remap`
[ Info:    File:`prognostic_edmfx_adv_test_column`
[ Info:    File:`prognostic_edmfx_aquaplanet`
[ Info:    File:`prognostic_edmfx_aquaplanet_gpu`
[ Info:    File:`prognostic_edmfx_bomex_box`
[ Info:    File:`prognostic_edmfx_bomex_column`
[ Info:    File:`prognostic_edmfx_bomex_column_implicit`
[ Info:    File:`prognostic_edmfx_bomex_fixtke_column`
[ Info:    File:`prognostic_edmfx_bomex_pigroup_column`
[ Info:    File:`prognostic_edmfx_bomex_stretched_column`
[ Info:    File:`prognostic_edmfx_dycoms_rf01_column`
[ Info:    File:`prognostic_edmfx_gabls_column`
[ Info:    File:`prognostic_edmfx_gcmdriven_column`
[ Info:    File:`prognostic_edmfx_rico_column`
[ Info:    File:`prognostic_edmfx_simpleplume_column`
[ Info:    File:`prognostic_edmfx_trmm_column`
[ Info:    File:`prognostic_edmfx_trmm_column_0M`
[ Info:    File:`rcemipii_box_diagnostic_edmfx`
[ Info:    File:`rcemipii_sphere_diagnostic_edmfx`
[ Info:    File:`ref_counter.jl`
[ Info:    File:`restart_mpi_baroclinic_wave_rhoe`
[ Info:    File:`single_column_hydrostatic_balance_ft64`
[ Info:    File:`single_column_nonorographic_gravity_wave`
[ Info:    File:`single_column_precipitation_test`
[ Info:    File:`single_column_radiative_equilibrium_allsky_idealized_clouds`
[ Info:    File:`single_column_radiative_equilibrium_clearsky`
[ Info:    File:`single_column_radiative_equilibrium_clearsky_prognostic_surface_temp`
[ Info:    File:`single_column_radiative_equilibrium_gray`
[ Info:    File:`sphere_aquaplanet_rhoe_equilmoist_allsky_gw_raw_zonallyasymmetric`
[ Info:    File:`sphere_aquaplanet_rhoe_equilmoist_allsky_gw_res`
[ Info:    File:`sphere_aquaplanet_rhoe_nonequilmoist_allsky`
[ Info:    File:`sphere_baroclinic_wave_conservation`
[ Info:    File:`sphere_baroclinic_wave_conservation_ft64`
[ Info:    File:`sphere_baroclinic_wave_conservation_source`
[ Info:    File:`sphere_baroclinic_wave_rhoe`
[ Info:    File:`sphere_baroclinic_wave_rhoe_equilmoist`
[ Info:    File:`sphere_baroclinic_wave_rhoe_gpu`
[ Info:    File:`sphere_baroclinic_wave_rhoe_topography_dcmip_rs`
[ Info:    File:`sphere_held_suarez_rhoe_equilmoist_hightop_sponge`
[ Info:    File:`sphere_held_suarez_rhoe_equilmoist_topography_dcmip`
[ Info:    File:`sphere_held_suarez_rhoe_hightop`
[ Info:    File:`sphere_held_suarez_rhoe_topography_dcmip`
[ Info:    File:`sphere_hydrostatic_balance_rhoe_ft64`
[ Info:    File:`sphere_nonorographic_gravity_wave`
[ Info:    File:`sphere_ssp_baroclinic_wave_rhoe_equilmoist_dcmip200`
[ Info:    File:`sphere_ssp_baroclinic_wave_rhoe_equilmoist_earth`
[ Info:    File:`target_gpu_implicit_baroclinic_wave`
[ Info:    File:`target_gpu_implicit_baroclinic_wave_4process`
[ Info: Comparing results against main path:/central/scratch/esm/slurm-buildkite/climaatmos-main/21de7fe

But I don't see where this new block is being printed, so I am guessing that there's some problems

charleskawczynski · 2024-09-23T16:12:03Z

Is this only printed for jobs with regression tests?

This is only printed inside the if-block when a PR is getting merged (

ClimaAtmos.jl/regression_tests/move_output.jl

Line 25 in 21de7fe

if in_merge_queue

).

Sbozzolo · 2024-09-23T16:28:06Z

Is this only printed for jobs with regression tests?

This is only printed inside the if-block when a PR is getting merged (

ClimaAtmos.jl/regression_tests/move_output.jl

Line 25 in 21de7fe

if in_merge_queue

).

Got it, thanks! Then, looks good to me.

Improve regression debug info during move

00d9eb1

charleskawczynski requested a review from Sbozzolo September 23, 2024 15:31

Sbozzolo approved these changes Sep 23, 2024

View reviewed changes

charleskawczynski added this pull request to the merge queue Sep 23, 2024

Merged via the queue into main with commit dfd2c9c Sep 23, 2024
14 of 16 checks passed

charleskawczynski deleted the ck/print_more_debug_info branch September 23, 2024 18:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve regression debug info during move #3324

Improve regression debug info during move #3324

charleskawczynski commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024

charleskawczynski commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024

Improve regression debug info during move #3324

Improve regression debug info during move #3324

Conversation

charleskawczynski commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024

charleskawczynski commented Sep 23, 2024

Sbozzolo commented Sep 23, 2024