RRTMGP introduces errors in restarted simulations

[Sbozzolo]

Testing with this configuration:

                    test_dict = Dict(
                        "test_dycore_consistency" => true, # We will add NaNs to the cache, just to make sure
                        "check_nan_every" => 3,
                        "log_progress" => false,
                        "moist" => "equil",
                        "precip_model" => "0M",
                        "config" => configuration,
                        "topography" => "NoWarp",
                        "turbconv" => "diagnostic_edmfx",
                        "dt" => "1secs",
                        "bubble" => bubble,
                        "viscous_sponge" => true,
                        "rayleigh_sponge" => true,
                        "insolation" => "timevarying",
                        "rad" => "clearsky",
                        "dt_rad" => "1secs",
                        "surface_setup" => "DefaultMoninObukhov",
                        # "implicit_diffusion" => true,
                        "call_cloud_diagnostics_per_stage" => true,  # Needed to ensure that cloud variables are computed
                        "t_end" => "3secs",
                        "dt_save_state_to_disk" => "1secs",
                        "enable_diagnostics" => false,
                        "output_dir" => joinpath(output_loc, job_id),
                    )

Here are some of the differences (there's more)

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669424f0 <= 1.1920929f-5
     Context: name = integrator.u.c.uₕ.components.data.1
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669423f0 <= 1.1920929f-5
     Context: name = integrator.u.c.uₕ.components.data.2
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 1.6163609f-5 <= 1.1920929f-5
     Context: name = integrator.u.f.u₃.components.data.1
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669424f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜu.components.data.1
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669423f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜu.components.data.2
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669424f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜuʲs.1.components.data.1
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669423f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜuʲs.1.components.data.2
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669424f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜu⁰.components.data.1
              what = reading and simulating
              config = sphere equil 0M NoWarp

Test Failed at /home/sbozzolo/repos/ClimaAtmos.jl/test/restart.jl:42
  Expression: maximum(error) <= 100 * eps(eltype(arr1))
   Evaluated: 0.3669423f0 <= 1.1920929f-5
     Context: name = integrator.p.precomputed.ᶜu⁰.components.data.2
              what = reading and simulating
              config = sphere equil 0M NoWarp

[akshaysridhar]

Are these differences also true for scalars? Do we get exact matches across states if the default rad = nothing is used?