Radiance simulator Updates

Release notes

Version 3.2 Release Note

Radiance simulator v3.2 was released in July 2023. The following lists detail the changes made between versions 3.1 and 3.2.

RadSim capabilities

  • The radsim_geo_obs.py script now optionally outputs scan line and scan position columns in the output obs data file to aid matchups of simulations to observations.

NWP model-specific capabilities

  • Unified Model: enable 1.5m specific humidity (stash 3237) as an optional alternative input to existing 1.5m relative humidity (stash 3245) for the RTTOV 2m q input.

RTTOV interface

  • The radiative transfer model used for simulations has been updated to RTTOV v13.2 and new features of this model have been exploited (see below). RadSim v3.2 can only be used with RTTOV v13.2.
  • New SURFEM-Ocean emissivity model is selected via fastem_version=7. This is now the default microwave sea surface emissivity model.
  • New rayleigh_depol config namelist option (true by default).
  • New opdep13_gas_clip config namelist option (true by default).
  • New pol_mode config namelist option (“empirical”/1 by default) and new rttov_pol_coeff_file option to optionally specify full path to the pol coef file required for pol_mode=2 (if unspecified, code assumes this file is in the rttov_hydrotable_dir directory and has its unmodified default file name).
  • MFASIS-NN model is available via vis_scatt_model=4. The directory or full file path to the NN coef file are specified in rttov_mfasis_nn_dir and rttov_mfasis_nn_file respectively. NB the run_mfasis option is now a short-cut to enable MFASIS-NN rather than the look-up-table (LUT) version. MFASIS-LUT may be run by setting vis_scatt_model=3.

All bug fixes and updates for RadSim v2, v3.0, and v3.1 listed here have been applied in v3.2.

Version 3.1 Release Note

Radiance simulator v3.1 was released in Feburary 2022. The following lists detail the changes made between versions 3.0 and 3.1.

RadSim capabilities

  • RadSim v3.1 is compatible with RTTOV v13.0 and v13.1. The latest version of RTTOV is always recommended.
  • RadSim can optionally compute satellite zenith and azimuth angles for geostationary sensors. Activated by new option calc_geo_sat_angles, and calculated angles are for a geostationary sensor above location geo_sat_lat, geo_sat_lon, and at altitude geo_sat_height.
  • Footprint simulations: two new options have been added, write_footprint_file and read_footprint_file, that specify a separate netCDF file that can be used to create and subsequently read footprint data for footprint simulations in cases where the observation locations and footprints, and the model grid remain the same between runs such as for GEO sensors. This can speed up subsequent runs.

NWP model-specific capabilities

  • Enable ingest of ECMWF CAMS GRIB fields including CAMS aerosol species for aerosol-affected simulations.
  • ICON: allow optional ingest and use of liquid and ice cloud particle size fields for VIS/IR cloud simulations.
  • ICON: add alternative GRIB paramIds for cell latitude/longitude datasets for compatibility with newer versions of ecCodes.
  • All GRIB fields except ICON: previously GRIB files had to contain a multi-level field after the surface pressure field in order to correctly compute pressure levels from the coefficients stored in the GRIB file. This restriction no longer applies.
  • Unified Model: enable use of bulk cloud fraction (stash 266) if area cloud fraction (stash 265) is unavailable (but area cloud fraction should be used if possible).

Other updates

  • The code had the definitions of “validity” and “data” times the wrong way round. This has been addressed in the code, and the contents of the “validity_time” and “data_time” attributes in the output netCDF files are now swapped compared to previous releases. Where they differ, “data time” refers to the analysis time and “validity time” to the forecast time of specific fields.

All bug fixes and updates for RadSim v2 and v3.0 listed here have been applied in v3.1.

Version 3.0 Release Note

Radiance simulator v3.0 was released in June 2021. The following lists detail the changes made between versions 2.2 and 3.0.

RadSim capabilities

  • Enable simulation of satellite footprints: this is done by taking the mean radiance over all grid points that fall within an ellipse of user-specified dimensions around each observation.
  • New Python script radsim_geo_obs.py to generate obs data files for geostationary sensors.
  • New options write_tskinjac, write_wind10mjac, write_emissjac to output additional Jacobians for Tskin, 10m wind u/v components, and surface emissivity.
  • Option to output channel height assignments consistent with the NWP SAF CADS (Cloud/Aerosol Detection Software) package. Activated by setting the new config namelist variable cads_height_assign_threshold to a positive value (usually 0.01) representing the threshold.
  • New option write_geom_height to output geometric heights of pressure levels calculated by RTTOV.

NWP model-specific capabilities

  • Support for ingest of HARMONIE GRIB fields.
  • Support for ingest of JMA GRIB files (clear-sky simulations only).
  • A new option use_all_atlas_months has been added for use with the NWP SAF profile datasets. If enabled, all 12 months of emissivity and/or BRDF atlas data are loaded so that the correct monthly emissivities/BRDFs can be used with each profile. Note that this may require a lot of memory.

RTTOV interface

  • The radiative transfer model used for simulations has been updated to RTTOV v13.0 and new features of this model have been exploited (see below). RadSim v3.0 cannot be used with RTTOV v12.
  • New configuration namelist variable rttov_coeffs_options which can be used to specify additional text in the optical depth coefficient filenames (e.g. “_o3co2” or “_ironly”).
  • New configuration namelist variable ssu_co2_cell_pressure which can be used to specify the cell pressures when using the SSU PMC shift coefficient files.
  • Updated universal gas constant to latest value from NIST consistent with RTTOV v13.
  • Remove deprecated RTTOV options fix_hgpl and reg_limit_extrap and use RTTOV v13 default option values.
  • New RTTOV options available: rayleigh_single_scatt, rayleigh_max_wavelength, rayleigh_min_pressure, dom_rayleigh (the last only available with cloud scattering, not clear-sky). Also ice_polarisation for RTTOV-SCATT.
  • Change default VIS/IR cloud ice parameterisation (ircloud_ice_scheme) to the Baran 2018 scheme.
  • Extend support for RTTOV VIS/IR CLW Deff scheme to all input models by using the RTTOV v13 internal parameterisation of effective diameter (Deff).
  • Update the RadSim CLW Deff parameterisation for ICON (using the density field) to be consistent with the RTTOV v13 implementation of Martin et al (1994).
  • RTTOV-SCATT updates to use the new default NWP SAF hydrotable files. For UM fields, the frozen cloud concentration is assigned to cloud ice since totalice no longer exists.
  • Implement flux conversion to kg/kg in RadSim for rain/snow (the RTTOV-SCATT flux conversion feature is deprecated). Output rain/snow fields are in kg/kg regardless of input data units (affects NWP SAF profile datasets and UM fields).
  • New config variables default_brdf_land and default_brdf_seaice which can be used to override the RTTOV default land/seaice BRDFs either where the BRDF atlas is not used or where the atlas has no data.

Technical updates

  • New configuration namelist variable output_file allows optional specification of output file name.
  • If output_file is unspecified, default output filenames are now based on the data validity time (e.g. the forecast time) of the first set of fields in the NWP model file rather than the nominal validity time (e.g. analysis time).
  • The GRIB API library is no longer supported as it is deprecated. RadSim must be compiled against the ecCodes library.
  • Updated radsim_plot_example.py script to enable plotting diffs of datasets.

Internal/other changes

  • The RTTOV option to supply cloud concentrations to RTTOV as layer averages is now used: internal change, this has no impact on outputs.
  • Disable ingest and use of individual liquid/ice cloud fractions for VIS/IR cloud simulations as this is not currently a recommended way of running cloudy RTTOV simulations.
  • Relative humidities calculated by RadSim are clipped to a minimum value of 0.1% in order to avoid negative values that sometimes occurred in the high atmosphere.

All bug fixes and updates for RadSim v2 listed here have been applied in v3.0.

Version 2.2 Release Note

Radiance simulator v2.2 was released in September 2019. The following lists contain details of the changes made between versions 2.1 and 2.2.

Enhancements:

  • The radiative transfer model used for simulations has been updated to RTTOV v12.3 and new features of this model have been exploited (see below). RadSim v2.2 cannot be used with RTTOV v12.2.
  • RadSim enables simulations for all solar-affected channels supported by RTTOV. This includes clear-sky and scattering simulations, the latter using all RTTOV solvers including the MFASIS fast visible cloud parameterisation and use of the RTTOV land surface BRDF atlas.
  • RadSim enables clear-sky simulations using the PC-RTTOV and HTFRTC Principal Components-based models supported by RTTOV.
  • Ingest of fields from the DWD ICON model in GRIB format is now supported.
  • Ingest of ECMWF fields in netCDF format is now supported.
  • It is now possible to modify the background CO2 profile used in the simulations.
  • If the ECMWF fields total/low/medium/high cloud cover are present in input GRIB or netCDF files, they are read in, interpolated to observation locations, and written out with the other profile data.
  • New options in RTTOV v12.3 can be specified in the configuration namelist: the lambertian_fixed_angle option, the MW clw_cloud_top parameter, and the RTTOV-SCATT cc_threshold parameter.
  • The default value of the cldstr_threshold parameter is now 1E-5 which can speed up visible/IR cloudy simulations with only a very small impact on radiances. The default value of the ircloud_idg parameter is now 2 (the recommended value).
  • New configuration namelist variables have been added to allow all RTTOV coefficient filenames to be specified explicitly if desired, instead of relying on filenames to be constructed from the platform, satid and inst variables.
  • A new namelist option ircloud_clw_scheme allows the RTTOV cloud liquid water optical properties to be selected for visible/IR cloud simulations (currently for DWD ICON GRIB model data only).
  • A new configuration namelist variable write_latlon has been added to output only the profile latitude/longitude data which can be useful for plotting simulated radiances without writing out the full profiles.
  • Promote some internal variables/calculations to 64-bit reals to improve consistency in outputs between compilers, particularly for Jacobians. This changes outputs slightly compared to RadSim v2.1.
  • A new Python script radsim_plot_example.py is included which can be used to plot simulated radiances.

Bug fixes:
The bugs in RadSim v2.1 listed here are fixed in v2.2.

Version 2.1 Release Note (selected changes)

Radiance simulator v2.1 was released in June 2018. The following lists contain details of the changes made between versions 2.0 and 2.1.

Enhancements:

  • The radiative transfer model used for simulations has been updated to RTTOV v12.2 and new features of this model have been exploited (see below). RadSim v2.1 cannot be used with RTTOV v12.1.
  • The code allows temporal interpolation of fields at multiple forecast times with ECMWF GRIB files and UM fieldsfiles.
  • Additional columns for satellite azimuth angle and observation date/time (satazim, year, month, day, hour, minute) have been defined as optional inputs in version 2 of the observation data file.
  • Ozone has been enabled as an option with ECMWF GRIB files and the NWP SAF 60L profile dataset.
  • When an emissivity atlas has no data for a given location, RadSim sets the qflag_emiss flag and runs the simulation using fixed emissivity values for the profile. Previously such profiles were omitted from the simulations.
  • The RTTOV parallel interfaces may be used which enable multi-threaded simulations using OpenMP. The number of threads to use is specified by the new nthreads variable in the configuration namelist and the number of profiles passed to RTTOV per call can be specified using the new nprofs_per_call variable in the configuration namelist.
  • More RTTOV options (including new options in RTTOV v12.2) can be specified in the RadSim configuration namelist. New options are: addrefrac, do_lambertian, apply_band_correction, do_nlte_correction, cldstr_threshold, apply_reg_limits, reg_limit_extrap, interp_mode, fix_hgpl, lradiance, mw_clw_scheme, mw_clw_calc_on_coef_lev.
  • The output file format is now netCDF-4 classic using internal compression for array variables.
  • RadSim may be compiled against either the ecCodes library or the GRIB API library. The former has now superseded the latter.
  • A new Python script radsim_run.py has been created which can be used to generate a configuration namelist and then optionally run RadSim from the command line.

Bug fixes:
The bugs in RadSim v2.0 listed here are fixed in v2.1.

Version 2.0 Release Note (selected changes)

Radiance simulator v2.0 was released in May 2017. The following lists contain details of the changes made between versions 1.1 and 2.0.

Enhancements:

  • The radiative transfer model used for simulations has been updated to RTTOV-12 (the latest available release). RTTOV-11 is no longer supported.
  • The code has been restructured to allow batching of input profiles. This enables the simulation of enormous datasets (of order 10 million data points or more) in a single run which was previously not possible on standard hardware. The maximum number of profiles allowed in a batch is provided as a tuneable parameter. (Aside: this batching is separate from the existing batching of simulations according to output array size which may be tuned separately).
  • Maximum output array size (value in MB) has been added as a tuneable parameter.
  • Interpolation of land-sea-mask and orography has been added. In the former case the result is a land fraction. Interpolated values can be used in lieu of providing them in the observation meta-data input file but file values remain the default.
  • A new coastline flag has been added to indicate when the assumed surface type does not match the land fraction obtained from model LSM interpolation. The level of mismatch required to trigger the flag can be changed by the user.
  • The method of horizontal interpolation has been optimised better for use with multiple fields.
  • Interpolation of non-uniform rectangular grids has been added. Such grids are sometimes used in high-resolution limited area models.
  • Calculation of relative humidity from specific humidity has been added. This is not used in the simulations but is useful to have when model profiles are written to the output file.
  • Use of both types of emissivity atlas for each of IR and MW has been enabled.
  • The default sea-ice fraction threshold for the classification of sea/sea-ice has been changed from 0.5 to 0.2. This is now in line with the Met Office operational classification of sea-ice for satellite data assimilation and also the definition used by The National Snow and Ice Data Center (NSDIC).
  • A new version of the observation meta-data file has been introduced. This allows more flexibility in the provision of input data with a user-specified number and ordering of columns.
  • Additional columns (viewid, scanline, scanpos) have been defined as possible inputs in version 2 of the observation data file.

Bug fixes:

  • Fixed calculation of the first (lowest) pressure level when using Met Office UM PP files. This is technically a bug in the generation of PP files, not the Radiance Simulator. The difference is minor and should have no significant effect on results.
  • Fixed calculation of the last (highest) pressure level when using Met Office UM PP files. This has no significant effect on results.
  • Fixed reading of Met Office UM fieldsfiles for numbers of grid points that are not multiples of 1024, which is the default ‘sector’ size for such files. This affects limited area models, and does not affect UM global model datasets currently in use.
  • Fixed problem interpolating for data points on the global model grid that lie between the 0 degree meridian and the first model grid point where previously the wrong grid cell was selected.
  • Fixed problem reading in 32-bit fieldsfiles where a temporary storage array was of insufficient size to hold a surface wind field which has more elements than there are grid points. This would previously have caused the run to fail.

Version 1.1 Release Note (selected changes)

Radiance simulator v1.1 was released in October 2015. The following lists contain details of the changes made between versions 1.0 and 1.1.

Enhancements:

  • RTTOV coefficient files in HDF5 format can now be used instead of the ASCII ‘.dat’ files.
    This is essential for simulating very new instruments as coefficient files will be generated only in HDF5 format in future.
  • Met Office Fieldsfiles containing 32-bit values (standard files contain 64-bit values) can now be used.
    This can help save disk space. There is no discernible difference in accuracy when using these files.
  • Simulations involving very large numbers of profiles and channels are automatically split into batches to reduce the memory overhead, avoiding the situation on some systems whereby the virtual memory limit was exceeded for heavily loaded runs.
  • netCDF error handling has been improved so that all errors originating from writing to netCDF files are reported explicitly. This is mainly of use in testing and should not affect users in most cases.
  • Interpolation of other input fields added (surface height and trace gas concentrations but note that trace gases are not used in simulations).
  • Option to change default zenith angle for model grid runs.

Bug Fixes:

  • Fixed assignment of half-level pressures in RTTOV profiles for models where the number of half-levels is greater than the number of levels.
  • This caused runtime failures and therefore does not affect the results of completed runs.
  • Fixed output of half-level pressures (applies only when write_profiles option is switched on).
    This was disabled in the previous version.
  • Fixed incorrect netCDF storage data type for some floating point fields.
    Fields produced with version 1.0 can still be used however.

  • Fixed problem with uninitialised pointers causing runs from Met Office Fieldsfiles to fail on some systems. This does not affect results of completed runs.

  • Fixed problem where the number of half-level pressure values was not initialised correctly before pressure level calculation for Met Office Fieldsfiles. This caused runtime failures on some systems but did not affect results of completed runs.
  • Changed fieldsfile input to read and discard unwanted fields instead of using file positioning, in order
    to avoid a compiler bug with the pgfortran compiler whereby the file was positioned incorrectly on some READ statements.
  • Fixed assignment of cloud fields in RTTOV profiles for IR scattering simulations when the input profiles have a negative z-axis.