|
NWP SAF
AAPP
Installation
Guide
|
Document ID: NWPSAF-MO-UD-005
Version:
7.3
Date:
01 June 2015 |
4 Testing
After having successfully installed AAPP on your workstation you may
start testing AAPP. Several different test cases are provided. You
should run the ones that are relevant to your application. The test
cases supplied with AAPP v7.1 are listed in the following table.
Other test cases may be available on the AAPP ftp server.
Title
|
File
|
Purpose
|
Scripts
|
Notes
|
NOAA-16 HRPT
|
noaa16_test.tgz |
NOAA-16 HRPT from 2001 using TBUS.
Processes raw HRPT to level 1d for AMSU-A, AMSU-B and HIRS,
and AVHRR to level 1b.
|
NOAA16_RUN.sh
NOAA16_Compare.sh
clean.sh
|
Same data as were provided with AAPP v6, but updated
reference output. No external libraries needed. |
NOAA-18 HRPT |
noaa18_test.tgz |
NOAA-18 HRPT from 2005 using TLE. Processes raw HRPT to
level 1d for AMSU-A, MHS and HIRS, and AVHRR to level 1b. |
NOAA18_RUN.sh
NOAA18_Compare.sh
clean.sh |
Same data as were provided with AAPP v6, but updated
reference output. No external libraries needed. |
ATMS and CrIS processing
|
ATMS_CrIS.tgz
|
Ingest of ATMS/CrIS Sensor Data Records in HDF5 and/or
BUFR format. Pre-processing for ATMS/CrIS (ATMS spatial
filtering, CrIS channel selection, mapping, etc.). BUFR
encode facility.
|
atms_cris_bufr.sh
atms_cris_sdr.sh
process_atms_cris_1c.sh
clean.sh
|
atms_cris_bufr.sh requires
BUFR library
atms_cris_sdr.sh requires
HDF5 library
Reference 1c files are provided in case you only want to run
process_atms_cris_1c.sh |
FY-3A MWTS and MWHS
|
FY3A.tgz |
MWTS and MWHS ingest (HDF5) and pre-processing (map ATMS
to CrIS)
|
MWTS_MWHS.sh
clean.sh
|
Requires HDF5 library
|
MAIA
|
MAIA_noaa19.tgz |
MAIA2.1 and MAIA3, using
forecast files. |
run_atovs_avhrr.sh
run_maia.sh
clean.sh
|
Requires MAIA3 data file to
be installed.
Use of the forecast files requires GRIB-API, but it will
revert to climatology if GRIB-API is not available.
|
MetOp-A AHRPT
|
metopa_20100128.tgz |
MetOp-A AHRPT, with the
ability to exercise the following:
- IASI OPS-LRS and pre-processing
- IASI PC code
- ATOVS and AVHRR processing from level 0 to level 1
- MAIA3 (using climatology)
- BUFR encode/decode
|
run_atovs_avhrr.sh
run_atovs_bufr.sh
run_iasi_OPS.sh
run_iasi_PC_bufr.sh
run_maia3.sh
clean.sh
|
run_atovs_avhrr.sh
should be run first.
run_iasi_PC_bufr.sh
requires HDF5 and BUFR libraries.
run_atovs_bufr.sh
requires BUFR library
run_iasi_OPS.sh
requires
- $PATH_OPS to point to the "perl" directory of an
OPS_LRS installation,
- $DIR_IASICONFIG to point to a set of IASI config files
(available from the AAPP ftp server)
|
MAIA4
|
MAIA4_test_1gran.tgz
|
MAIA4 and viirs_paste
|
run_maia4.sh
run_viirs_paste.sh
|
Introduced with AAPP v7.5
(June 2013)
Requires HDF5 Fortran library and GRIB_API library.
Forecast files and VIIRS SDR files are supplied. A single
granule is used here; cases with more granules can be
supplied on request.
Note that the HDF5 library tool "nagg" is an alternative to
"viirs_paste", and can run faster, especially when
concatenating large input files.
|
VIIRS to CrIS
|
viirs_to_cris.tgz
|
VIIRS to CrIS mapping, optionally including MAIA4
|
viirs_to_cris_run.sh
viirs_to_cris_withMAIA4.sh
|
Introduced with AAPP v7.10
(June 2015)
Requires HDF5 Fortran library and GRIB_API library.
VIIRS and CrIS SDR files are supplied. Forecast files are downloaded from the Internet. A single
granule of VIIRS data is supplied, but the scripts will work with more granules.
|
Each test case is contained within a zipped tar file. To unpack a
test case, copy the file to a suitable directory and type
tar -xzf name.tgz
or if your system does not support the "-z" option in tar
gunzip -c
name.tgz
tar -xf -
Before running any of the test cases, you should set the
environment variable AAPP_PREFIX to the installation directory for
AAPP v7 (i.e. the directory containing your ATOVS_ENV7 file):
export
AAPP_PREFIX=.....
Specific instructions for
running a test case are provided in the corresponding
README.txt files.
Where reference output files are provided, these were
generated at the Met Office on RedHat Enterprise Linux 6 with
ifort 12.0.4 Fortran compiler.
4.1 Test case example: NOAA data test cases
These test cases make use of the AAPP chain script AAPP_RUN_NOAA
(installed to directory AAPP/bin). (For AAPP v1 to v5 this
script was called AAPP_RUN and was in directory AAPP)
To run the NOAA-18 test case (for example), unpack the noaa18_test.tgz
file, set up environment variable AAPP_PREFIX then enter the
following commands:
cd
noaa18_test
NOAA18_RUN.sh
Check the output files NOAA18.OUT and NOAA18.ERR in
directory work to
see if it ran OK. You can compare with the reference output in
directory work_ref. If
all looks OK, you can compare the level 1c and 1d output files by
typing
NOAA18_Compare.sh
then examine the results in directory compare.
Some differences are to be expected due to compiler differences,
but the maximum differences should be only a few hundredths of a K
for AMSU and HIRS brightness temperatures, somewhat larger in
some of the AVHRR derived quantities.
Note that TBUS and TLE data for the NOAA test cases are stored
within the directory structure of the test case, not within the
main AAPP installation tree. This ensures the test case is
self-contained.
4.2 Possible Problem Areas
Some portability problems may still require manual adaptions.
Recommendations are provided in the list of known bugs and problems
in the AAPP webpage. The AAPP_7 package has been tested on Linux,
AIX and SUN.
Problems can be reported through the Feedback Form available
from the NWP SAF.
4.3 Preparation for Processing Your Own HRPT Read-Out Data
Before trying to use AAPP for processing NOAA HRPT read-out data
from your own station check first whether
i) |
your station delivers the data in the format expected by
AAPP, i.e. the 10-bit words of the down-linked HRPT data are
stored right-justified in 16-bit words.
If this is not the case you have to reformat the data. If
your data are in packed 10-bit words then you can use the
tool unpack_noaa_hrpt to
unpack the data, but you will probably need to change the
values of parameters bytes_in
and words_out in unpack_noaa_hrpt.F,
since different stations use different conventions. |
ii) |
your station position is contained in
(dest.dir.)/AAPP/src/navigation/libnavtool/stations.txt
If this is not the case insert it and copy the updated file
additionally to
(dest.dir.)/AAPP/data/navigation/
and modify the variable "STATION" in the environment
variable file ATOVS_ENV77 with the name of your
station. Note: if "STATION" is empty then AAPP_RUN_NOAA will
fail! |
To process METOP AHRPT data you must make sure that your station
produces PFS Level 0 data files, as defined by EUMETSAT. If the
reception station does not deliver files in L0 format then the user
should consider using METOPizer software to pre-process the raw
AHRPT (go to www.eumetsat.int
and navigate to Data ->
Data Delivery -> Support Software & Tools ).
The METOPizer programs tvcdu_to_ccsds
and ccsds_to_l0 are likely
to be needed. Please note the following advice from EUMETSAT:
If you are reading CADU packets then
the installation procedure has changed. The Metopizer used to
include a Reed-Solomon library licensed under the GPL which has to
be installed separately now. There is an INSTALL file inside the
package with instructions.
To process NPP and JPSS data you will need an external processing
package to generate Sensor Data Record (SDR) files suitable for
input to AAPP. Suitable packages are supplied by NASA ("IPOPP") and
by the University of Wisconsin ("CSPP"). Details are available
elsewhere.
4.4 Processing scripts
Direct
readout processing
The AAPP_RUN_NOAA chain script processes NOAA HRPT data through to
level 1d. The script offers the following options:
- Specify the instruments to be processed (from the list "AMSU-A
AMSU-B HIRS MSU AVHRR DCS ATMS CRIS")
- Specify the output grids required at level 1d (from the list
"AMSU-A AMSU-B HIRS ATMS CRIS")
- Specify an output directory to receive the product files
- Omit avh2hirs if not
required
In early versions of AAPP, it was suggested that the user should
customise the script to suit his own requirements, but now that the
functionality is provided by command-line options, this should be
unnecessary. If you do wish to customise the script, please give it
a different name, to avoid confusion.
If your HRPT reception system automatically delivers the orbit
number, time, date, etc. (e.g. in the file name) then you may pefer
to use a simplified script, as in AAPP_RUN_NOAA_simplified.
In this example we use environment variables "ATOVS" and "AVHRR" to
control which parts of AAPP are run.
To process METOP AHRPT data you may use the AAPP_RUN_METOP
script which works as follows:
- Looks for PFS level 0 files in an input directory (dump mode
assumed, i.e. one file per instrument)
- Processes data for the specified instruments (from the list
"AMSU-A MHS HIRS AVHRR IASI")
- Generates 1d output on the requested grids ("AMSU-A MHS HIRS
IASI")
- Sends output files to the specified output directory
The script allows the user to take two passes through the data - the
first to generate traditional HIRS, AMSU-A and MHS 1d files, and the
second optional pass to run OPS-LRS and generate products on the
IASI grid. For full details, please see the description in
the source.
If your station does not generate one file per instrument per pass
(e.g. it uses 3 minute granules) then you will need to concatenate
the granules, otherwise the HIRS calibration will fail. Level 0
granules can be concatenated using the Unix "cat" command.
Processing
externally supplied level 1b files
Many users also use AAPP to process global or regional ATOVS data,
which are typically supplied at level 1b or 1c. To do this you need
to call atovin and/or atovpp directly from your
script. For example, to convert ATOVS data from level 1b to level 1d
on the HIRS grid you would typically do the following:
ln
-sf amsua_1bfile aman.l1b
ln -sf amsub_1bfile ambn.l1b
ln -sf hirs_1bfile hrsn.l1b
atovin AMSU-A AMSU-B HIRS
atovpp -i "AMSU-A
AMSU-B HIRS" -g
"HIRS"
Note that if you are ordering data from NOAA CLASS, you should
untick the box that says "include archive header" (or do this in
your user preferences). Otherwise you will have to strip off the
first 512 bytes from each 1b file before running atovin (e.g. dd bs=512 skip=1).
AAPP does not currently
ingest TOVS level 1b data from CLASS (i.e. NOAA-14 and earlier).
This capability is planned for a future AAPP upgrade.
In the case of METOP data distributed in BUFR, you would need to run
aapp_decodebufr_1c and
then atovpp. But note that
global METOP data are distributed in short (typically 3 minute)
granules, and there is no guarantee that the granules for the
different instruments will be aligned in time. Therefore a utility
has been provided to concatenate level 1c files, called combine_1c,
in order to ensure that the instrument to be mapped covers a wider
time interval than the instrument to be used as a grid. It is up to
the user to write his script such that the appropriate input files
are concatenated.
If difficulties are encountered, please seek advice using the
Feedback Form available from the NWP
SAF.
Processing
ATMS and CrIS data
ATMS and CrIS data may be received as near-real-time global BUFR
files or as SDR files. You will need to run some or all of the
following AAPP tools:
- aapp_decodebufr_1c,
to convert from BUFR to AAPP-style level 1c
- atms_sdr or cris_sdr,
to convert from SDR in HDF5 format to AAPP-style level 1c
- atms_beamwidth, to
modify the effective beamwidth and noise characteristics of ATMS
(at level 1c)
- atovpp, to create a
CrIS level 1d file (ATMS + CrIS, various options) or to create
an ATMS level 1d file on the original ATMS grid
- cris_channels is for
users who need to generate a channel-selected 1c dataset,
independent of atovpp. For example, RARS stations can use it
during the conversion from SDR to channel-selected BUFR product.
Uses the channel selection in CRIS.fdf. Note that AAPP
convention is to use the standard channel numbering system,
where channel 1 is 650 cm-1 and there are 1305
channels.
- atms_thin1d, to
create a spatially thinned ATMS product (AMSU-like sampling)
- aapp_encodebufr_1c
- BUFR encode the 1c ATMS or CrIS files, or
- Create a BUFR-encoded level 1d file for ATMS or CrIS. This
option was developed for use at the Met Office (and uses some
local BUFR descriptors), but may be useful to others.
For more information on the processing, see AAPP document
NWPSAF-MO-UD-027 "Pre-processing of ATMS and CrIS".
In some cases you will need to do some preparation of the data
before running the AAPP utilities. Data often arrive with one file
per 32-second granule - which is too short for effective use of atms_beamwidth and atovpp.
Also, in some cases the granules may not arrive in time order. One
solution to this problem that has been employed at the Met Office
for ATMS/CrIS BUFR data is to aggregate files as follows:
- Use the time stamp in the file name to compute a granule
number, defined as the second of day divided by 32. Append this
to the file name.
- Also compute an aggregation number, equal to the granule
number divided by the number of granules you wish to aggregate
(e.g. 10). Append this to the file name also.
- When all 10 granules have arrived for a given aggregation
number, concatenate the individual granules (unix cat
will work for BUFR data) and proceed to the AAPP processing.
A similar approach could be used for processing HDF5 granules,
except that atms_sdr or cris_sdr would be run first,
and the *.l1c granules aggregated using combine_1c.
This problem does not arise with HDF5 data from NOAA/CLASS because
files can be delivered already aggregated into chunks of 8 minutes
(15 granules).
Processing
archived data from EUMETSAT Data Centre
AAPP can ingest the following types of data from EUMETSAT Data Centre:
- EPS level 0
- EPS level 1 (native format) for AVHRR or IASI (but not
for AMSU, MHS or HIRS)
- BUFR level 1 for AMSU, MHS, HIRS, IASI
Note that unless you specify a regional subset, then the input files
will be full-orbit. For AVHRR and IASI this results in quite large
input files. If you wish to process AVHRR files that contain more
than 33 minutes of data, the following changes are needed in AAPP:
- For level 0 processing, increase the value of mx_hrpscn in
hrptdc.h. Normally set to 12000; increase to 32767 which is the
maximum possible, since the AVHRR 1b format holds avh_h_scnlin
as a 2-byte integer. Note that a full-orbit would be 36400
scans.
- Increase the value of avh_mxscn in avhrcl.h. Also
normally set to 12000.
It is preferable to specify a geographical subset (defined by
lat/lon limits) when you order the data.
The applicable AAPP scripts are:
- AAPP_RUN_METOP, for
processing level 0. You will need to obtain appropriate orbital
elements files.
- convert_avh1b, to
convert AVHRR EPS level 1b to AAPP 1b format - but with scaled
radiances instead of the original instrument counts
- convert_iasi1c, to
convert IASI EPS level 1b to AAPP 1c format
- aapp_decodebufr_1c,
to decode BUFR files
4.5 Satellite attitude
Some users may wish to manually set the values of roll, pitch and
yaw ("attitude") for a particular satellite. The default attitude
for each satellite is set in the file ${DIR_NAVIGATION}satid.txt.
Look for the lines that say "number of orbital bulletin dependant
attitude biases". Underneath are listed the default yaw, roll and
pitch (in milliradians) for up to four types of bulletin (in the
order TBUS, Argos, 2-line, Spot). These values may be changed by the
user if required, or attitude biases inserted if there are no values
already present (i.e. older satellites). This file also specifies
misalignment parameters for individual instruments.
If you wish to change the attitude for specific orbits then you need
to create a file ${DIR_NAVIGATION}/ana/estatt_${BUL}_${SATIMG}.txt,
where ${BUL}
is "tbus", "tle" or "spm" and ${SATIMG}
is the name of the satellite, e.g. "noaa16". This is a text file
with the following values on each line
yaw
roll pitch orbit 0
where yaw, roll
and pitch are in
milliradians and "0" means the data are good. If this
file exists, it is read by the scripts avhrcl, amsuacl,
amsubcl, etc. If the orbit being processed (or the previous orbit)
matches a line in the file then the corresponding attitude values
are used; if not then the default (from satid.txt) is used.
4.6 Satellite manoeuvres
Some satellites (notably MetOp) undergo pre-planned manoeuvres in
order to maintain the correct orbit. For MetOp these may be either
"in-plane" (no change to attitude) or out-of-plane (in which the
satellite is rotated so that the station-keeping rockets are
pointing at 90 degrees to the direction of motion). For AAPP users
the in-plane manoeuvres are usually of little concern, however the
out-of-plane manoeuvres (conducted typically every 1-2 years) can
cause problems with the quality control of the orbital elements
files (TBUS, 2-line or Spot).
When an out-of-plane manoeuvre is announced it is recommended for
users to temporarily disable the quality control mechanisms by
setting an environment variable as follows:
PAR_NAVIGATION_EXTRAP_LIMIT=100
export
PAR_NAVIGATION_EXTRAP_LIMIT
This may be either inserted into your ATOVS_ENV7 file or into your
calling script. When the manoeuvre is finished, and new orbital
elements have been ingested, then you can set PAR_NAVIGATION_EXTRAP_LIMIT="".
This facility was introduced in AAPP version 6.7.
To check whether a bulletin has been successfully
ingested, look at your index file, e.g.
$DIR_NAVIGATION/tle_db/tle_M02.index. The second number in
each line should be "0" if the corresponding bulletin was OK. To
ingest a specific TLE file by hand (e.g. published in the MetOp
Admin message) then you can run tleing
by hand with the "-f" option (type tleing
-h for usage instructions), e.g.
tleing
-s M02 -f ./2011-10/tle_20111005_1130.txt
If you manually ingest a bulletin in order to correct a geolocation
error, remember to delete the old satpos file, otherwise AAPP will
continue to use that satpos file for the remainder of the day and
you will still get the error.
4.7 On the use of working directories
The ATOVS_ENV7 setup defines an environment variable $WRK (default
$HOME/tmp), which is used by several of the AAPP scripts. For
example, AAPP_RUN_NOAA and
AAPP_RUN_METOP do a "cd" to
this directory during the script execution. Other scripts assume you
are already in a suitable working directory (e.g. atovpp). You can
set up your working directory before running the script, e.g. to use
the current directory do
WRK=$PWD
&& export WRK
If you are running multiple instances of AAPP on the same machine
(e.g. processing global and local data at the same time), it
is very important that they do not attempt to use the same working
directory, otherwise you will get unpredictable results.
One way to do this would be to define temporary working directories
before running a script, e.g.
WRK=$PWD/tmp_$$_$(date
+%H%M%S_%N) && export $WRK
mkdir
-p $WRK
cd $WRK
where the temporary directory has been created accoring to the shell
process number ($$) and the system time. Remember to delete the
temporary directory when you have finished with it.
Most users are unlikely to need this level of complexity, but the
issue has been raised in the past via the NWP SAF Helpdesk,
therefore we mention it as something to be aware of when you are
designing your processing systems.
Cover Page