wgrib2: -ave, -fcst_ave
and -fcst_ave options are very similar;
they both make temporal averages.
The -fcst_ave option uses the verification time
and the -ave option uses the reference time
for the temporal average.
You would use -fcst_ave to temporally average
a single forecast run. For example, you have a 3 week forecast with
output every 6 hours. You could use -fcst_ave
to find the forecast for the second week.
You would use -ave to temporally average
the results of different analyses. Suppose you have
analyses every 6 hours and you want to find the analysis for the month.
The input grib file has to be processed in a special order. Don't worry,
a grib file can be ordered very easily with the sort command. wgrib2 reads the data
sequentially and when ever it encounters a new variable/level/chemical-type,
it starts the averaging process. The length of the averaging depends on
how many records it finds to average. For example, to make a daily
average, a file has to be in the following order.
U500 2000-01-02 00Z start ave
U500 2000-01-02 06Z
U500 2000-01-02 12Z
U500 2000-01-02 18Z end ave
V500 2000-01-02 00Z start ave
V500 2000-01-02 06Z
V500 2000-01-02 12Z
V500 2000-01-02 18Z end ave
Z500 2000-01-02 00Z start ave
Z500 2000-01-02 06Z
Z500 2000-01-02 12Z
Z500 2000-01-02 18Z end ave
To make a daily average of the above file, you need to specify the
output file and the time interval between samples. The time
units are the same as used by GrADS (hr, dy, mo, yr).
$ wgrib2 input.grb -ave 6hr out.grb
If the file is not sorted, you can use the unix sort by,
$ wgrib2 input.grb | sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | \
wgrib2 -i input.grb -set_grib_type c3 -ave 6hr output.grb
If you want to make daily means from 4x daily monthly files
and assuming that more than one variable/level is in the monthly file.
$ wgrib2 input.grb | sed 's/\(:d=........\)/\1:/' | \
sort -t: -k3,3 -k5,5 -k6,6 -k7,7 -k4,4 | \
wgrib2 input.grb -i -set_grib_type c3 -ave 6hr daily.ave.grb
Using -fcst_ave is like using
-ave except you use the verification
time instead of the reference time. To make an inventory that
use the verification time instead of the reference time, you type,
$ wgrib2 input.grb -vt -var -lev -misc
The sed command will be alterered very slightly when making the
sort (:d=) -> (:st=).
Suppose we have a month of analyses at 3 hour intervals and want
to make a monthly mean for Nov. 2014. Using the above approach, the steps
1. cat narr.201411????.grb2 >tmp.grb2
2. wgrib2 tmp.grb2 | \
3. sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | \
4. wgrib2 tmp.grb -i -set_grib_type c3 -ave 3hr narr.201411
The first line creates a file with all the data.
The second line make an inventory.
The third line sorts the inventory in the order for -ave to process.
The fourth line makes the average by processing data in the order
determined by the inventory created by line 3.
The above approach processes one average at a time and requires a
minimal amout of memory. However, if you count the I/O operations,
you find that there are 4 I/O operations for every field as well as
the writes of the monthly means. In addition, the read (line 4) is
HPC file systems are very fast for large files that are read
sequentially. On the other hand, HPC file systems are horrible for
small random access reads like in the previous example. Making
monthly means by averaging 3 hourly NARR data was taking about
three quarters of an hour on a multi-million dollar machine.
The problem was that the file system was optimized for large
sequential reads rather than small random-access reads.
The following shows another approach.
1. cat narr.201411????.grb2 | \
2. wgrib2 - \
3. -if ":HGT:200 mb:" -ave 3hr narr.201411 \
4. -if ":UGRD:200 mb:" -ave 3hr narr.201411 \
5. -if ":VGRD:200 mb:" -ave 3hr narr.201411 \
6. -if ":TMP:200 mb:" -ave 3hr narr.201411
The first line copies the data in chronological order and
writes it to the pipe.
The second line has wgrib2 read the grib data from the pipe.
The third line selects the Z200 fields and runs the averaging
option on it. We are assuming the narr.* fields only have
one Z200 field and narr.201411???? puts the data into
Lines 4-6 apply the averaging option to other fields.
The above approach computes the mean of Z200, U200, V200 and T200 data
at the same time with the use of more memory.
The I/O consists of asequential read of all the files and the
writes of the monthly means. The above script only creates
the mean of Z200, U200, V200 and T200 but you could write a
very long command line and compute the mean of all the fields in
the file. Here are the guts of a
bash script, fast_grib2_mean.sh, which creates and runs the command line.
1. wgrib2 $1 -match_inv | cut -f4- -d: | sed -e 's/:n=.*//' >$tmp
2. cmd="cat $* | $wgrib2 - -set_grib_type c3 "
3. while read line
5. cmd="$cmd -if '$line' -ave $dt $out "
6. done <$tmp
7. eval $cmd
1. $1 is the first file to average.
Line 1 creates a file with the field names (minus date codes)
2. cmd is the command line that is being built
3. loop over all the lines in file $tmp
5. generate the "-if/-ave" for the cmd line
6. bash syntax to have the while loop read from $tmp
7. run the command line
Making the NARR monthly means using the above approach uses large
sequential reads which is optimal for the HPC file system. The run
time went from 3/4 of an hour to maybe a minute.
Limitations by wgrib2 version
There is a limit in the maximum number of -if/-ave clauses.
Wgrib2 v2.0.1 can process 1000 regular expressions and accept
5000 words on the command line. Since each -if/-ave clause takes 5
words on the command line and you need to include the name of the
input file, you get a limit of 999 -if/-ave clauses.
Wgrib2 v2.0.2 is optimized for processing large numbers of if/-ave clauses.
All regular expressions are evaluated in parallel. In addition the
limits in number of regular expressions and words are doubled. Use
the "wgrib2 -config" command to see the current limitations.
-ave (time interval) (output grib file)
-fcst_ave (time interval) (output grib file)
only works with PDT=4.0, 4.1 and 4.8
support for PDT 4.2 and 4.12 by -ave added 7.2016