Skip Navigation Links 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center

About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2-ncep_norm

wgrib2: -ave, -fcst_ave


The -ave and -fcst_ave options are very similar; they both make temporal averages. The -fcst_ave option uses the verification time and the -ave option uses the reference time for the temporal average.

You would use -fcst_ave to temporally average a single forecast run. For example, you have a 3 week forecast with output every 6 hours. You could use -fcst_ave to find the forecast for the second week.

You would use -ave to temporally average the results of different analyses. Suppose you have analyses every 6 hours and you want to find the analysis for the month.

The input grib file has to be processed in a special order. Don't worry, a grib file can be ordered very easily with the sort command. wgrib2 reads the data sequentially and when ever it encounters a new variable/level/chemical-type, it starts the averaging process. The length of the averaging depends on how many records it finds to average. For example, to make a daily average, a file has to be in the following order.

U500 2000-01-02 00Z             start ave
U500 2000-01-02 06Z
U500 2000-01-02 12Z
U500 2000-01-02 18Z             end ave
V500 2000-01-02 00Z             start ave
V500 2000-01-02 06Z
V500 2000-01-02 12Z
V500 2000-01-02 18Z             end ave
Z500 2000-01-02 00Z             start ave
Z500 2000-01-02 06Z
Z500 2000-01-02 12Z
Z500 2000-01-02 18Z             end ave
To make a daily average of the above file, you need to specify the output file and the time interval between samples. The time units are the same as used by GrADS (hr, dy, mo, yr).
$ wgrib2 input.grb -ave 6hr out.grb
If the file is not sorted, you can use the unix sort by,
$ wgrib2 input.grb | sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | \
   wgrib2 -i input.grb -set_grib_type c3 -ave 6hr output.grb
If you want to make daily means from 4x daily monthly files and assuming that more than one variable/level is in the monthly file.
$ wgrib2 input.grb |  sed 's/\(:d=........\)/\1:/' | \
  sort -t: -k3,3 -k5,5 -k6,6 -k7,7 -k4,4 | \
  wgrib2 input.grb -i -set_grib_type c3 -ave 6hr daily.ave.grb

Using -fcst_ave is like using -ave except you use the verification time instead of the reference time. To make an inventory that use the verification time instead of the reference time, you type,

$ wgrib2 input.grb -vt -var -lev -misc 
The sed command will be alterered very slightly when making the sort (:d=) -> (:st=).

Fast Averaging

Suppose we have a month of analyses at 3 hour intervals and want to make a monthly mean for Nov. 2014. Using the above approach, the steps would be

1.  cat narr.201411????.grb2 >tmp.grb2
2.  wgrib2 tmp.grb2 |  \
3.     sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | \
4.     wgrib2 tmp.grb -i -set_grib_type c3 -ave 3hr narr.201411

The first line creates a file with all the data.
The second line make an inventory.
The third line sorts the inventory in the order for -ave to process.
The fourth line makes the average by processing data in the order
  determined by the inventory created by line 3.

The above approach processes one average at a time and requires a minimal amout of memory. However, if you count the I/O operations, you find that there are 4 I/O operations for every field as well as the writes of the monthly means. In addition, the read (line 4) is random access.

HPC file systems are very fast for large files that are read sequentially. On the other hand, HPC file systems are horrible for small random access reads like in the previous example. Making monthly means by averaging 3 hourly NARR data was taking about three quarters of an hour on a multi-million dollar machine. The problem was that the file system was optimized for large sequential reads rather than small random-access reads. The following shows another approach.

1.  cat narr.201411????.grb2 | \
2.     wgrib2 - \
3.        -if ":HGT:200 mb:" -ave 3hr narr.201411 \
4.        -if ":UGRD:200 mb:" -ave 3hr narr.201411 \
5.        -if ":VGRD:200 mb:" -ave 3hr narr.201411 \
6.        -if ":TMP:200 mb:" -ave 3hr narr.201411 

The first line copies the data in chronological order and
   writes it to the pipe.
The second line has wgrib2 read the grib data from the pipe.
The third line selects the Z200 fields and runs the averaging
  option on it.  We are assuming the narr.* fields only have
  one Z200 field and narr.201411???? puts the data into
  chronological order.
Lines 4-6 apply the averaging option to other fields.

The above approach computes the mean of Z200, U200, V200 and T200 data at the same time with the use of more memory. The I/O consists of asequential read of all the files and the writes of the monthly means. The above script only creates the mean of Z200, U200, V200 and T200 but you could write a very long command line and compute the mean of all the fields in the file. Here are the guts of a bash script,, which creates and runs the command line.

1.  wgrib2 $1 -match_inv | cut -f4- -d: | sed -e 's/:n=.*//' >$tmp
2.  cmd="cat $* | $wgrib2 - -set_grib_type c3 "
3.  while read line
4.  do
5.    cmd="$cmd -if '$line' -ave $dt $out "
6.  done <$tmp
7.  eval $cmd

1. $1 is the first file to average.
   Line 1 creates a file with the field names (minus date codes)
2. cmd is the command line that is being built
3. loop over all the lines in file $tmp
5. generate the "-if/-ave" for the cmd line
6. bash syntax to have the while loop read from $tmp
7. run the command line

Making the NARR monthly means using the above approach uses large sequential reads which is optimal for the HPC file system. The run time went from 3/4 of an hour to maybe a minute.

Limitations by wgrib2 version

There is a limit in the maximum number of -if/-ave clauses. Wgrib2 v2.0.1 can process 1000 regular expressions and accept 5000 words on the command line. Since each -if/-ave clause takes 5 words on the command line and you need to include the name of the input file, you get a limit of 999 -if/-ave clauses.

Wgrib2 v2.0.2 is optimized for processing large numbers of if/-ave clauses. All regular expressions are evaluated in parallel. In addition the limits in number of regular expressions and words are doubled. Use the "wgrib2 -config" command to see the current limitations.


-ave (time interval)  (output grib file)
-fcst_ave (time interval)  (output grib file)

   only works with PDT=4.0, 4.1 and 4.8
    support for PDT 4.2 and 4.12 by -ave added 7.2016

See also:

NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5830 University Research Court
College Park, Maryland 20740
Climate Prediction Center Web Team
Page last modified: Aug 5, 2016
Disclaimer Privacy Policy