Skip Navigation Links www.nws.noaa.gov 
NOAA logo - Click to go to the NOAA home page National Weather Service   NWS logo - Click to go to the NWS home page
Climate Prediction Center
 
 

 
About Us
   Our Mission
   Who We Are

Contact Us
   CPC Information
   CPC Web Team

 
HOME > Monitoring_and_Data > Oceanic and Atmospheric Data > Reanalysis: Atmospheric Data > wgrib2
 

wgrib2: wgrib for GRIB-2

read and write grib2 files

Introduction

wgrib2 is not simple upgrade of wgrib for grib-2 files. wgrib can slice and dice grib-1 files. wgrib2 is more like four drawers of kitchen utensils as well as the microwave and blender. This kitchen miracle was done by a more developer friendly design and the contributions of many people. Some functions include,

  • inventory and read grib-2 files (grib-1 is not supported)
  • manipulate (reorder, extract) grib-2 records (messages)
  • convert to ieee, text, binary, comma separated values (CSV)
  • convert to netcdf-3, netcdf-4 (optional package)
  • export to a mysql database (optional package)
  • geolocation of grid points
  • interpolate to new grid (-new_grid, optional package)
  • change grid-relative to earth relative winds (-new_grid, optional package)
  • writing grib-2
  • parallel processing by using threads (OpenMP)
  • support for flow-based (multiprocessor) programming

The development of wgrib2 has been influenced by the need be a module within other programs. Consequently wgrib2 can read and write from pipes. These following programs use wgrib2.

  • g2ctl a control maker for GraADS
  • atl_g2ctl, alt_gmp alternative versions of g2ctl/gribmap for for GraADS
  • grib-filter/g2subset cgi-bin programs for Nomads (NOAA OperationalModel Archive & Distribution System)
  • g2grb.gs enables GrADS to write grib2 files
  • rNOMADS R interface for NOAA weather data
  • grb1to2.pl grib1 to grib2 converter
  • writefortran routine to write grib2 (testing)

Grib1 versus Grib2

GRIB2 has arrived and has begun replacing the the older standard GRIB (GRIB1). GRIB2 fixes the limitations in GRIB1, adds more metadata and adds compression. The new standard is almost a complete superset of older standard (except for a few locally defined variables were not included and the vague "product is valid for X period of time" needs to be more explicit like "product is average forecast for X period of time".)

I like grib2 better than grib1 because

  • I have tools to convert from grib1 to grib2 without loosing metadata or data
  • My tools are either the same or much better with grib2
  • I haven't needed to write C or Fortran to do simple conversions like calculate the upward flux from the net and downward fluxes
  • I haven't needed to write C or Fortran to convert from binary to grib2.
  • compression

    News

    8/2014: Make your wgrib2 executable smaller on linux by, strip wgrib2 strip removes symbols which are used in debugging from 6.3MB -> 2.2 MB (case 1) 4.7 MB -> 1.9 MB (case 2) 7/2014: wgrib2 v2.0.0 is replaced by v2.0.1 to fix errors in -set_metadata
    Problem affects, grb1to2.gs, g2grb2.gs 5/2012: OpenMP, experimental gctpc geolocation package
    6/2010: Interpolate to new grids using the IPOLATES library (optional package)

    8/2009: Recommended that you update to v1.8.1, v1.8.0 and v1.8.1 added bug fixes to core routines.
    Can now create new complex packed files, and have converted a GrADS readable file (HDF) into grib-2.

    5/2008: mysql support, thanks Niklas Sondell
    custom regional grids, write your own grib files

    3/2008: Recommended that you update to v1.7.1 as this version fixes a bug with the old-style -i option and has a much faster -grib option.

    1/2008: Updated grib tables, updated netcdf, added a grib2 output routine

    Kristian Nilssen has sent in a module to write netcdf files. Compiled cleanly and converted grib2 into netcdf. Will be released as an alpha option because the output format may change.

    Sergey Varlamov has improved the netcdf module to add multiple output times.

    2/2007: Manfred Schwarb went over the code last Christmas-New Years and sent in 31 patches. 31 patches sounds like a lot but he must have used a spell checker on the code. Anyways some of the patches were for the more unpolished code in the package which caused me to go over that code. Manfred also sent in a script that can read a grib2 table from the web documentation and create C code. Much better than typing by hand or any other script that I had in my tool box.

    9/16/2005 alpha version is released to the public.

    Some Good Stuff

    Usage

    • each option call corresponds to a subroutine call
    • type wgrib2 to see primary options
    • inventory format is specified on command line by options
    • if no "inv" option is specified, -s is used

    Joining the development effort

    • Source code: knowledge of C and some grib-2
    • Donated source code / non-trivial bug fixes must either be GNU or public domain.
    • Prefer PD bug fixes for PD code and GNU bug fixes for GNU code.
    • Bug reports are important
    • Contact wesley.ebisuzaki@noaa.gov

    Contributions by

    • Wesley Ebisuzai: many modules
    • Reinoud Bokhorst: tosubmsg, checksum
    • Jaakko Hyvätti: gribtab
    • Dusan Jovic: staggered grids, proj4 code
    • Kristian Nilssen: netcdf module
    • Karl Pfeiffer: georeference
    • Pablo Romero: unix_time
    • Manfred Schwarb: many modules
    • Arlindo da Silva: openGrADS, bbox
    • Niklas Sondell: mysql module
    • Sergey Varlamov: netcdf module improvements, georeferencing updates
    • thanks to the people who report the bugs and more who provide the fixes!

    Documentation

    The options

    Selecting Fields/Records/Message

    Select/Ignore by name/level/time/etc

    • -match process records that match a posix extended regular expression
    • -not process records not matching a regular expression
    • -match_inv the inventory used by -match and -not
    • -i reads inventory from stdin for record selection

    Select by number (better to use previous method)

    • -d dump specific record
    • -for select a range of records to process (nth message)
    • -for_n select a range of records to process (nth message/submessage)

    Selective Processing: if options

    After selecting the fields to process, you can refine the processing by the "if" options. For example, you selected the winds and temperatures. You can use the "if" to send the winds to one file and the temperatures to another file. In general it is faster to use the -match/-not options because there is less processing of the data.

    • -if if this then that
    • -not_if if not this then that
    • -fi closes -if (often an output option closes the if)

    Individual Grid Point Data

    • -ij print value at grid point (i, j)
    • -ijlat print lat, lon, value at grid point (i, j)
    • -ilat print lat, lon, value of Nth grid point
    • -lon prints the lat-lon, and value of the grid point nearest the specified lat-lon
    • -max prints the maximum value
    • -min prints the minimum value
    • -stats prints some statistics about the fields
    • -V verbose inventory (shows stats)

    Regridding, Interpolating to new grids

    Wgrib2 has the ability to convert grib files from one grid to another. The conversion uses a user-selected interpolation scheme: bilinear (default), bicubic, nearest neighbor and budget. The supported grids (7/2010) include lat-lon, gaussian, Lambert conformal and polar stereographic. This capability uses the NCEP IPOLATES library and is an optional package.

    Note: the interpolation uses scalar and vector interpolation schemes. For the vector quantities the V field must immediately follow the corresponding U field.

    Modifying grid data

    Wgrib2 allows options to modify the grid point data. The various -undefine options set the grid values to undefined. This reduces the size of the CSV and grib output. The various -import functions allow you to overwrite the data with the contents of a file. The -rpn option runs a calculator with 10 registers. Unlike an ordinary calculator that works with numbers, this calculator works with grids. These options allow you to
    • set grid points to undefined based on region
    • set grid points to undefined based on value
    • Change units, useful for making CSV files
    • Compute new fields
      • thickness
      • find max/min/ave value in a time series
      • wind speed/direction
    • Convert binary data to grib2
    • Convert GrADS compatible data data to grib2 (g2grb.gs)
    • Merge subregions; i.e., combine tiles
  • various -import options
  • -rpn Reverse Polish Notation calculator
  • -ijundefine undefine/maskout regions
  • -undefine undefine/maskout regions
  • -undefine_val undefine/maskout grid points by value
  • -grib_out writes the current field in grib format

Exporting data to other programs

  • -netcdf: write data in netcdf format
  • -mysql: export data to a mysql database
  • -mysql_speed: export data to a mysql database
  • -spread: write data for spreadsheets
  • -csv: write in column separated values, another one for spreadsheets
  • -text: data in text format
  • -bin: data native binary floating point
  • -ieee: data in big endian IEEE format
  • -ijbox: write a rectangular grid of data
  • -AAIG: arcinfo ascii grid, GIS

Specialized options used by other programs

  • -ctl_inv: print parameters needed by g2ctl
  • -ctl_ens: print parameters needed by g2ctl
  • -domain: print out the lat-lon domain that covers the grid
  • -cyclic: needed by g2ctl
  • -lev0: print level information needed by g2ctl
  • -limit: web programs that need controls of cpu usage

Seeing the details of the grib file

  • -code_table_I.J: to see contents of code table I.J
  • -flag_table_I.J: to see contents of flag table I.J
  • -get_byte show selected bytes of the grib message
  • -get_int show selected 4-byte integers of the grib message
  • -SecI: see descriptions about section I, I = 0..6,
  • -Sec_len: see length of each section
  • -0xSec I see hex dump of a section I
For a short list of options, type "wgrib -h"
For a complete list, type "wgrib -help all"
To search for an option, type "wgrib -help keyword"

Writing grib2

Writing grib2 is problem with no perfect solution. Some people want speed, some people want ease of use. Some people want it to run the code on a hundred processors and other want to run it on Windows XP. Here is a list of techniques that involve wgrib2. Here is the background for techniques that use the import function.
  1. Using the ieee packing option allows you to make some very light weight C programs that write grib2. You can even use standard Unix utilities. See -grib_ieee.
  2. The basic -import_ieee/bin/text option.
  3. Example of converting HDF to grib2
  4. using -set_metadata

Machines able to run wgrib2

  • Redhat linux: 32/64-bit with IPOLATES
  • Ubuntu: 32/64-bit with IPOLATES
  • Mandriva linux (old report)
  • AIX: use makefile, some fiddling is needed
  • Solaris, needs gnu make and gcc (old report)
  • HPUX, needs changes to makefiles
  • XP home/32, Windows 7 home premium/64: used cygwin and had to download many libraries modified makefiles use builtin-jasper library.
  • Intel-based Mac
  • The IPOLATES library has been tested with on g95, gfortran, openf95 and AIX fortran
The makefile works on Redhat and Ubuntu (with needed installed options). For other systems, expect that you will need to modify the makefile.

System dependencies: Windows and others

The wgrib2 source code is written to the POSIX C standard. No problem with windows, unix, linux, 32/64-bit integers, big/little endian. The main difference between the windows and linux versions is that windows treats binary and text files differently. Consequently you can send a grib or binary data to STDOUT in unix/linux with no problems. Under windows, the results are implementation dependent. The libraries, on the other hand, may not run all machines.

32-bit executables are limited to 2 GB files. (The fseek function is limited to signed long integers.)

The maximum number of open files is system dependent.

The netcdf conversion is done in memory, so the speed is dependent on the amount of available physical memory.

Source Code and Compling Hints

The wgrib2 code is written to the POSIX standard. The regular expression package requires the regcomp and regexec routines which are POSIX-2. (This package is optional for systems which don't include these routines.) The wgrib2 code can be compiled with 32 or 64 bit pointers and integers. However, the code has to be compiled in a like manner to all the libraries. (You can't mix 32 and 64 bit libraries and wgrib2 code.) Some packages are optional (netcdf, mysql) you shouldn't enable them if not needed.

While compiling wgrib2, you may see warnings about unknown pragmas. Pragmas are "comments" that are used when compiling a OpenMP version of the code. If you compile without OpenMP, your code will run slower and you'll see many warnings about unidentified pragmas.

Netcdf3, the default, often has problems compiling cleanly on 32-bit machines. (It fails in the testing phase.) The work-around is to compile with Netcdf4 or without Netcdf. The best solution is to move to a 64-bit system. This will allow you to process files that are larger than 2GB. All the linux systems that I have access to are 64 bits, so this problem isn't going to be resolved soon.

Precompiled code from External Sites

There are many sites with precompiled versions of wgrib2. This list is neither exhaustive nor an endorsement of the sites. I have not tested the wgrib2 executables from these sites and YMMV.

Status

The code is being developed on a Redhat linux x86_64 platform in standard (POSIX) C. The wgrib2 source code requires an integer size of 32 or more bits. Incompatibilites tend to be related to problems in compiling of the external libraries (ex. netcdf, jasper).

Change logfile

Solution #1:Unknown Template or Segmentation Fault

The wgrib2 source code includes various libraries. This makes compiling on different machines easier. Three of the libraries (g2clib, iplib and gctpc) are different from the official versions for a good reason. If you have problems with seg faults (wgrib2 versions prior to 1.9.8) try adding the option "-g2clib 0". Unfortunately this option does not emulate the bug in encoding/decoding constant fields that is present in g2clib and g2lib (1/2014). Fortuantely many of the application programers have taken steps to avoid this bug.

G2clib doesn't decode fields when it includes an unknown template. For versions of wgrib prior to 1.9.8, you can turn off g2clib by "-g2clib 0".

Solutions #2:Segmentation fault

Higher resolution files may need a bigger stack especially when the jpeg2000 (Jasper) or interolation (IPOLATES) libraries are called. In the following case, a jpeg2000 compressed grib file (9 million grid points) is read. The problem goes away when the stack is increased.

sh-3.00$ wgrib2 rtgssthr_grb_0.083_awips.grib2 -V
Segmentation fault

-sh-3.00$ ulimit -s 30000
-sh-3.00$ wgrib2 rtgssthr_grb_0.083_awips.grib2 -V
1:0:vt=2009062900:surface:anl:TMP Temperature [K]:
    ndata=9331200:undef=3246169:mean=287.003:min=271.33:max=312.8
    grid_template=0:
        lat-lon grid:(4320 x 2160) units 1e-06 input WE:NS output WE:SN res 48
        lat 89.958000 to -89.958000 by 0.083000
        lon 0.042000 to 359.958000 by 0.083000 #points=9331200
Here is an example interpolating to a .1x.1 degree grid using the IPOLATES library. The problem disappears after increasing the stack size by 2 orders of magnitude over the default value.
bash-3.2$ wgrib2 gfs.t00z.sfluxgrbf06.grib2 -match "GRD:10" -new_grid_winds earth -set_grib_type c2 -new_grid latlon 0:3600:.1 -90:1801:.1 junk.grb2
36.1:23114245:d=2011111700:UGRD:10 m above ground:6 hour fcst:
Segmentation fault

bash-3.2$  ulimit -s 4400000
bash-3.2$  wgrib2 gfs.t00z.sfluxgrbf06.grib2 -match "GRD:10" -new_grid_winds earth -set_grib_type c2 -new_grid latlon 0:3600:.1 -90:1801:.1 junk.grb2
36.1:23114245:d=2011111700:UGRD:10 m above ground:6 hour fcst:
36.2:23114245:d=2011111700:VGRD:10 m above ground:6 hour fcst:
112.1:63622055:d=2011111700:UGRD:10 m above ground:0-6 hour ave fcst:
112.2:63622055:d=2011111700:VGRD:10 m above ground:0-6 hour ave fcst:

Bugs

1/2014: Jasper library bug? Grib files that have large grids and that are compressed with jpeg-2000 can have problems. Have seen this problem in GrADS, wgrib2 and it has been reported in WCT. If you exceed the critical grid size, the program will have problems reading the jpeg-2000 compressed grib message. The critical grid size seems to vary with the program and how it was compiled. I've tried the Fedora and Ubuntu updates to the Jasper library and that didn't fix the problem. Note that critical size was larger for wgrib2 than GrADS. So it was possible to rewrite the file using a different compression which GrADS could then read. We found problems when the grid size was 6 million. The problem has been reported at 5 million for different programs. The common stack size was quickly ruled out by setting the stack size to a large number and by running the programs on a system with an unlimited stack size.

Please report them to wesley.ebisuzaki@noaa.gov. When you report bugs, try to make them reproduceable on a linux machine and include sample data.

Distribution

The source code modules for wgrib2 are either in the public domain or under the GNU licence depending on the authors of the various modules. Wgrib2 uses libraries that are in the public domain, under various GNU licences, the Image Power JPEG-2000 Public Licence, libpng licence, the zlib licence, the netcdf licence*, HDF5* licence, MySQL licence* and perhaps others. How about one licence to rule them all?

* optional package


NOAA/ National Weather Service
National Centers for Environmental Prediction
Climate Prediction Center
5830 University Research Court
College Park, Maryland 20740
Climate Prediction Center Web Team
Page last modified: Aug 18, 2011
Disclaimer Privacy Policy