wgrib2: wgrib for GRIB-2
Utility to read and write grib2 files
wgrib2 is not simple upgrade of wgrib for grib2 files. wgrib
can slice and dice grib1 files. wgrib2 is more like four
drawers of kitchen utensils as well as the microwave and blender.
This kitchen miracle was done by a more developer friendly design
and the contributions of many people. Some functionality include,
- inventory and read grib2 files
- create subsets
- create regional subsets by cookie cutter or projections
- export to ieee, text, binary, CSV, netcdf and mysql
- write of new grib2 fields
- parallel processing by using threads (OpenMP)
- parallel processing by flow-based programming
- fortran and C interface
OpenMP is used to speed up wgrib2 by running loops over
multiple cores. With gcc and icc, OpenMP is on by default.
In general, you set the number of threads to be less or equal
to the number of CPU cores available. Usually I set the number to 3 on
a quad-core machine. That leaves one core for unrelated work. So
I added "export OMP_NUM_THREADS=3" to my .bashrc.
I also added "export OMP_WAIT_POLICY=PASSIVE" to my .bashrc
in order to reduce CPU utilization at the expense of run time.
Compiling: NetCDF Configuration
Wgrib2 can be configured with (1) no NetCDF support, (2) NetCDF3 support or
(3) NetCDF4 support. For a person who only wants to convert grib2 to netcdf,
option 3 is the obvious choice. In some situations, wgrib2 may be
called a hundred times a minute (heavy-duty processing, nomads) with no need
for netcdf support. (NetCDF4 increases the executable size almost 3.5x).
In this case, wgrib2 should be compiled with option 1. I use option 2 because I
don't mind the 50% increase in executable size, and I have a 64-bit machine.
32-bit machines should be configured for either no NetCDF support or NetCDF4 support
because the NetCDF3 code fails during internal testing. (The netCDF3 library is old
and no longer supported.)
You can see the NetCDF configuration of a compiled wgrib2 by by typing "wgrib2 -config".
- Option 1: No NetCDF support
- fast compile
- small executable (3.9 MB as of 2014)
- compile works on 32+ bit machines
- makefile configuration: USE_NETCDF3=0, USE_NETCDF4=0
- Option 2: NetCDF3 support (default in makefile)
- fast compile
- modest increase in executable size (5.7 MB vs 3.9 MB as of 2014)
- compile may fail on 32-bit machines
- makefile configuration: USE_NETCDF3=1, USE_NETCDF4=0
- Option 3: NetCDF4 support
- slower compile
- 3.5x increase in executable size (13.5 MB vs 3.9 MB as of 2014)
- compile works on 32+ bit machines
- requires fortran90 compiler
- makefile configuration: USE_NETCDF3=0, USE_NETCDF4=1
- prompts for downloading netcdf4/hdf5 libraries
Execution: Segment Faults
Wgrib2 is used in NCEP operations and by many daily processing jobs
at CPC. Seg-faults are pretty uncommon to wgrib2. The common problems
are in related to experimental/alpha features and the following 3 problems.
The jpeg2000 library (Jasper) could seg-fault with larger grids. This
problem was fixed with an Ubuntu/Debian update to Jasper (fall 2015) which was
incorporated into wgrib2 v2.0.3. If your copy of wgrib2 is using another version of the
Jasper library, then your copy of wgrib2 may be vunerable to this
Wgrib2 can decode grib using g2clib. Using the g2clib that was
not included with the wgrib2 sorc code is could cause a seg-fault. Wgrib2 had patches
to g2clib that were not fully incorporated until g2clib v1.4.0. In addition,
g2clib can be compiled either with a 32 or 64-bit interface. The default
makefile assumes that a 32-bit interface is used on 32-bit processors and a
64-bit interface is used 64-bit processors. Third-parties often followed
NCEP's lead and used a 64-bit interface on 32-bit processors and
a 32-bit interface on 64-bit processors.
Finallyl, the third problem is a compiler bug and only affects
This problem appears in gcc 2.4.7's handling of unsigned int
looping variables when OpenMP is enabled. The problem is
fixed by gcc 2.5.0.
- wgrib2 v2.0.4
- wgrib2 was compiled with OpenMP support
- number of threads is greater than one
- creating simple packed grib messages
- all the grid points are undefined
- compiled with older versions of gcc such as 2.4.7
There a potential for seg-faults by reading using specially crafted png or jpeg2000
compressed grib files. For example, a few days after the release of wgrib2 v2.0.3, the pnglib
was upgraded to fix a seg-fault when reading a specially crafted png file.
2/2017: wgrib2api a fortran interface for reading/writing grib2 based on wgrib2
2/2017: Starting with wgrib2 2.0.6, I test on
I am no longer using gcc/g95, opencc/openf95, xlc/xlf, and windows 7/cygwin.
- gcc/gfortran: Ubuntu, RedHat
- clang/gfortran: Ubuntu
- icc/ifort: Redhat, Suse
- Windows 10: Cygwin64
10/2016: wgrib2 2.0.5, in Cygwin and some other systems, the AEC library
may not pass the internal checks. Upgrade to libaec-1.0.0, https://gitlab.dkrz.de/k202009/libaec
7/2016: wgrib2 2.0.5 adds CCSDS/AEC compression, fixes to run-length encoding, simple packing, ieee output
2/2016: wgrib2 2.0.4 works with grib sections up to 4GB and up to 2G grid points. Fixes to the
Jasper library (large files), complex packing (decoding strange fields, encoding small files)
8/2014: Make your wgrib2 executable smaller on linux by, strip wgrib2
strip removes symbols which are used in debugging
from 6.3MB -> 2.2 MB (case 1) 4.7 MB -> 1.9 MB (case 2)
9/16/2005 alpha version is released to the public.
Some Good Stuff
- each option call corresponds to a subroutine call
- type wgrib2 to see primary options
- inventory format is specified on command line by options
- if no "inv" option is specified, -s is used
Joining the development effort
- Source code: knowledge of C and some grib-2
- Changes to existing source code should use the same licence as the original code
- New code (files) must either be GNU or public domain.
- Bug reports are important
- Contact email@example.com
- Wesley Ebisuzai: many modules
- Reinoud Bokhorst: tosubmsg, checksum
- DWD (Gregor Schee, Daniel Lee and others): AEC compression
- Jaakko Hyvätti: gribtab
- John Howard: callable wgrib2
- Dusan Jovic: staggered grids, proj4 code
- Kristian Nilssen: netcdf module
- Karl Pfeiffer: georeferencing
- Pablo Romero: unix_time
- Manfred Schwarb: many modules
- Arlindo da Silva: openGrADS, bbox
- Niklas Sondell: mysql module
- Sam Trahan: satellite tables
- Sergey Varlamov: netcdf module improvements, georeferencing updates
- thanks to the people who report the bugs and more who provide the fixes!
Select/Ignore by name/level/time/etc
- -match process records that match a posix extended regular expression
- -not process records not matching a regular expression
- -match_inv the inventory used by -match and -not
- -i reads inventory from stdin for record selection
Select by number (better to use previous method)
- -d dump specific record
- -for select a range of records to process (nth message)
- -for_n select a range of records to process (nth message/submessage)
Selective Processing: if options
After selecting the fields to process, you can refine the processing by
the "if" options. For example, you selected the winds and temperatures.
You can use the "if" to send the winds to one file and the temperatures to another file.
In general it is faster to use the -match/-not options because there is less processing
of the data.
- -if if this then that
- -not_if if not this then that
- -fi closes -if (often an output option closes the if)
Individual Grid Point Data
- -ij print value at grid point (i, j)
- -ijlat print lat, lon, value at grid point (i, j)
- -ilat print lat, lon, value of Nth grid point
- -lon prints the lat-lon, and value of the grid point nearest the specified lat-lon
- -max prints the maximum value
- -min prints the minimum value
- -stats prints some statistics about the fields
- -V verbose inventory (shows stats)
Regridding, Interpolating to new grids
Wgrib2 has the ability to convert grib files from one grid
to another. The conversion uses a user-selected interpolation
scheme: bilinear (default), bicubic, nearest neighbor and budget.
The supported grids (7/2010) include lat-lon, gaussian, Lambert conformal
and polar stereographic. This capability uses the NCEP IPOLATES library
and is an optional package.
Note: the interpolation uses scalar and vector interpolation
schemes. For the vector quantities the V field must immediately
follow the corresponding U field.
Exporting data to other programs
For a short list of options, type "wgrib -h"
- -netcdf: write data in netcdf format
- -mysql: export data to a mysql database
- -mysql_speed: export data to a mysql database
- -spread: write data for spreadsheets
- -csv: write in column separated values, another one for spreadsheets
- -text: data in text format
- -bin: data native binary floating point
- -ieee: data in big endian IEEE format
- -ijbox: write a rectangular grid of data
- -AAIG: arcinfo ascii grid, GIS
For a complete list, type "wgrib -help all"
To search for an option, type "wgrib -help keyword"
Writing grib2 is problem with no perfect solution. Some people
want speed, some people want ease of use. Some people want it
to run the code on a hundred processors and other want to run
it on Windows XP. Here is a list of techniques that involve
wgrib2. Here is the background
for techniques that use the import function.
- The basic -import_ieee/bin/text option.
- using -set_metadata
Example of converting HDF to grib2
Machines able to run wgrib2
The makefile works on Redhat and Ubuntu (with needed installed options).
For other systems, you may have to modify the makefile. The makefile
requires gnu make is a common version of make.
- linux: 64-bit with IPOLATES
- Redhat 6 Enterprise: primary development system
- Redhat 5 Enterprise: remove compile option "-Werror=format-security"
from the makefile. This is a mandatory compile option for some
distributions and is just a good idea. However, the gcc in RH5 doesn't support this option.
- Ubuntu: 64-bit with IPOLATES
- Ubuntu: clang and gfortran
- Redhat linux: 32-bit with IPOLATES, not tested recently, use netcdf4
instead of netcdf3
- Ubuntu: 32-bit with IPOLATES, not tested recently
- Mandriva linux (old report)
- AIX: use makefile, some fiddling with libraries is necessary, not tested recently
- Solaris, needs gnu make and gcc (old report)
- HPUX, needs changes to makefiles (old report)
- Windows: using Cygwin system produces 32-bit binaries. using netcdf4
- Windows: using Cygwin system produces 64-bit binaries
- Windows: compiled MingW (not recent), Watcom C, icc/ifort
- Windows/linux subsystem (ubuntu): compiled with gcc/gfortran
- Intel-based Mac
- The IPOLATES library has been tested with on AIX fortran, g95, gfortran, ifort, and openf95
System dependencies: 32 vs 64 bit, Windows vs Linux/UNIX
The wgrib2 source code is written to be portable; there are no
issues with big vs little endian or the size of the integer as
long as it meets the minimum C requirement (32-bits). For
windows, the cygwin's gcc supports the POSIX standard and allows
the standard source code to be compiled. It is possible to
compile the code with many of the POSIX extentions turned off
was done when compiling with the Watcom compiler.
Wgrib2 can be compiled with 32-bit systems. With a 32-bit version,
memory limitations will be important and the message size needs to
be much smaller than 2GB. File size is limited to 2GB unless
you restrict yourself to sequential reads and writes. (No -i
or -submsg options, no using the byte location.)
Wgrib2 can be compiled for 64-bit systems. On most systems (types,
not boxes), files can be larger than 2GB and the grib message can
be as large as 2GB. (Many options support up to 4GB.)
For most Windows 64-bit C compilers, "long int" is defined as 32 bits.
Consequently the ISO C random-access routine is limited to 2GB files.
The 64-bit cygwin gcc works the way it should and is recommended
for windows compilations.
The main difference
between the windows and linux versions is that windows treats
binary and text files differently. Consequently you can send
a grib or binary data to STDOUT in unix/linux with no problem.
Under windows, the results are implementation dependent.
In addition, POSIX1 and POSIX2 functions may not be supported
or may function differently. Consequently certain options
may work differently.
- The maximum number of open files is system dependent.
- The maximum number of arguments to wgrib2 may be limited wgrib2 or by the system.
- The netcdf conversion is done in memory, so the speed is dependent
on the free memory available.
- the accuracy of the trig functions depends on the compiler
- wgrib2 is usually compiled in fast mode rather than ieee mode.
Source Code and Compling Hints
The wgrib2 source code is written to the POSIX standard. Features requiring
POSIX2, such as regular expressions, can usually be turned off
in the makefile. The wgrib2 code can be compiled with 32 or 64 bit pointers and integers.
However, the code has to be compiled in a like manner for all the libraries.
Some packages are optional (netcdf, mysql) and enabling these options can really
increase the executable size.
While compiling wgrib2, you may see warnings about unknown pragmas. Pragmas
are "comments" that are used when compiling an OpenMP version of the code. The
default is to compile with OpenMP turned on. However, in some environments,
turning off OpenMP is preferred.
Netcdf3, the default, often has problems compiling cleanly on 32-bit machines.
(It fails in the testing phase.) The work-around is to compile with Netcdf4
or without Netcdf. The best solution is to move to a 64-bit system. This
will allow you to process files that are larger than 2GB. All the linux
systems that I have access to are 64 bits, so this problem isn't going to
be resolved soon.
Precompiled code from External Sites
There are many sites with precompiled versions of wgrib2. This list
is neither exhaustive nor an endorsement of the sites. I have
not tested the wgrib2 executables from these sites and YMMV.
Other programs that use wgrib2 (incomplete list)
a control maker for GraADS
- atl_g2ctl, alt_gmp
alternative versions of g2ctl/gribmap for for
- grib-filter/g2subset cgi-bin programs for Nomads (NOAA OperationalModel Archive & Distribution System)
- g2grb.gs enables GrADS to write grib2 files
- rNOMADS R interface for NOAA weather data
- grb1to2.pl grib1 to grib2 converter
- wgrib2api, a fortran API for reading/writing grib2
The code is being developed on a Redhat linux x86_64 platform in standard (POSIX) C.
The wgrib2 source code requires an integer size of 32 or more bits. Incompatibilites
tend to be related to problems in compiling of the external libraries (ex. netcdf,
11/2016: wgrib2 v2.0.5 AEC library doesn't compile, upgrade library source code
8/2015: wgrib2 v2.0.0-v2.0.2 -ieee does not work with OpenMP. Fixed 2.0.3
8/2015: complex packing: problem when creating if grid values are all undefined. Fixed 2.0.3
5/2015: callable wgrib2 and fatal errors. The error handling code can
cause a seg fault on subsequent calls to wgrib2. Problem has been fixed v2.0.3
1/2014: Jasper library bug? Grib files that have large grids and that are
compressed with jpeg-2000 can have problems. Have seen this problem in GrADS,
wgrib2 and it has been reported in WCT. update: 11/2016: this problem was
fixed by using Debian/Ubuntu version of the Jasper library included with wgrib2 2.0.3.
Please report them to firstname.lastname@example.org. When you report bugs,
try to make them reproducible on a linux machine and include sample data.
The source code modules for wgrib2 are either in the public domain or under the GNU
licence depending on the authors of the various modules. Wgrib2 uses libraries that
are in the public domain, under various GNU licences, the Image Power JPEG-2000 Public Licence*,
libpng licence*, the zlib licence*, the netcdf licence*, HDF5 licence*, MySQL licence* and perhaps others. How
about one licence to rule them all?
* optional package