wgrib2: wgrib for GRIB-2
read and write grib2 files
wgrib2 is not simple upgrade of wgrib for grib2 files. wgrib
can slice and dice grib1 files. wgrib2 is more like four
drawers of kitchen utensils as well as the microwave and blender.
This kitchen miracle was done by a more developer friendly design
and the contributions of many people. Some functionality include,
- inventory and read grib2 files
- create subsets
- create regional subsets by cookie cutter or new projections
- export to ieee, text, binary and CSV
- export to netcdf and mysql
- write grib2 fields like averages, accumulations, max/min values
- convert grid-relative to earth relative winds
- convert U and V to wind speed, direction
- writing of new grib2 fields
- parallel processing by using threads (OpenMP)
- parallel processing by flow-based programming
- C callable interface and fortran interface (soon)
OpenMP is used to speed up wgrib2 by running big loops over
multiple cores. To compile OpenMP into wgrib2, you need set
- "USE_OPENMP=1" in the makefile (default)
- compile wgrib2: "make"
- set two envirnoment variabls
- export OMP_NUM_THREADS=4 (ex. for 4 threads)
- export OMP_WAIT_POLICY=PASSIVE (or ACTIVE)
In general, you set the number of threads to be ≤ to
the number of CPU cores available. Usually I set the number to 3 on
a quad-core machine. That leaves one core for unrelated work.
I usually set OMP_WAIT_POLICY=PASSIVE in order to reduce
the CPU utiliation at the expense of the run time.
Compiling: NetCDF Configuration
Wgrib2 can be configured with (1) no NetCDF support, (2) NetCDF3 support or
(3) NetCDF4 support. For a person who only wants to convert grib2 to netcdf,
then option 3 is the obvious choice. In some situations, wgrib2 may be
called a hundred times a minute (ensemble processing, nomads) with no need
for netcdf support. (NetCDF4 increases the executable size almost 3.5x).
In this case, wgrib2 should be compiled with option 1. I use option 2 because I
don't mind the 50% increase in executable size and I make netcdf files once in
a while. 32-bit machines should be configured for either no NetCDF support or NetCDF4 support
because the 32-bit compiled NetCDF3 code fails the internal testing. (The netCDF3 library
is very old.) You can see the NetCDF configuration of a compiled wgrib2 by by typing "wgrib2 -config".
- Option 1: No NetCDF support
- fast compile
- small executable (3.9 MB as of 2014)
- compile works on 32+ bit machines
- makefile configuration: USE_NETCDF3=0, USE_NETCDF4=0
- Option 2: NetCDF3 support (default in makefile)
- fast compile
- modest increase in executable size (5.7 MB vs 3.9 MB as of 2014)
- compile fails on 32-bit machines
- makefile configuration: USE_NETCDF3=1, USE_NETCDF4=0
- Option 3: NetCDF4 support
- slower compile
- 3.5x increase in executable size (13.5 MB vs 3.9 MB as of 2014)
- compile works on 32+ bit machines
- requires fortran90 compiler
- makefile configuration: USE_NETCDF3=0, USE_NETCDF4=1
- prompts for downloading netcdf4/hdf5 libraries
Execution: Segment Faults
Wgrib2 is used in NCEP operations and by many daily processing jobs
at CPC. There are only two known causes of seg faults in the current released version of wgrib2.
The library used to handle jpeg2000 stores the arrays on the stack
and if the stack is too small, wgrib2 will seg fault. The solution is
to set the stack size to unlimited (requires root) or to set the stack
to some larger value.
ulimit -a : this will show the size of the stack in bash/linux
ulimit -s (new stack size in KB) : set the stack to some larger value
Note: you can only set the stack size once per window.
the stack size can be made larger than necessary,
the only drawback is it uses physical memory.
may require some experimentation
The other cause of seg faults is an unpatched g2clib. Wgrib2 source code
comes with a patched version of g2clib. Many distributions will use the
official version of g2clib which can causes seg faults. This problem has been
addressed in wgrib2 v1.9.8 which requires an option in order to enable
g2clib. If you think that any seg fault is a security risk, then you
should compile wgrib2 without g2clib support. You can check for this
add "-g2clib 0" to the command line, if fixes the seg fault, you have found the problem
ex. wgrib2 GRIBFILE -bin gribfile.bin -g2clib 0
The best fix is to get a new version of wgrib2 (1.9.8+). You can use
the old version of wgrib2 by adding the "-g2clib 0" option but that can
give different results when you are decoding a constant field with
a decimal scaling factor.
8/2014: Make your wgrib2 executable smaller on linux by, strip wgrib2
strip removes symbols which are used in debugging
from 6.3MB -> 2.2 MB (case 1) 4.7 MB -> 1.9 MB (case 2)
7/2014: wgrib2 v2.0.0 is replaced by v2.0.1 to fix errors in -set_metadata
Problem affects, grb1to2.gs, g2grb2.gs
5/2012: OpenMP, experimental gctpc geolocation package
6/2010: Interpolate to new grids using the IPOLATES library (optional package)
8/2009: Recommended that you update to v1.8.1, v1.8.0 and v1.8.1 added bug fixes to core routines.
Can now create new complex packed files, and have converted a GrADS readable file (HDF) into grib-2.
5/2008: mysql support, thanks Niklas Sondell
custom regional grids, write your own grib files
3/2008: Recommended that you update to v1.7.1 as this version fixes a bug with the old-style -i option and has a much faster -grib option.
1/2008: Updated grib tables, updated netcdf, added a grib2 output routine
Kristian Nilssen has sent in a module to write netcdf files. Compiled cleanly and
converted grib2 into netcdf. Will be released as an alpha option because the output
format may change.
Sergey Varlamov has improved the netcdf module to add multiple output times.
2/2007: Manfred Schwarb went over the code last Christmas-New Years and sent in 31 patches.
31 patches sounds like a lot but he must have used a spell checker on the code. Anyways
some of the patches were for the more unpolished code in the package which caused me to
go over that code. Manfred also sent in a script that can read a grib2 table from
the web documentation and create C code. Much better than typing by hand or any
other script that I had in my tool box.
9/16/2005 alpha version is released to the public.
Some Good Stuff
- each option call corresponds to a subroutine call
- type wgrib2 to see primary options
- inventory format is specified on command line by options
- if no "inv" option is specified, -s is used
Joining the development effort
- Source code: knowledge of C and some grib-2
- Donated source code / non-trivial bug fixes must either be GNU or public domain.
- Prefer PD bug fixes for PD code and GNU bug fixes for GNU code.
- Bug reports are important
- Contact firstname.lastname@example.org
- Wesley Ebisuzai: many modules
- Reinoud Bokhorst: tosubmsg, checksum
- Jaakko Hyvätti: gribtab
- Dusan Jovic: staggered grids, proj4 code
- Kristian Nilssen: netcdf module
- Karl Pfeiffer: georeference
- Pablo Romero: unix_time
- Manfred Schwarb: many modules
- Arlindo da Silva: openGrADS, bbox
- Niklas Sondell: mysql module
- Sergey Varlamov: netcdf module improvements, georeferencing updates
- thanks to the people who report the bugs and more who provide the fixes!
Select/Ignore by name/level/time/etc
- -match process records that match a posix extended regular expression
- -not process records not matching a regular expression
- -match_inv the inventory used by -match and -not
- -i reads inventory from stdin for record selection
Select by number (better to use previous method)
- -d dump specific record
- -for select a range of records to process (nth message)
- -for_n select a range of records to process (nth message/submessage)
Selective Processing: if options
After selecting the fields to process, you can refine the processing by
the "if" options. For example, you selected the winds and temperatures.
You can use the "if" to send the winds to one file and the temperatures to another file.
In general it is faster to use the -match/-not options because there is less processing
of the data.
- -if if this then that
- -not_if if not this then that
- -fi closes -if (often an output option closes the if)
Individual Grid Point Data
- -ij print value at grid point (i, j)
- -ijlat print lat, lon, value at grid point (i, j)
- -ilat print lat, lon, value of Nth grid point
- -lon prints the lat-lon, and value of the grid point nearest the specified lat-lon
- -max prints the maximum value
- -min prints the minimum value
- -stats prints some statistics about the fields
- -V verbose inventory (shows stats)
Regridding, Interpolating to new grids
Wgrib2 has the ability to convert grib files from one grid
to another. The conversion uses a user-selected interpolation
scheme: bilinear (default), bicubic, nearest neighbor and budget.
The supported grids (7/2010) include lat-lon, gaussian, Lambert conformal
and polar stereographic. This capability uses the NCEP IPOLATES library
and is an optional package.
Note: the interpolation uses scalar and vector interpolation
schemes. For the vector quantities the V field must immediately
follow the corresponding U field.
Modifying grid data
Wgrib2 allows options to modify the grid point data. The various -undefine options
set the grid values to undefined. This reduces the size of the CSV and grib output.
The various -import functions allow you to overwrite the data with the contents of a file.
The -rpn option runs a calculator with 10 registers. Unlike an ordinary calculator that
works with numbers, this calculator works with grids. These options allow you to
various -import options
-rpn Reverse Polish Notation calculator
-ijundefine undefine/maskout regions
-undefine undefine/maskout regions
-undefine_val undefine/maskout grid points by value
-grib_out writes the current field in grib format
- set grid points to undefined based on region
- set grid points to undefined based on value
- Change units, useful for making CSV files
- Compute new fields
- find max/min/ave value in a time series
- wind speed/direction
- Convert binary data to grib2
- Convert GrADS compatible data data to grib2 (g2grb.gs)
- Merge subregions; i.e., combine tiles
Exporting data to other programs
For a short list of options, type "wgrib -h"
- -netcdf: write data in netcdf format
- -mysql: export data to a mysql database
- -mysql_speed: export data to a mysql database
- -spread: write data for spreadsheets
- -csv: write in column separated values, another one for spreadsheets
- -text: data in text format
- -bin: data native binary floating point
- -ieee: data in big endian IEEE format
- -ijbox: write a rectangular grid of data
- -AAIG: arcinfo ascii grid, GIS
For a complete list, type "wgrib -help all"
To search for an option, type "wgrib -help keyword"
Writing grib2 is problem with no perfect solution. Some people
want speed, some people want ease of use. Some people want it
to run the code on a hundred processors and other want to run
it on Windows XP. Here is a list of techniques that involve
wgrib2. Here is the background
for techniques that use the import function.
- Using the ieee packing option allows you to make
some very light weight C programs that write grib2. You
can even use standard Unix utilities. See
- The basic -import_ieee/bin/text option.
Example of converting HDF to grib2
- using -set_metadata
Machines able to run wgrib2
The makefile works on Redhat and Ubuntu (with needed installed options).
For other systems, you may have to modify the makefile.
- Redhat linux: 64-bit with IPOLATES
- Redhat 6 Enterprise: primary development system
- Redhat 5 Enterprise: remove compile option "-Werror=format-security"
from the makefile. This is a mandatory compile option for some
distributions and is just a good idea. However, the gcc in RH5 doesn't support this option.
- Redhat linux: 32-bit with IPOLATES, not tested recently, use netcdf4
instead of netcdf3
- Ubuntu: 64-bit with IPOLATES
- Ubuntu: 32-bit with IPOLATES, not tested recently
- Mandriva linux (old report)
- AIX: use makefile, some fiddling with libraries is necessary, not tested recently
- Solaris, needs gnu make and gcc (old report)
- HPUX, needs changes to makefiles (old report)
- Cygwin with XP home/32, Windows 7 home premium/64: Use netcdf4 instead of netcdf3
- Intel-based Mac
- The IPOLATES library has been tested with on g95, gfortran, openf95 and AIX fortran
System dependencies: 32 vs 64 bit, Windows vs Linux/UNIX
The wgrib2 source code is written to be portable; there are no
issues with big vs little endian or the size of the integer as
long as it meets the minimum C requirement (32-bits). For
windows, the cygwin's gcc supports the POSIX standard and allows
the standard source code to be compiled. (I have received emails
saying that wgrib2 has been compile on Watcom and MingW compilers.)
The main difference between the 32 and the 64-bit versions is that
the 32-bit version is limited to 2 GB files. On most versions of
wgrib2, the number of grid points is limited to 2 G. The limit
would be higher for whose default integer uas more than 32-bits.
Note: wgrib2 can handle larger than 2GB files if they read through
The main difference
between the windows and linux versions is that windows treats
binary and text files differently. Consequently you can send
a grib or binary data to STDOUT in unix/linux with no problems.
Under windows, the results are implementation dependent.
Dpending on the windows compiler, POSIX1 and POSIX2
functions may not be supported or may function differently.
- The maximum number of open files is system dependent.
- The maximum number of arguments to wgrib2 may be limited by the system.
- Stack size may be limited by the system which can limit the size
of grids that can be decoded.
- The netcdf conversion is done in memory, so the speed is dependent
on the free memory available.
Source Code and Compling Hints
The wgrib2 source code is written to the POSIX standard. Features requiring
POSIX2, such as regular expressions, can usually be turned off
in the makefile. The wgrib2 code can be compiled with 32 or 64 bit pointers and integers.
However, the code has to be compiled in a like manner to all the libraries.
Some packages are optional (netcdf, mysql) and enabling these options can really
increase the executable size.
While compiling wgrib2, you may see warnings about unknown pragmas. Pragmas
are "comments" that are used when compiling an OpenMP version of the code. The
default is to compile with OpenMP turned on. However, in some environments,
turning off OpenMP is preferred.
Netcdf3, the default, often has problems compiling cleanly on 32-bit machines.
(It fails in the testing phase.) The work-around is to compile with Netcdf4
or without Netcdf. The best solution is to move to a 64-bit system. This
will allow you to process files that are larger than 2GB. All the linux
systems that I have access to are 64 bits, so this problem isn't going to
be resolved soon.
Precompiled code from External Sites
There are many sites with precompiled versions of wgrib2. This list
is neither exhaustive nor an endorsement of the sites. I have
not tested the wgrib2 executables from these sites and YMMV.
Other programs that call wgrib2 (incomplete list)
The code is being developed on a Redhat linux x86_64 platform in standard (POSIX) C.
The wgrib2 source code requires an integer size of 32 or more bits. Incompatibilites
tend to be related to problems in compiling of the external libraries (ex. netcdf,
8/2015: wgrib2 v2.0.0-v2.0.2 -ieee does not work with OpenMP. Fixed 2.0.3
8/2015: complex packing: problem when all data is undefined. Fixed 2.0.3
5/2015: callable wgrib2 and fatal errors. The error handling code can
cause a seg fault on subsequent calls to wgrib2. Problem has been fixed
in the development version.
1/2014: Jasper library bug? Grib files that have large grids and that are
compressed with jpeg-2000 can have problems. Have seen this problem in GrADS,
wgrib2 and it has been reported in WCT. If you exceed the critical grid
size, the program will have problems reading the jpeg-2000 compressed grib
message. The critical grid size seems to vary with the program and how it
was compiled. I've tried the Fedora and Ubuntu updates to the Jasper library
and that didn't fix the problem. Note that critical size was larger for wgrib2
than GrADS. So it was possible to rewrite the file using a different compression
which GrADS can read. We found problems when the grid size was 6 million.
The problem has been reported at 5 million for different programs. The common
stack size was quickly ruled out by setting the stack size to a large number and
by running the programs on a system with an unlimited stack size.
Please report them to email@example.com. When you report bugs,
try to make them reproduceable on a linux machine and include sample data.
The source code modules for wgrib2 are either in the public domain or under the GNU
licence depending on the authors of the various modules. Wgrib2 uses libraries that
are in the public domain, under various GNU licences, the Image Power JPEG-2000 Public Licence,
libpng licence, the zlib licence, the netcdf licence*, HDF5* licence, MySQL licence* and perhaps others. How
about one licence to rule them all?
* optional package