1. Evaluation of Sterling, Virginia WSR-88D (KLWX) Rainfall Estimates Post-Calibration
During the period February 14-17, 2000 a team from the WSR-88D Operational Support
Facility (OSF) visited the Sterling, Virginia WSR-88D radar site (KLWX) to perform newly
developed and enhanced calibration procedures to improve the radar measurements of reflectivity
and resulting derived products such as rainfall and VIL. This effort was motivated in part by the recent poor performance of the KLWX rainfall products relative to rain gauges for the Hurricane Floyd heavy rain event of 14-17 September 1999 where significant radar rainfall underestimation was typical (NWS 2000). This section describes the quantitative validation effort performed by the Hydrologic Research Laboratory (HRL) in cooperation with the OSF Applications Branch (Tim O'Bannon), Sterling Weather Forecast Office (SOO Steve Zubrick), and Eastern Region Scientific Services Division (Julie Gaddy) to compare radar rainfall estimates from the WSR-88D Precipitation Processing System (PPS) rainfall algorithm with corresponding rain gauge data to evaluate the integrity of the reflectivity measurements after the new calibration procedures were performed at LWX.
1.2 Calibration Results
Details of the enhanced calibration procedures can be obtained from the OSF Engineering
Branch. Briefly, the new procedures aim to calibrate the Internal Noise Source of the radar
receiver, something which has previously been difficult to do, which then permits accurate
calibration of the antenna gain using solar calibration procedures. Power measurement errors can
also be corrected. Beta testing of the new procedures was performed at four WSR-88D radars in
1999, and in all cases the adjustments derived from the procedures resulted in an increase in the
derived rainfall estimates (i.e., a negative reflectivity bias (dB) existed). Comparisons with a
limited rain gauge dataset demonstrated improvements relative to the gauge measurements after
the procedures were run.
Prior to the February OSF visit to Sterling, HRL's experience in casual monitoring of
LWX radar rainfall estimates relative to rain gauges and quantitative runs of the Stages II and III Precipitation Processing algorithms (which compute real-time gauge-radar mean field biases) for the Mid-Atlantic River Forecast Center region at HRL indicated that the Sterling radar was
underestimating by a factor of about two compared to rain gauges. Such a G/R bias of 2.0
implies a reflectivity measurement that is too low by 4.2 dB if using the convective Z-R relation
(Z=300 R1.4) and 3.6 dB if using the tropical relation (Z=250 R1.2) for example. During the Sterling calibration in mid-February, the OSF determined a net reflectivity calibration correction of +4.3 dB was needed. This is in approximate agreement with our experience of radar rainfall underestimation prior to that date.
1.3 Gauge-Radar Analysis Procedures
Storm-total rainfall estimates from the WSR-88D PPS algorithm were compared with
corresponding operational IFLOWS (Integrated Flood Observing and Warning System) rain
gauge measurements for all rainfall events affecting the Maryland, northern Virginia, eastern
West Virginia, and southern Pennsylvania regions, i.e., those areas within 230 km of KLWX.
Table 1 lists the rain events analyzed in this study.
Table 1. List of the ending dates of the 25 analyzed rain events. The dominant type of rainfall (stratiform vs. convective) is also listed as well as the particular Z-R relation being used by the PPS algorithm for that event.
3/17/2000 stratiform Z=130 R2.0
3/22 stratiform Z=130 R2.0
3/26 convective Z=300 R1.4
3/28 stratiform Z=300 R1.4
4/4 stratiform Z=200 R1.6
4/9 stratiform Z=200 R1.6
4/19 stratiform Z=200 R1.6
4/23 convective Z=300 R1.4
4/26 stratiform Z=300 R1.4
5/6 convective Z=300 R1.4
5/11 convective Z=300 R1.4
5/14 convective Z=300 R1.4
5/21 stratiform Z=300 R1.4
5/23 stratiform Z=300 R1.4
5/24 stratiform Z=300 R1.4
5/25 convective Z=300 R1.4
5/28 stratiform Z=300 R1.4
5/29 stratiform Z=300 R1.4
6/3 convective Z=300 R1.4
6/7 stratiform Z=300 R1.4
6/14 convective Z=300 R1.4
6/18 convective Z=300 R1.4
6/20 convective Z=300 R1.4
6/22 convective Z=300 R1.4
6/26 convective Z=300 R1.4
A. Radar Rainfall Estimates
Storm-total rainfall estimates from the PPS algorithm running in real-time on the
Warning Decision Support System (WDSS) computer at the Sterling forecast office were used in
this study. Permission to FTP into the WDSS computer via the Internet was provided by Steve
Zubrick, and an automated Unix shell script was set up on HRL computers to transfer the PPS's
HYPROD.DAT and PPS.INPUT files once per hour at the top of every hour around the clock.
The first file contains the digital storm-total rainfall array used by the PPS to produce the PUP's graphic Storm-Total Precipitation (STP) product. This digital array is a polar-gridded rainfall product with spatial resolution of 1 deg in azimuth by 2 km in range out to 230 km. The data resolution is in hundredths of inches of rainfall converted from the internal resolution within the PPS algorithm of 0.1 mm (see Fulton et. al., 1998 for details).
The second file, PPS.INPUT, containing all of the PPS adaptable parameters, was also
FTP'ed and archived so that the parameter settings in use on the WDSS were documented and
saved for each event. Arguably the most important adaptable parameters for the PPS are the two
Z-R coefficients (multiplicative factor, A, and exponent, b, in Z=A Rb) and the maximum rain rate threshold (hail cap threshold) above which rain rates, or equivalently reflectivity, are capped. The Sterling WFO varied the Z-R coefficients between three OSF-defined relations for the rain events (see Table 1): 1) Convective (Z=300 R1.4), 2) Marshall-Palmer Stratiform (Z=200 R1.6), and 3) Eastern U.S. Winter Stratiform (Z=130 R2.0). The maximum rain rate threshold remained unchanged for all events at a fixed value of 103.8 mm hr-1 which corresponds to 53 dBZ if the convective Z-R relationship is used.
The decision was made to access the PPS's HYPROD.DAT file as run on the semi-operational WDSS computer instead of the operational Concurrent RPG because FTP access is
not available on this closed system. To do these gauge-radar analyses using RPG rainfall
products would have required the manual dumping of the WSR-88D HYPROD.DAT files onto
SCSI tapes and shipment to HRL for analysis at the end of each rain event, something which is
neither practical nor efficient. An alternative option of accessing via FTP the HYPROD.DAT
file from the existing Open RPG running in a beta-test operational mode in the Silver Spring
Metro Center at NWS Headquarters using a real-time base data input feed from KLWX was
investigated as well, however attempts to decode the ORPG's HYPROD.DAT file using existing
software at HRL was not successful. Therefore the use of the PPS output files from the Sterling
WDSS was deemed the most practical and easiest approach. The WDSS is running the Build 9
version of the PPS which is not the most current version; however, changes to the software in the
current Build 10 were not deemed to impact this quantitative analysis.
"Storm-total" rainfall estimates were defined as the rainfall estimates from the time when
the PPS first started accumulating rainfall until just prior to when the internal rainfall arrays are re-initialized back to zeros an hour after the end of the event. This time period is recorded in the header data within the HYPROD.DAT file and is determined by the WSR-88D Precipitation Detection Function (PDF). Two of the three relevant PDF parameters for the "Significant Precipitation" category (Precipitation Rate Threshold, and Nominal Clutter area) are included in the PPS.INPUT file (the third relevant parameter, Precipitation Area Threshold, is not listed in the PPS.INPUT file and is assumed here to default to 0 km2 in the WDSS implementation of the PDF/PPS). The setting of these parameters determines when rainfall is first detected and thus when the PPS starts to accumulate rainfall from an initial zero-valued array. For all rainfall events, these parameters were set to reasonable values of -2.0 dBR and 50 km2. With these settings, the PPS starts accumulating rainfall when reflectivity echo (or alternatively rain rate assuming some given Z-R relationship) in any of the four lowest elevation angles exceeds -2.0 dBR (22.0 dBZ assuming the convective Z-R relation) when averaged over an area of 50 km2. The rainfall event ends (i.e., the rainfall arrays are reset) when an hour without reflectivity echo of this area and intensity passes within the radar scanning domain.
The delineation between stratiform and convective events (see Table 1) was based on
examination of the storm-total rainfall images. Stratiform events were dominated typically by
bright band enhancement (as evidenced by a broad circular band of enhanced rainfall) and strong
range degradation of the radar estimates at the far ranges due to overshooting of the shallow rain and/or the typical decrease in reflectivity with altitude. Convective events showed no such
obvious signatures in the rainfall images. The height of the melting level from observed
sounding data also correlated with the nature of the rainfall; low melting level heights were
typical with stratiform events, and vice-versa. These delineations were consistent with personal
observations of the weather for each of the rain events as they occurred.
Rain gauge data were obtained in near-real-time from all IFLOWS gauges within 230 km
range of KLWX. As many as 238 gauges reported for any given event, with typically around 190
gauges reporting. Many of these gauges were located in the mountainous western half of the
radar scanning domain, and none were located in the azimuth sector from about 120 to 220
degrees. IFLOWS gauges are tipping bucket gauges with 1 mm (0.04 inches) resolution. The
individual tips are accumulated into 15-minute accumulations by the IFLOWS data processing
software at each of several collection sites.
Other rain gauge data such as from ASOS or cooperative observers were not utilized due
to the large amount of manual work needed to access and decode/reformat the data for input into
the analysis routines. The IFLOWS data was easily accessible in digital form using automated
data transfer procedures.
Originally when this evaluation project was started, the near-real-time IFLOWS gauge
data was downloaded as ASCII listings from the IFLOWS web site (http://www.afws.net/ and,
for example for Maryland, at the end of each rain
event, but a fair amount of manual intervention was necessary to make the listings useable by the
analysis programs. In order to lessen this workload, I contacted the IFLOWS data manager, John
Bollinger, in Kentucky, and he graciously agreed to provide the 15-minute IFLOWS gauge data
in digital form suitable for FTP access on a daily basis for all gauges in Maryland, Virginia, West Virginia, and Pennsylvania. A decoding and processing program (prepiflows15.f) was written to decode, sum over the rain event, and format the gauge data for input into the gauge-radar analysis program.
In addition, I obtained the IFLOWS Master Listing file containing metadata such as the
gauge IDs, latitude/longitude locations, and altitude from Stan Campbell with the IFLOWS
program. Another program (azraniflows.f) was written to compute the range and azimuth of each
gauge relative to the LWX radar and extract all gauges that fall within 230 km of the radar.
A third program (gagradscat.f) is the main gage-radar analysis program. It decodes the
HYPROD.DAT file to extract the storm-total rainfall array, and then the single polar grid bin of
rainfall overlying each gauge was paired with the corresponding gauge measurement knowing the
latitude and longitude (and therefore range/azimuth) of each gauge as determined from the
azraniflows.f program. This information was then read into MATLAB, a commercial data
visualization program, and plotted using a program script. The MATLAB script performs an
additional quality-control step to remove gauge-radar pairs for which the gauge measured no rain
and the radar measured rain. This is to remove gauge measurements that are likely bad due to a
clogged funnel or other mechanical gauge problems. No attempt is made to search neighboring
polar radar bins surrounding the bin overlying the gauge in order to find a "better" match with the gauge. Because a single polar grid bin is paired with the gage measurement regardless of radar range, one can expect increased gauge-radar scatter for the gauges at far range compared to close-in gauges due to the much larger beam width at far ranges compared with the fixed, relatively-tiny gauge orifice size. This is the long-standing sampling issue between gauge and radar. No other manual gauge quality control was peformed.
C. Atmospheric Sounding Data
After analyzing a few rain events in March, it became obvious that there was a need to
isolate gauge-radar pairs for which the radar beam was near or above the melting level in the
mixed phase or ice region from those in which the beam was sensing liquid drops. Often the rain
gauges at the farthest ranges were reporting precipitation while the radar estimated little to no
precipitation due to overshooting and/or beam widening effects (Fulton et al., 1998). This often
caused an alignment of points along the gauge rainfall axis (with radar rainfall values of 0.0 mm)in gauge-radar scatter plots. Additionally bright band contamination (overestimation) was
common particularly in the cool events with low freezing levels which caused gauge-radar points
to appear near the other extreme in the plots. The bright band contamination and range
degradation effects of the radar estimates always showed up clearly in the storm-total radar
rainfall images, and this information was used to identify whether the event was predominantly
stratiform or convective (see Table 1). Because inclusion of such gauge-radar pairs would
significantly bias the results, an additional quality control procedure was implemented using
atmospheric sounding data.
In order to condition the gauge-radar analyses based on whether the radar beam
above a gauge was sensing liquid or otherwise, the observed atmospheric sounding
data for the Dulles International Airport (IAD) sounding site, located coincident
with the KLWX radar, was obtained for all rain events from the NOAA Forecast Systems
Laboratory (FSL) web site (http://www.fsl.noaa.gov/fsl/docs/data/fsl-data.html
The height of the 0 deg C level was determined and averaged from each observed
sounding between the time the rain event started and ended based on the PPS start
and end times recorded in the WSR-88D's HYPROD.DAT file. Since some rain events
lasted several days, the actual freezing level height measured at the fixed 00
or 12 UTC times may have varied from the mean value over that period. In addition
to assuming a fixed, mean freezing level height over time, an assumption is made
that the freezing level height is constant in space over the 230 range of the
radar. Although both of these assumptions are clearly not realistic beyond first
order, attempting to account for spatial and temporal changes in the freezing
level height in these gauge-radar comparisons would have added significantly to
the complexity of the analysis programs and probably not changed the conclusions
The height of the radar beam was computed using standard equations found in Doviak
and Zrnic (1993) assuming fixed beam elevation angles of 0.5, 1.5, 2.4 and 3.4 degrees and
standard atmosphere beam refraction. The static terrain-based hybrid scan sector file for the
KLWX radar was decoded and used to determine what elevation angle was used by the PPS
algorithm to compute rainfall for each 1 deg x 2 km polar grid bin. Knowing the elevation and
azimuth angles and range, the height above mean sea level for each polar grid bin was computed
within the gagradscat.f program and compared with the observed mean height of the melting
level from the sounding data. Gauge-radar pairs were flagged according to whether the TOP of
the radar beam above the gauge (obtained by adding 0.5 degrees to the elevation angles given
above) was above or below this melting level. Pairs identified as lying below the melting level
were termed "QC'ed" gauge-radar pairs implying they were quality controlled based on the
melting level height where the radar was likely sensing liquid rain drops.
Several simple statistics were computed for the gauge-radar pairs. The mean gauge-radar
ratio (G/R) is defined as:
where the summations are over all storm-total gauge-radar pairs for a given rainfall event. The
correlation coefficient between gauge and radar rainfall estimates was also computed. The pairs
were either all aggregated together regardless of freezing level height or conditioned on the
freezing level height above the gauge relative to the top of the beam overlying that gauge that
provided the reflectivity data to compute the rainfall estimates. Also, the gauge-radar pairs were stratified into three equal-width range intervals (0-77 km; 77-153 km; 153-230 km) in order to evaluate the contribution of well-known range degradation effects on the radar estimates.
1.4 Overall Results For All Events
All gauge-radar storm-total rainfall pairs for 23 of the 25 rain events spanning about three
months were combined together so that general conclusions could be drawn(1). Individual events will be presented in the Section V. Scatter plots of gauge vs. radar estimates are shown in Figure 1. These plots only include the QC'ed gauge-radar pairs, i.e., the ones for which the top of the radar beam above the gauge was below the 0 deg C level. These gauges are necessarily the ones at the innermost ranges of the radar scanning domain. Figure 1a combines all events together while Figs. 1b and 1c are plots for just the convective and stratiform events, respectively. The large scatter in these plots is disappointing but not unexpected given the variety of error sources associated with estimating rainfall from reflectivity measurements as well as errors in gauge measurement of rainfall. The mean gauge-radar ratio G/R is 0.82, or a radar overestimation by 22% on average. However, any given location for a given rainfall event can be either significantly over- or underestimated as illustrated in the figure. Keep in mind that these results depend critically upon which particular Z-R relation was used in generating the rainfall estimates (this varies depending on the event; see Table 1). Use of other relations would result in different statistics. For all convective events, the Convective Z-R relation (Z=300 R1.4) was used
in the PPS. However, the Eastern U.S. Winter Stratiform (Z=130 R2.0), the Marshall-Palmer Stratiform (Z=200 R1.6), as well as the Convective Z-R relations were all used for at least one of the events defined as stratiform. Evaluation of the sensitivity to the choice of Z-R relation is being pursued for a sample of these events, but results are not yet available.
Figure 1b presents results for the 11 convective events. It is clear that radar
overestimation is common. The mean G/R ratio of 0.58 translates to radar overestimation
relative to the gauges by a factor of 1.7 (=0.58-1). As an illustration, this bias would translate to a reflectivity measurement bias of +3.3 dB, i.e., a radar that is too "hot" by 3.3 dB, assuming use of the Convective Z-R relation, for example, and assuming that all of this mean gauge-radar bias can be attributed solely to reflectivity calibration error.
The stratiform events are included in the scatter plot of Fig. 1c. There are more widely
scattered pairs (with a correspondingly lower correlation coefficient) compared to the convective
events in Fig. 1b. The G/R ratio is 1.10 implying radar estimates that are underestimated by a
factor of 0.9 on average compared to the rain gauges. Note that there is clear dependence of
radar performance (with respect to rainfall estimates) on the type of rainfall, convective vs.
stratiform. These results show that the LWX radar tends to overestimate for convective events
and underestimate for stratiform events. These results could change, however, if different Z-R
relations were used. Since the "right" Z-R relation is not known a priori for each event, it is not possible to easily separate out the influence of reflectivity calibration errors from errors
associated with use of an inappropriate Z-R relation. The Convective Z-R relation may actually
perform better than one of the stratiform relations for a stratiform event, and vice-versa.
Figure 1a. Gauge-radar scatter plot of storm-total rainfall for 11 convective and 12 stratiform events. Only QC'ed pairs are included here.
Figure 1b. Gauge-radar scatter plot of storm-total rainfall for 11 convective events. Only
QC'ed pairs are included here.
Figure 1c. Gauge-radar scatter plot of storm-total rainfall for 12 stratiform events. Only QC'ed pairs are included here.
1.5 Individual Event Results
The results for each of the 25 rainfall events is presented in this section with figures and
tables specific to each case. Table 2 summarizes various information for each event such as the
gauge-radar ratio, rain type, number of gauge-radar pairs, etc. Figs. 2a-y show the plots for each event.
Table 2. List of rainfall events along with the dominant rain type (S stratiform or C convective), mean freezing level (FZL) height, number of gauge-radar pairs, G/R ratio and correlation coefficient for the three cases of a) all pairs, b) QC'ed pairs, and c) all pairs separated into three range intervals (inner, middle, outer). Asterisked freezing level heights are conservative (low) estimates when sounding data was unavailable.
||No. QC pairs
Figure 2 (to follow over the coming pages). Storm-total radar rainfall (mm), contoured IFLOWS gauge rainfall (mm), gauge-radar scatter plot for all gauges and segregated by range, QC'ed gauge-radar scatter plot, location of IFLOWS gauges relative to LWX radar (bold dots indicate QC'ed gauges), and range dependence of gauge and radar rainfall derived from available
gauge-radar pairs stratified into 23 km-wide range bands. The start and end dates and times of
the event is listed at the top of the third and fourth panels.
Figure 2a. March 13, 2000
Figure 2b. March 22, 2000
Figure 2c. March 26, 2000.
Figure 2d. March 28, 2000
Figure 2e. April 4, 2000
Figure 2f. April 9, 2000. No QC'ed image is generated since no QC'ed pairs existed.
Figure 2g. April 19, 2000.
Figure 2h. April 23, 2000.
Figure 2i. April 26, 2000.
Figure 2j. May 6, 2000.
Figure 2k. May 11, 2000.
Figure 2l. May 14, 2000.
Figure 2m. May 21, 2000.
Figure 2n. May 23, 2000.
Figure 2o. May 24, 2000.
Figure 2p. May 25, 2000.
Figure 2q. May 28, 2000.
Figure 2r. May 29, 2000.
Figure 2s. June 3, 2000.
Figure 2t. June 7, 2000.
Figure 2u. June 14, 2000.
Figure 2v. June 18, 2000.
Figure 2w. June 20, 2000.
Figure 2x. June 22, 2000.
Figure 2y. June 26, 2000.
Since the enhanced "super-calibration" procedures have been performed at the Sterling,
Virginia (KLWX) WSR-88D radar in mid-February, 25 rainfall events in the Maryland, Virginia,
West Virginia and Pennsylvania region have been examined with the intention to compare the
rainfall estimates from the Precipitation Processing System (PPS) rainfall algorithm with
corresponding operational IFLOWS rain gauge measurements. Analysis procedures were
developed and applied to the individual cases. Additionally, all of the cases were pooled together to develop long-term statistics as well as statistics segregated by storm type (stratiform and convective).
Based on comparisons over all of the storm event periods (which ranged from about 12
hours up to several days in length) for all events over the 3-month period from March to June
2000, the mean gauge-radar ratio (G/R) is 0.82, implying a radar overestimate by 22% on
average. This contrasts to our experience prior to calibration in which the LWX radar
underestimated by roughly a factor of 2. This value is computed based only on gauge-radar pairs
for which the top of the particular radar beam above each gauge as used in the PPS lies below the mean 0 deg C level during the rain event period as obtained from Dulles International Airport (IAD) soundings which are colocated with the KLWX radar at the Sterling Weather Forecast Office. Therefore these pairs are ones in which the radar is likely sensing liquid raindrops, thereby removing major known problems in radar estimation of rainfall when the beam is sensing mixed phase hydrometeors in the bright band or sensing hydrometeors at high altitudes that are not necessarily representative of the conditions near ground level. This was important as many of the events were in the early spring when the melting level was relatively low and the rainfall systems were shallow in depth. Individual event G/R ratios varied widely from this mean value (see Table 2).
When the cases were segregated into convective and stratiform events, the G/R ratios that
resulted varied enough to conclude that there are noticeable differences in the integrity of the
PPS rainfall estimates depending on the storm type. The mean stratiform G/R ratio, based on 12
events, is 1.10, implying that storm-total radar estimates are 91% of the gauge amounts on
average (i.e., radar underestimation). The mean convective G/R ratio, based on 11 events, is
0.58, implying a non-trivial radar overestimate by 72% on average.
It should be kept in mind, however, that these results are critically dependent on the
choice by the forecast office of a particular Z-R relation to be used for each event. Three
different Z-R relations were used in the 25 events, the "convective" Z-R (Z=300 R1.4), the Marshall-Palmer stratiform Z-R (Z=200 R1.6), and the new "Eastern U.S. Winter Stratiform" Z-R (Z=130 R2.0). Had different relations been used, these results would be different. Since it is not possible to know what the "right" Z-R relation should be for any event, it is difficult to determine that observed gauge-radar biases should be blamed only on reflectivity calibration errors or the use of an inappropriate Z-R relation. Both uncertainties contribute to radar rainfall estimation errors. If one was absolutely certain that the new calibration procedures produced accurate calibration, then the observed gauge-radar biases presented in this study could mean that the Z-R relations available for use by the forecast offices are not the best. Alternatively, if one was sure of the proper Z-R relation for a given event and they observed the biases presented here, then they could probably say with confidence that the calibration of the radar was in error. Analyses of the sensitivity to the choice of Z-R relation is on-going. Analyses of the radar rainfall estimates compared to the gauges is also being pursued by examining Archive II data for a sample of events and changing the reflectivity calibration offset with a fixed Z-R relation. One can then perform relative comparisons by varying the Z-R relations to determine if the radar estimates are better relative to the gauges.
Doviak, R. and D. Zrnic, 1993: Doppler Radar and Weather Observations. Academic Press, Inc.
United States, 562 pp.
Fulton, R., J. Breidenbach, D.-J. Seo, D. Miller, and T. O'Bannon, 1998: The WSR-88D Rainfall
Algorithm. Wea. Forecasting, 13, 377-395.
National Weather Service, 2000: Hurricane Floyd Floods of September 1999. Service
Assessment. Available from NWS Office of Hydrology, 1325 East-West Highway,
Silver Spring, MD 20910. 28 pp.
1. Two rain events were excluded: 26 April and 22 June. The first event had severe bright band contamination, and the second event had problems with widespread anomalous propagation contamination of the rainfall estimates. Both cases produced gauge-radar pairs that were obvious outliers, and therefore these cases were excluded from further analyses.