AVN-Based MOS Precipitation Type Guidance for the United States
by Rebecca L. Allen and Mary C. Erickson
On September 1, 2000, the National Weather Service began disseminating forecasts of precipitation type based on the application of the Model Output Statistics (MOS) (Glahn and Lowry 1972) technique to output from the Aviation run (AVN) of NCEP's Global Spectral Model (Kanamitsu 1989). Forecast equations predict the conditional probabilities of freezing, frozen, and liquid precipitation, and these probabilities in turn are used to determine a categorical forecast of precipitation type. The AVN MOS precipitation type guidance was developed in a similar way to the current NGM-based MOS precipitation type (Erickson 1995), and is meant to serve as an alternative to the NGM precipitation type system. This AVN MOS precipitation type guidance is available for approximately 1000 sites in the contiguous United States and Alaska. Forecasts are valid every 3 hours from 6 to 72 hours after both the 0000 and 1200 UTC model runs. This Technical Procedures Bulletin summarizes the development, testing, and dissemination of the AVN MOS precipitation type forecasts.
The MOS approach statistically relates observed predictand data to predictor data such as forecasts from dynamical models, surface observations, and geoclimatic information. In applying the MOS technique to precipitation type, multiple linear regression was used to develop the statistical equations.
The predictands used in this development were obtained from the hourly METAR observations of present weather. The present weather observations valid at 0000, 0300, 0600, 0900, 1200, 1500, 1800, and 2100 UTC were classified into one of four categories: freezing, frozen, or liquid precipitation, or the null category. The null category included those cases where no precipitation of any kind occurred, or cases where we were unable to determine the exact type of precipitation. These instances are also referred to as missing cases. Freezing precipitation was comprised of freezing rain, freezing drizzle, ice pellets, or any precipitation in combination with any of these three events. Frozen precipitation was defined as pure snow or snow grains. Finally, liquid precipitation included rain, drizzle, thundershowers, or any mixture of liquid precipitation with snow. Every non-missing case in the sample was characterized by three mutually exclusive binary predictands, each taking a value of 1 or 0 for freezing/no freezing, frozen/no frozen, and rain/no rain, respectively. For example, a snow event was characterized as a non-freezing/frozen/non-liquid event. A separate linear regression equation was developed for each predictand to forecast the conditional probability of that precipitation type occurring. The probability equations (and subsequent forecasts) are conditional upon precipitation occurring since only precipitation cases were used to develop the forecast equations.
The definitions of the precipitation type categories are identical to those used in the NGM-based MOS precipitation type guidance. As before, ice pellets were included in the freezing category. We have often been asked to separate the freezing rain and ice pellet ("sleet") events. At this time, we do not feel this is possible since neither event, by itself, occurs frequently enough to get an adequate sample on which to develop robust forecast equations. In addition, as we observed with the NGM model soundings, the AVN model composite soundings of temperature for freezing rain and ice pellet cases do not differ significantly enough to be able to distinguish between the two events by using the MOS technique. Given these reasons, along with the fact that some researchers have included freezing rain, drizzle, and ice pellets together when conducting studies of "freezing precipitation" (for example, Bernstein 2000), we believe that continuing to categorize ice pellets and freezing rain/drizzle as one precipitation type category is appropriate.
One difference to note between the NGM precipitation type development and this AVN development is a significant change in the observational data. There has been a large increase in the number of automated stations. These ASOS sites have limitations in their ability to report present weather. For example, ASOS sites often have trouble distinguishing between light rain and light snow, and therefore report unknown precipitation (UP). Because one cannot be sure of the exact precipitation type, cases of UP were not included in the equation development.
Predictors offered in the regression process included AVN model data, geoclimatic information, and, for some projections, surface observations. The AVN model fields included temperature, wet-bulb temperature, temperature advection, and the u- and v- wind components at levels of 500, 700, 850, 925, and 1000 mb. In a departure from previous developments, the temperature and wet-bulb temperature at 2 m above the surface were used as predictors. Thicknesses were also offered for many layers including 1000-700 mb, 1000-850 mb, 1000-925 mb, 925-850 mb, 850-700 mb, and 700-500 mb. Model fields valid at the forecast projection, as well as 3 hours before and after the forecast projection, were included as possible predictors.
Several predictors derived from model fields were made available to the regression. These included the pressure (mb) of the freezing level, as well as a predictor based on the vertical profile of wet-bulb temperature forecast by the model. This "zr" predictor identifies cases where freezing precipitation is likely to occur based on the presence of a sufficiently cold surface layer, and a warm layer aloft that will allow for the melting of the frozen precipitation. Finally, several single-station regression predictors were offered that give the probability of snow occurring at an individual station based on the forecast value of a particular model field (1000-850 mb thickness, 850 mb temperature, and 2-m temperature). As with the raw model fields, derived predictors valid at the forecast projection, as well as 3 hours before and after the forecast projection, were included as possible predictors.
Geoclimatic predictors included the sine and cosine of the day of the year, which help infer variations within the season, and the conditional monthly relative frequencies of freezing, frozen and liquid precipitation. These relative frequencies were computed by using 2 years of data, and are valid for the 12-h period centered on the forecast projection. The relative frequencies provide specific information about individual stations, some of which might have similar model forecasts, but regularly experience vastly different weather due to local effects.
Surface observations taken at 0300 and 1500 UTC (for the 0000 and 1200 UTC model runs, respectively) were also offered to the regression for the 6- through 18-h projections. Both surface temperature and the average of the surface temperature and dew point were included as predictors. In addition, the present weather reported at the aforementioned hours was included in the form of three binary predictors: freezing precipitation/no freezing precipitation, frozen/no frozen, and liquid/no liquid. These binary predictors were formed by using the same precipitation type classification scheme that was used to develop the predictands.
Predictors were offered to the regression in a continuous, point-binary, or grid-binary (Jensenius 1992) form. To compute a point-binary variable, the original gridded predictor is first interpolated to the specific station and then compared to the appropriate binary cutoff. The resulting value of the predictor is either 0 or 1, according to whether the predictor is less than, or greater than/equal to the breakpoint, respectively. Conversely, to compute a grid-binary variable, the binary cutoff is applied at the model gridpoints, and then that field of 0's and 1's is smoothed and interpolated to specific stations. The resulting variable can have any value between 0 and 1. This technique provides a smoother transition, both spatially and temporally, between the extremes of the predictor than does the point-binary approach. In this development, the model fields were offered as continuous and grid-binary variables, and the surface observations were offered as point-binaries.
The most frequently chosen predictors included the relative frequency of freezing precipitation, the zr predictor, the 2-m temperatures and wet-bulb temperatures, grid binaries of various thicknesses and temperatures, and the four single-station regression predictors. The surface observations were also chosen quite frequently in the equations for the 6- through 18-h projections.
Since freezing rain, ice pellets, and snow are relatively rare events in some parts of the country, stations were combined into geographic regions in order to develop stable forecast relationships. These forecast equations are then applied to any station within a region. This also allows forecasts to be produced for stations that were not included in the developmental sample, but that are located within a region for which an equation was developed. Figures 1 and 2 show the four regions used in the contiguous U. S. and the two regions used in Alaska, respectively. These regions were developed based on climatology and geographic similarity. Precipitation type guidance was not developed for stations in Puerto Rico, Hawaii, southern Florida, and parts of California. Consequently, MOS precipitation type guidance is not available for these sites. Also, it was not possible to develop freezing rain equations for all projections in Alaska. For these projections, the probability of freezing precipitation will be missing, and the categorical forecast will be based only on the frozen and liquid probabilities.
d. Developmental Sample
The developmental sample consisted of precipitation type observations and predictor data for 611 stations in the contiguous U.S. and 32 sites in Alaska. Although forecasts will be produced for roughly 1000 stations, only those that report present weather reliably were used to develop the equations. The final equations were derived by using 3 years of data from the 1997-98, 1998-99, and most of the 1999-2000 (up through April 2000) cool seasons. For developmental purposes, the cool season is defined as September 16 through May 15 for the contiguous U.S., and September 1 through May 31 for Alaska.
e. Equation Characteristics
In the precipitation type development, the equations to predict each of the three conditional probabilities for a specific projection were developed simultaneously. As a result, all three equations contain the same terms, but the coefficients vary among the predictands. Simultaneous development and the nature of the mutually exclusive binary predictands insure that the forecasts sum to 100%.
The multiple linear regression routine that produces the forecast equations uses a forward selection method where predictors are added to the equation until a specified stopping criterion is reached. In the precipitation type development, the regression procedure stopped when 10 terms had been added to the equation, or when none of the remaining terms reduced the variance by an additional 0.50%.
Both primary and secondary equations were developed for the 6- to 18-h projections. The primary equations included surface observations as predictors, and are used in the operational setting when observations are available. When no observations are available for a particular station, the secondary equations are used. In the development of these secondary equations, surface observations were not offered as potential predictors.
f. Determining Thresholds for Categorical Forecasts
Categorical forecasts, that is, a forecast of freezing (Z), frozen (S), or liquid (R) precipitation, conditional upon precipitation occurring, are produced from the probabilistic forecasts. The probability forecasts are compared to two threshold probabilities in order to determine the categorical precipitation type forecast. These threshold probabilities were calculated from the developmental sample for each forecast projection, model cycle, and region. For precipitation type, we chose thresholds that maximized the threat score on the dependent sample, while also maintaining a bias between 0.98 and 1.02.
3. POST PROCESSING
Twice each day, after the 0000 and 1200 UTC AVN model runs, the conditional probability of precipitation type equations are evaluated by using the appropriate AVN model predictors, recent surface observations, and geoclimatic predictors. Once the probability forecasts have been generated, they are normalized. Any probabilities less than zero are set to zero, and the remaining probabilities are divided by the sum of the positive probabilities to obtain the normalized probability. In other words, all of the precipitation type probability forecasts will be greater than or equal to zero and less than or equal to one, and the three probabilities will sum to 100%. Next, the categorical forecasts are generated by comparing the normalized forecast probabilities to the threshold probabilities. In making an operational forecast, the following procedure is used to choose the categorical forecast. To begin, the freezing precipitation probability is compared to the first threshold probability. If the freezing precipitation probability is greater than the threshold value, then freezing precipitation (Z) is chosen as the categorical forecast. If not, the freezing probability and frozen probability are added together and compared to the next threshold value. If this threshold value is exceeded, then frozen (S) is chosen as the categorical forecast. If neither threshold value is exceeded, liquid precipitation (R) is chosen as the categorical forecast.
4. OPERATIONAL PRODUCTS
AVN MOS precipitation type forecasts are available in the new AVN MOS forecast bulletin distributed under the WMO headers of FOPA20, FOUS21-26, and FOAK37-39, and the AWIPS product identifier MAV. Technical Procedures Bulletin No. 463 (Dallavalle and Erickson 2000) describes the complete AVN MOS message. In the forecast bulletin, probabilistic and categorical precipitation type forecasts are available every 3 hours from 6 to 60 hours, and then every 6 hours through 72 hours. A sample of the precipitation type portion of the forecast bulletin is shown in Fig. 3. The POZ line contains the conditional probability of freezing precipitation, and the POS line contains the conditional probability of frozen precipitation. The categorical precipitation type is shown in the TYP line: Z indicates freezing precipitation, S indicates frozen precipitation, and R indicates liquid precipitation. The probabilistic and categorical forecasts valid every 3 hours will also eventually be available from 6 through 72 hours in a binary format (BUFR) message. For users outside the National Weather Service, the new guidance is available through NOAAPORT, the Family of Services, and specific military communication circuits.
The AVN MOS forecast message is produced for 1060 sites in the U.S. and Puerto Rico. Of these 1060 sites, there is no precipitation type guidance for sites in Puerto Rico, Hawaii, southern Florida, and parts of California (see Fig. 1). Precipitation type forecasts are issued for the remainder of the sites from September 1 through May 31. Therefore, the forecast bulletins for some sites never contain the POZ, POS, and TYP lines, and all the bulletins will be missing these lines from June 1 through August 31.
In the process of developing the AVN MOS precipitation type guidance, a verification of the probabilistic and categorical forecasts for an independent sample was done. In order to evaluate the proposed MOS system, the developmental sample was divided into two parts: a dependent sample used to develop test forecast equations, and an independent sample on which to evaluate the forecasts made by the test equations. The data from September 1999 through March 2000 were held out of the developmental sample to use as independent data for verification. Forecasts for approximately 650 stations in the contiguous U.S. and Alaska were included in the verification. As a basis of comparison, we also verified the operational NGM MOS precipitation type forecasts for the same sample. It is important to note, though, that precipitation type forecasts based on the NGM are not produced for the 39-, 45-, 51-, 57-, and 63- through 72-h projections. Note that the precipitation type forecasts are only verified for cases where precipitation occurred.
Figure 4 shows the P-scores for both the AVN and NGM precipitation type probability forecasts for projections of 6 through 72 hours from the 0000 UTC model run. The P-score is the mean squared error for probability forecasts. The values can range from 0 to 2 where a smaller score represents a more accurate forecast. The P-scores for the contiguous U.S. and Alaskan sites combined show that the new AVN MOS precipitation type guidance is more accurate than the operational NGM MOS forecasts at the very early projections and beyond 48 hours, and slightly less accurate at the in-between projections. The verification results for the 1200 UTC model run forecasts (not shown) were similar.
Verifications of the categorical forecasts are presented in Figs. 5 and 6. Figure 5 shows the Heidke skill scores (HSS) for the categorical forecasts for the 0000 UTC model run. An HSS between 0 and 1 represents a skillful forecast, with a 1 denoting a perfect forecast. Figure 5 shows that the AVN categorical precipitation type forecasts are as skillful as, and quite often more skillful than, the NGM MOS categorical forecasts at nearly all projections. Figure 6 shows the threat score for the freezing category for both systems. The threat score ranges from 0 to 1, where 1 is a perfect forecast. Here again, the AVN MOS freezing forecasts from the 0000 UTC model run had higher threat scores than the NGM MOS for most projections. The erratic nature of this figure is due to the small number of freezing precipitation events in the verification sample. For most projections, the number of observed freezing precipitation events is only 1 to 2% of all precipitation events.
It is clear from the three verification figures that the primary AVN MOS forecasts are more accurate than the secondary AVN MOS forecasts at virtually all projections where both are available. Despite this lower accuracy, these secondary forecasts are preferable to a missing forecast in the event that the observational predictors are not available.
Figures 4-6 show that the AVN MOS exhibits skill comparable to that of the NGM MOS precipitation type guidance. At some projections, the verification scores are lower for the AVN MOS. We believe that this is due to the smaller developmental sample on which the AVN MOS precipitation type equations were developed. Also, changes made to the AVN model during the developmental period may have resulted in a less stable sample than was used to develop the NGM MOS precipitation type system. We will redevelop the forecast equations as more developmental data become available. In the meantime, the availability of more detailed precipitation type guidance based on the AVN model, the additional stations, the extension to 72 hours, and the fact that the differences in skill between the AVN and NGM MOS are small, contribute to making this a useful guidance product for forecasters.
For additional verification scores of precipitation type and other MOS element developments, please see the Test Results web page located at http://www.nws.noaa.gov/tdl/synop/results.htm .
6. OPERATIONAL CONSIDERATIONS
At the time this Bulletin was written, the new AVN MOS precipitation type guidance had just been implemented. Therefore, we had not had a chance to monitor the product's performance in a true operational setting.
It is important to point out that these are conditional probabilities of precipitation type. Given that precipitation occurs, these are the probabilities that precipitation will fall in the specified form. Therefore, it is suggested that one use this guidance in conjunction with the probability of precipitation (PoP) forecasts. If the conditions are not right for precipitation to occur, the precipitation type forecast is virtually meaningless.
The MOS technique can account for some systematic model biases, as well as the reduced skill of the model with increasing projection. Unfortunately, the MOS guidance is not able to overcome a bad model forecast; the guidance will reflect the main patterns in the AVN model output. Also, the MOS equations are tuned to the specific sample on which they were developed. For example, if the developmental sample was a relatively dry period, the equations might not perform as well during a significant wet period. Additionally, future changes to the AVN model that affect the model biases may affect the performance of the MOS forecasts.
The user may notice the absence of the probability of liquid precipitation forecasts in the alphanumeric message. Remember that the three probability equations were developed simultaneously and, therefore, the probabilities produced from the equations sum to 100%. In the interest of space, the probability of liquid precipitation was omitted, but can be obtained by subtracting the sum of the POZ and POS from 100%.
Keep in mind that for some forecast projections in Alaska no equations to forecast the probability of freezing precipitation were developed. Thus, when the categorical forecasts are produced for those projections, the possibility of freezing precipitation occurring is not taken into account.
Bernstein, Ben C., 2000: Regional and local influences on freezing drizzle, freezing rain, and ice pellet events. Wea and Forecasting, 15, 485 - 508.
Dallavalle, J. P., and M. C. Erickson, 2000: AVN-based MOS guidance - The alphanumeric messages. NWS Technical Procedures Bulletin No. 463, NOAA, U.S. Dept. of Commerce, 9 pp.
Erickson, Mary C., 1995: NGM-based MOS precipitation type forecasts for the United States.NWS Technical Procedures Bulletin No. 421, NOAA, U.S. Dept. of Commerce, 12 pp. Glahn, H.R., and D.A. Lowry, 1972: The use of Model Output Statistics (MOS) in objective weather forecasting. J. Appl. Meteor., 11, 1203-1211.
Jensenius, J. S., Jr., 1992: The use of grid-binary variables as predictors for statistical weather forecasting. Preprints Twelfth Conference on Probability and Statistics in the Atmospheric Sciences, Toronto, Amer. Meteor. Soc., 225-230.
Kanamitsu, M.., 1989: Description of the NMC global data assimilation and forecast system. Wea. and Forecasting, 4, 335 - 342.