Part IV:   What products can be derived from an  ensemble forecast?

      In general, three types of product can be derived from an ensemble: a most probable single solution or consensus forecast, uncertainty measure and a distribution of all possible solutions. Depending on circumstances, most probable single solution could be represented by simple ensemble mean, median or mode of members. Besides, some efforts are also made on constructing such a consensus forecast through more sophisticated methods such as linear regression, ensemble MOS (Gneiting et. al., 2005), performance-based weighting (Woodcock and Engel, 2005), clustering (Greybush et. al., 2007) and Bayesian Model Averaging (BMA) (Raftery et. al., 2005). Note that the “most probable or best solution” is not measured by a particular single realization but by average over a large number of realizations from a statistically reliable EPS (see Part 6). Advantages of the most probable single forecast are information highly compacted with only one single value, easy to understand and use, less confusing, simple and acceptable to most general users and public besides being more accurate statistically. Disadvantages of ensemble averaging are smoothing out spatial details, overestimating light precipitation area coverage and underestimating heavy precipitation area coverage (Du et. al., 1997), and even misleading in bi-modal or multi-modal situations besides providing no uncertainty information. For example, ensemble mean wind speed is very misleading when large uncertainty exists in wind direction among ensemble members. Therefore, we suggest that ensemble spread should also be used to help us to correctly interpret ensemble mean information: e.g., mean is probably more trustable when it’s associated with small spread and less trustable when associated with large spread. Under “large spread” situation, other products should also be looked at for further assessment. 

    Forecast variance among ensemble members can be used to quantify forecast uncertainty. Standard deviation of members with respect to ensemble mean is usually defined as forecast variance known as ensemble spread. Large (small) spread indicates a low (high) confident forecast. To more insightfully interpret the significance of spread information in a forecast, ensemble spread is often normalized by or compared with climate anomaly (Hart and Grumm, 2001). Given that natural variability of a field is quite different over different geographical zones (e.g., large in mid- and high-latitudes and small in tropics), it is sometimes useful too to normalize spread by an averaged spread over a past time period in space to reflect true predictability of atmospheric motion. Advantages of spread are information highly compacted, easy to understand and ability of distinguishing between phase and intensity or amplitude uncertainty of a weather system if combining with ensemble mean information. A main disadvantage is that spread information doesn’t tell how members are actual distributed such as normally distributed or skewed or multi-modal distribution. For some variables such as precipitation, forecast variance is somehow related to their ensemble mean values. Under such circumstances, spread doesn’t really reflect truth predictability of an event. By normalizing spread of such variables by their mean value might be helpful. In reality, since an ensemble system is not perfectly designed (see Part 3), spread doesn’t perfectly reflect true predictability or perfectly correlate with forecast skill. Therefore, a post-processing or calibration is necessary to derive various versions of predictability or confidence related measurement based on raw spread information combining with other information (e.g., IM et. al., 2006).

    As discussed in Section 2.2, a complete forecast should be a distribution rather than a single value. To retain full or maximum information in a forecast, many distributive-type of products can be made from an ensemble such as probability, postal stamp chart (individual members), clustering, spaghetti chart, plume diagram (time evolution of forecast values at a given location), extremes or envelope, and three-represented-value type (such as 10-50-90%). A probabilistic forecast can be in 2-D or 3-D spatial distribution at a given time for a given category or a 1-D distribution of probability over all categories at a given location and time or a 1-D time evolution of a probabilistic value (probogram) at a given location. By the way, it needs to keep in mind that the “probability” derived from a finite ensemble (sampling issue) is, theoretically, not true probability but relative frequency. Main advantage of distributive and probabilistic product is full information contained and conveyed, which greatly reduces the chance of having “surprising storms/events” (both missing and false alarm) in forecasts by providing probabilistic heads-up or heads-down information of less-predictable but high-impact weather events in advance. This important piece of information might be captured by only a few members in the ensemble and, therefore, could be completely filtered out in other types of product such as ensemble mean. On the other hand, since distributive and probabilistic products are not deterministic but a range of possibilities, users are often hesitative to make a decision based on probability. How to use probabilistic information in decision-making will be briefly discussed in Section 2.7. As always, each type of products has its own strength and weakness. 2D-probability is displayed only for a given threshold, so that one needs to look at other thresholds at the same time to get a full picture. In conditional probably, one needs to quilt two or more pieces of information together, e.g., only knowing the probability of precipitation type being snow if precipitation occurs (a given condition) is not enough, he also needs to know what is the chance to have the precipitation based on full ensemble members in order to know the whole story. Displaying all individual members together, known as Postal Stamp chart, is found useful for forecasters to have a qualitative first glance to know how diverse among ensemble members, what kind of extremes could be and what to expect on average. Grouping of members into several regimes (Clustering and Tubing, e.g., Alhamed et. al., 2002; Yussouf et. al., 2004; Marzban and Sandgathe, 2006; Atger, 1999a) will provide useful insight in many situations such as multi-modal and regime transition period. Although, mathematically, one can always cluster members into several categories, the separation among clustered groups is not always physically or meteorologically significant. To ensure the separation of clustered groups is meteorologically significant, clustering technique needs to be carefully designed and the significance of separation between groups needs to be statistically tested besides the necessity to have a sufficiently large ensemble size. Spaghetti chart (a group of contours of a selected value from each member being plotted together) is widely used. Since it gives full solutions of all members, one can clearly picture what the mean, mode, distribution and outlier are. Disadvantage is that it’s not always pre-known which contour value is proper to be picked for a particular event since there are only a few values (usually one or

two) can be displayed in a spaghetti chart. Envelope display is good to see possible extremes at both low and high ends but lacks of detailed member distribution within. However, for a given location, the time evolution of both extreme values and member distribution can be displayed together as a plume diagram. In practice, to reduce data volume while still possibly to retain critical forecast information, a compromised version of full probabilistic distribution is the three-represented-value type product. Two values represent extremes at low and high ends (e.g., either extreme values themselves or values associated with 90% and 10% probability or similar), while one value represents a most probable solution like median, mode or mean. All the above three types of products (most probable, uncertainty and distribution) can sometimes be combined into one single product such as Fig. 3 (nicely showing uncertainties in both wind speed and direction) although it’s not always easy to do so. How to condense abundant ensemble information into a simple product which can be easily understood and used by forecasters and end-users is an issue needs to be further studied. Storm Prediction Center of NCEP developed a lot of multi-variable joint-probability type of products which is found a meaningful way in fire weather, convection and winter storm forecasting (Fig. 4, Bright et. al., 2004 and Weiss et. al., 2007).

    Usually, IC perturbations are centered around a control analysis, the control member tends to remain near the center of an ensemble cloud and close to ensemble mean in linear or quasi-linear situation. Hence, ensemble mean often verifies more accurate than

Figure 3

Figure 4

other perturbed members, which implies that control member could have different statistical property comparing to other perturbed members. Therefore, a question is that control member should be included or excluded or be weighted more in calculating final ensemble products such as probability, spread and mean? Practically, this question only matters when ensemble size is small (say less than 50) since including or excluding control member should not make much difference for a large size ensemble. Also, in a highly nonlinear situation or longer range time integration, control member could be anywhere within the ensemble cloud. In that case, control member should perform similarly as other perturbed members and, therefore, be included in the calculation without being distinguished. Only in the situation where ensemble size is small and flow is relatively linear, this question becomes relevant. The answer to this question may vary depending mainly on two factors: whether an EPS is biased or unbiased and has sufficient spread or not. If an EPS is unbiased and has sufficient spread or over-dispersive, control member is often more accurate and plays more important role and, therefore, it should be included in calculating probability, be weighted more in calculating ensemble consensus forecast but may be excluded in calculating spread to maximize spread in an under-dispersive ensemble (since the inclusion of control run might reduce spread because the control forecast is probably closer to ensemble mean under linear or quasi-linear situations). Otherwise, if an EPS is either severely biased or under-dispersive, truth is often outside the ensemble cloud. In that case, control member (near the center of ensemble cloud) has less chance to be correct than some perturbed members and, therefore, might not be included in ensemble products to avoid a possibly degraded forecast. By the way, from the above discussion, one might see that the difference between ensemble mean and control member could be used as a measure of nonlinearity: the larger this difference, the higher nonlinear a flow exhibits.

Contact  Jun Du