Although, given 100% accurate ICs and other conditions, the atmospheric system itself and numerical prediction models should be deterministic in theory. But, in reality, due to intrinsic uncertainties in IC and model configurations plus chaotic nature of nonlinear models, forecast uncertainty and predictability limit is a very real and important property of NWP. Without quantifying uncertainty, a forecast is incomplete. A complete forecast should be in a probabilistic distribution with uncertainty expressed but not in a single deterministic value. Ensemble forecasting is a dynamical approach to quantify forecast uncertainty. It’s a relatively new but a rapidly developing branch of NWP and expected to become a main tool in weather, climate and water prediction in near future. Ensemble forecasting is most valuable when large uncertainty is around and forecasters don’t know what solution to choose from in mainly high-impact events and has minimal value when weather is quiescent and highly predictable (although one still needs ensemble to identify such occasions). Currently, many major NWP centers around the world have already operationally implemented various ensemble prediction systems as part of their daily production although those EPSs are still in primitive and evolving stage. More and more forecasters and other users are lean toward using ensemble products instead of single deterministic model run nowadays. This trend will certainly continue in years to come. The ensemble-based NWP paradigm is superior to the single-value base forecast by providing flow-dependent uncertainty information besides an improved most probably single solution and taking observation and IC errors into account too. To complete a smooth transition from single-forecast based paradigm to ensemble based one, much effort is needed. Education, coordination and training should play a key role in this transition. The recent U.S. National Research Council report - Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts - is a good example of this effort. The currently going on UCAR COMET classroom training courses and local weather forecasting office’s training workshop etc. are all good formats in training and education efforts.

The primary mission of ensemble forecasting is to reliably quantify forecast uncertainty and accurately describe a flow-dependent forecast error distribution to have the truth be always encompassed by ensemble cloud. In general, three types of product can be derived from an ensemble: a most probable single solution or consensus forecast, uncertainty measure and a distribution of all possible solutions. There are still many areas to be explored to maximize the utility of an ensemble, for example, how to better express and convey ensemble information to users in a comprehensive and easily-understandable way; complementary role between higher-resolution single model run and lower-resolution ensemble forecasts; better post-processing of ensemble forecasts including statistical and dynamical approaches, ensemble MOS, usage of re-forecasting or hindcast dataset and downscaling; and economic-value based decision-making process in using forecast uncertainty and probabilistic information.

To better accomplish the ensemble forecasting mission, a 3-dimentional type of EPS is needed by fully capturing all uncertainty sources from IC dimension, model-configuration dimension and history-memory dimension. Certainly, many needs to be further investigated and improved such as how to best couple with data assimilation in IC perturbation generating, stochastic physics perturbation and the value of history-memory dimension. Flow-dependent adaptive multi-EPS across the full spectrum of multi-scales is an area of interest and exploration. Theoretical ensemble or error dynamics is yet to be developed to fully understand how error evolves and propagates in the governing equations of a model.

Uncertainty is the only thing certain in the real world. Downstream application of meteorological ensemble forecasting is high in demanding and a rapidly growing area. Besides driving many downstream prediction systems like hydrology, air quality, storm surge, ocean wave, dispersion, geological prediction and electricity generation etc., adaptive/targeted observation is a special area of application within meteorology itself. Taking uncertainty into picture is a step forward in science and a way to better serve society and people. For the further reading about predictability of weather and climate, readers are referred to Palmer and Hagedorn (2006).

References

Alhamed, A., S. Lakshimivarahan and D. J. Stensrud, 2002. Cluster analysis of multi-model ensemble data from SAMEX. Mon. Wea. Rev., 130, 226-255.

Aligo, E.A., W.A. Gallus and M. Segal, 2007. Summer Rainfall Forecast Spread in an Ensemble Initialized with Different Soil Moisture Analyses. Wea. Forecasting, 22, 299–314.

Anderson, J. L, 1996. A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Climate, 9, 1518-1530.

Anderson, J. L., 2001. An ensemble adjustment Kalman filter for data assimilation. Mon. Wea. Rev., 129, 2884-2903.

Atger, F., 1999a. Tubing: An Alternative to Clustering for the Classification of Ensemble Forecasts. Wea. Forecasting, 14, 741–757.

Atger F., 1999b. The skill of ensemble prediction systems. Mon. Wea. Rev., 127, 1941–1953.

Barkmeijer, J., 1996. Constructing fast-growing perturbations for the nonlinear regime. J. Atmos. Sci., 53, 2838-2851.

Barkmeijer, J., M. Van Gijzen and F. Bouttier, 1998. Singular vectors and the estimates of analysis-error covariance matrix. Quart. J. Roy. Meteor. Soc., 124, 1695-1713.

Berrocal, V.J., A.E. Raftery and T. Gneiting, 2007. Combining Spatial Statistical and Ensemble Information in Probabilistic Weather Forecasts. Mon. Wea. Rev., 135, 1386–1402.

Bishop, C. H. and Z. Toth, 1999. Ensemble transformation and adaptive observation. J. Atmos. Sci., 56, 1748-1765.

Bishop, C. H., B. J. Etherton and S. Majumdar, 2001. Adaptive sampling with the ensemble transform Kalman filter. Part I: theoretical aspects. Mon. Wea. Rev., 129, 420-436.

Bowler, N. E. 2006. Comparison of error breeding, singular vectors, random perturbations and ensemble Kalman filter perturbation strategies on a simple model. Tellus, 58A, 538-548.

Brankovic, C., T. N. Palmer, F. Molteni, S. Tibaldi and U. Cubasch, 2006. Extended-range predictions with ECMWF models: Time-lagged ensemble forecasting. Quart. J. Roy. Meteor. Soc., 116, 867-912.

Brier, G. W., 1950. Verification of forecasts expressed in terms of probability. Mon. Wea. Rev., 75, 1-3.

Bright, D.R. and S.L. Mullen, 2002a. Short-Range Ensemble Forecasts of Precipitation during the Southwest Monsoon. Wea. Forecasting, 17, 1080–1100.

Bright, D.R. and S.L. Mullen, 2002b. The Sensitivity of the Numerical Simulation of the Southwest Monsoon Boundary Layer to the Choice of PBL Turbulence Parameterization in MM5. Wea. Forecasting, 17, 99–114.

Bright, D. R., S. J. Weiss, J. J. Levit, M. S. Wandishin, J. S. Kain and D. J. Stensrud, 2004. Evaluation of short-range ensemble forecasts during the 2003 SPC/NSSL Spring Program. Preprints, 22nd Conf. on Severe Local Storms, Hyannis, MA, P15.5.

Brooks, H. E., M. S. Tracton, D. J. Stensrud, G. DiMego and Z. Toth, 1995. Short-range ensemble forecasting: Report from a workshop, 25-27 July 1994. Bull. Amer. Meteor. Soc., 76, 1617-1624.

Brown, T. A., 1974. Admissible scoring systems for continuous distributions. Manuscript P-5235, The Rand Corporation, Santa Monica, CA, 22 pp. [Available from The Rand Corporation, 1700 Main St., Santa Monica, CA 90407-2138]

Buizza, R., 1994. Sensitivity of optimal unstable structures. Quart. J. Roy. Meteor. Soc., 120, 429-451.

Buizza R., A. Hollingsworth, F. Lalaurette and A. Ghelli, 1999a. Probabilistic predictions of precipitation using the ECMWF Ensemble Prediction System. Wea. Forecasting, 14, 168–189.

Buizza, R., M. Miller and T. N. Palmer, 1999b. Stochastic representation of model prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887-2908.

Cai, M., E. Kalnay and Z. Toth, 2002. Bred vectors of the Zebiak-Cane model and their application to ENSO prediction. J. Climate, 16, 40-56.

Cao, H, 2002. Memorial dynamics of systems and its applications. Chinese Geology Press, Beijing, China, 192pp. (in Chinese)

Chen, H. S., 2006. Ensemble prediction of ocean waves at NCEP. Proceedings of the 28th Ocean Engineering Conference in Taiwan, National Sun Yat-Sen University, November 2006, pp25-37.

Chen, J., J. Xue and H. Yang, 2003. Impact of physical parameterization schemes on mesoscale heavy rain simulations. Acta Meteorologica Sinica, 61, 203-218.

Cheung, K. W. C. and J. C. L. Chan, 1999a. Ensemble forecasting of tropical cyclone motion using a barotropic model. Part I: perturbations of the environment. Mon. Wea. Rev., 127, 1229-1243.

Cheung, K. W. C. and J. C. L. Chan, 1999b. Ensemble forecasting of tropical cyclone motion using a barotropic model. Part II: perturbations of the vortex. Mon. Wea. Rev., 127, 2617-2640.

Chien, F.C., Y.C. Liu and B.J.D. Jou, 2006. MM5 Ensemble Mean Forecasts in the Taiwan Area for the 2003 Mei-Yu Season. Wea. Forecasting, 21, 1006–1023.

Cui, B., Z. Toth, Y. Zhu, D. Hou and S. Beauregard, 2005. Statistical post-processing of operational and CDC hindcast ensembles. Preprints, 21th Conf. on Wea. Analysis and Forecasting and 17th Conf. on Numerical Weather Prediction. Washington, DC, Aug. 1-5, 2005, Amer. Meteor. Soc., 12B.2.

Du, J., 2004. Hybrid Ensemble Prediction System: a New Ensembling Approach. Preprints, Symposium on the 50th Anniversary of Operational Numerical Weather Prediction, University of Maryland, College Park, Maryland, June 14-17, 2004, Amer. Meteor. Soc., CD-ROM (paper p4.2, 5pp). [available online: http://www.emc.ncep.noaa.gov/mmb/SREF/reference.html].

Du, J., 2005: Impact of Model Error and Imperfect Initial Condition Perturbations on Ensemble-Based Probabilistic Forecasts: UNPREDICTABLE SPOTS. Preprints, 17th Conference on Numerical Weather Prediction/21st Conference on Weather Analysis and Forecasting, Washington DC., Aug. 1-5, 2005, Amer. Meteor. Soc. (paper 15B.6) [available online http://www.emc.ncep.noaa.gov/mmb/SREF/reference.html].

Du, J. and G. DiMego, 2008. A regime-dependent bias correction approach. 19th Conf. on Probability and Statistics, Jan. 20-24, 2008, New Orleads, LA, paper 3.2.

Du, J., G. DiMego, M. S. Tracton, and B. Zhou 2003. NCEP short-range ensemble forecasting (SREF) system: multi-IC, multi-model and multi-physics approach. Research Activities in Atmospheric and Oceanic Modelling (edited by J. Cote), Report 33, CAS/JSC Working Group Numerical Experimentation (WGNE), WMO/TD-No. 1161, 5.09-5.10.

Du, J., G. Gayno, K. Mitchell, Z. Toth and G. DiMego, 2007a. Sensitivity study of T2m and precipitation forecasts to initial soil moisture conditions by using NCEP WRF ensemble. 22nd WAF/18th NWP conference, Park City, UT, AMS.

Du, J., J. McQueen, G. DiMego, T. Black, H, Juang, E. Rogers, B. Ferrier, B. Zhou, Z. Toth and M. S, Tracton, 2004. The NOAA/NWS/NCEP short-range ensemble forecast (SREF) system: evaluation of an initial condition vs. multi-model physics ensemble approach. Preprints (CD), 16th Conference on Numerical Weather Prediction, Seattle, Washington, Amer. Meteor. Soc.

Du, J., S. L. Mullen and F. Sanders, 1997. Short-range ensemble forecasting of quantitative precipitation. Mon. Wea. Rev., 125, 2427-2459.

Du, J., S. L. Mullen and F. Sanders, 2000. Removal of distortion error from an ensemble forecast. Mon. Wea. Rev., 128, 2427-3351.

Du, J. and M. S. Tracton, 1999. Impact of lateral boundary conditions on regional-model ensemble predicion. Research Activities in Atmospheric and Oceanic Modelling (edited by H. Ritchie), Report 28, CAS/JSC Working Group Numerical Experimentation (WGNE), WMO/TD-No. 942, 6.7-6.8.

Du, J. and M. S. Tracton, 2001. Implementation of a real-time short-range ensemble forecasting system at NCEP: an update. Preprints, 9th Conference on Mesoscale Processes, Ft. Lauderdale, Florida, Amer. Meteor. Soc., 355-356.

Du, J., R. Yu, C. Cui and J. Li, 2007b. Using Mesoscale Ensemble to Predict Forecast Error Distribution and Target Observation. Acta Oceanologica Sinica, submitted.

Ebert, E.E., 2001. Ability of a Poor Man's Ensemble to Predict the Probability and Distribution of Precipitation. Mon. Wea. Rev., 129, 2461–2480.

Ebisuzaki, W. and E. Kalnay, 1991. Ensemble experiments with a new lagged average forecasting scheme. WMO, Research activities in atmospheric and oceanic modeling. Report 15, 6.31-32.

Eckel, F. A., 2005. Plan for the joint ensemble forecast system (JEFS). Air Force Weather Agency.

Eckel. F. A. and C. F. Mass, 2005. Aspects of Effective Mesoscale, Short-Range Ensemble Forecasting. Wea. Forecasting, 20, 328-350.

Eckel, F. A. and M. K. Walters, 1998. Calibrated probabilistic quantitative precipitation forecasts based on the MRF ensemble. Wea. Forecasting, 13, 1132-1147.

Ehrendorfer, M., R.M. Errico and K.D. Raeder, 1999. Singular-Vector Perturbation Growth in a Primitive Equation Model with Moist Physics. J. Atmos. Sci., 56, 1627–1648.

Epstein, E. S., 1969. Stochastic-dynamic prediction. Tellus, 21, 739-759.

Epstein, E. S., 1969: A scoring system for probability forecasts of ranked categories. J. Appl. Meteor., 8, 985–987.

Errico, R., 1982. What is an adjoint model? Bull. Amer. Meteor. Soc., 78, 2577-2591.

Errico, R. and D. Baumhefner, 1998. Predictability experiments using a high-resolution limited area model. Mon. Wea. Rev., 115, 488-504.

Errico, R. and T. Vukicevic, 1992. Sensitivity analysis using an adjoint of the PSU-NCAR mesoscale model. Mon. Wea. Rev., 120, 1644-1660.

Farrell, B. F., 1990. Small error dynamics and the predictability of atmospheric flows. J. Atmos. Sci., 47, 2409-2416.

Fischer, M., A. Joly and F. Lalaurette, 1998. Error growth and Kalman filtering within an idealized baroclinic flow. Tellus, 50A, 596-615.

Franz, K., N. Ajami, J. Schaake and R. Buizza, 2005. Hydrologic Ensemble Prediction Experiment Focuses on Reliable Forecasts, Eos, 86, No. 25.

Gao S.-T., 2007. Atmospheric mesoscale dynamics and forecasting methods. Meteorological Press, Beijing, China, 215pp. (in Chinese)

Gao, J., J. Du, M. Xue and K. Droegemeier, 2007. An efficient approach of ensemble Kalman filter for data assimilation. Geophys. Res. Lett., submitted.

Gao J. and M. Xue, 2007. An efficient dual-resolution approach for ensemble data assimilation and tests with simulated Doppler radar data. Mon. Wea. Rev., in press.

Gilmour, I. and L. A. Smith, 1997. Enlightenment in shadows. Applied nonlinear dynamics and stochastic systems near the millennium (edited by J. B. Kadtke and A. Bulsara), AIP, 335-340.

Glahn, H. R. and D. A. Lowry, 1972. The use of model output statistics in objective weather forecasting. J. Appl. Meteor., 11, 1203-1211.

Gneiting, T., A. E. Raftery, A. Westveld and T. Goldman, 2005. Calibrated probabilistic forecasting using Ensemble Model Output Statistics and Minimum CRPS estimation, Mon. Wea. Rev., 133, 1098-1118.

Gombos, D., J. A. Hansen, J. Du and J. McQueen, 2007. Theory and Applications of the Minimum Spanning Tree Rank Histogram. Mon. Wea. Rev., 135, 1490–1505.

Gray, M. E. B. and G. J. Shutts, 2002. A stochastic scheme for representing convectively generated vorticity sources in general circulation models. APR Turbulence and Diffusion Note, 285. Met Office, UK.

Greybush, S. J., S. E. Haupt and G. S. Young, 2007. The regime dependence of optimally weighted ensemble model consensus forecasts. Wea. Forecasting, (in revision).

Grimit, E. P., T. Gneiting T, V. J. Berrocal and N. A. Johnson, 2006. The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification. Quart. J. Roy. Meteor. Soc., 132, 2925-2942.

Grimit, E.P., and C.F. Mass, 2002. Initial Results of a Mesoscale Short-Range Ensemble Forecasting System over the Pacific Northwest. Wea Forecasting, 17, 192–205.

Grimit E. P. and C. F. Mass, 2007. Measuring the Ensemble Spread-Error Relationship with a Probabilistic Approach: Stochastic Ensemble Results. Mon. Wea. Rev., 135, 203-221.

Hagedorn, R., T. M. Hamill and J. S. Whitaker, 2007. Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part I: 2-meter temperature. Submitted to Mon. Wea. Rev.

Hamill, T. M., 2001. Interpretation of rank histograms for verifying ensemble forecasts. Mon. Wea. Rev., 129, 550-560.

Hamill, T. M., 2006. Ensemble-based atmospheric data assimilation. Chapter 6 of Predictability of Weather and Climate, Cambridge Press, 124-156.

Hamill, T. M. and S. J. Colucci, 1997. Verification of Eta-RSM short-range ensemble forecasts. Mon. Wea. Rev., 125, 1322-1327.

Hamill, T. M. and S. J. Colucci, 1998. Evaluation of Eta-RSM ensemble probabilistic precipitation forecasts. Mon. Wea. Rev., 126, 711-724.

Hamill, T. M. and J. Juras, 2006. Measuring forecast skill: is it real skill or is it the varying climatology?. Quart. J. Roy. Meteor. Soc., 132, 2905-2923.

Hamill, T. M., R. Hagedon and J. S. Whitaker, 2007. Probabilistic forecast calibration using ECMWF and GFS ensemble reforecasts. Part II: precipitation. Submitted to Mon. Wea. Rev.

Hamill, T. M., J. A. Hansen, S. L. Mullen and C. Snyder, 2004a. Meeting summary: workshop on ensemble forecasting in the short to medium range. Submitted to Bull. Amer. Meteor. Soc. [available online: http://www.cdc.noaa.gov/people/tom.hamill/cv.html].

Hamill, T.M., S.L. Mullen, C. Snyder, Z. Toth and D.P. Baumhefner, 2000. Ensemble Forecasting in the Short to Medium Range: Report from a Workshop. Bull. Amer. Meteor. Soc., 81, 2653–2664.

Hamill, T.M. and J.S. Whitaker, 2007. Ensemble Calibration of 500-hPa Geopotential Height and 850-hPa and 2-m Temperatures Using Reforecasts. Mon. Wea. Rev., 135, 3273–3280.

Hamill, T.M., J. S. Whitaker and S.L. Mullen, 2006. Reforecasts: An Important Dataset for Improving Weather Predictions. Bull. Amer. Meteor. Soc., 87, 33–46.

Hamill, T.M., J. S. Whitaker and X. Wei, 2004b. Ensemble Reforecasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts. Mon. Wea. Rev., 132, 1434–1447.

Hansen, J.A., 2002. Accounting for Model Error in Ensemble-Based State Estimation and Forecasting. Mon. Wea. Rev., 130, 2373–2391.

Hart, R. E. and R.H. Grumm, 2001. Using Normalized Climatological Anomalies to Rank Synoptic-Scale Events Objectively. Mon. Wea. Rev., 129, 2426–2442.

Harvey, L. O. Jr., K. R. Hammond, C. M. Lusk and E. F. Mross, 1992. The application of signal detection theory to weather forecasting behavior. Mon. Wea. Rev., 120, 863-883.

Hersback, H., 2000. Decomposition of the continuous ranked probability score for ensemble prediction system. Wea. Forecasting, 15, 559-570.

Hoffman, R. N. and E. Kalnay, 1983. Lagged average forecasting, an alternative to Monte Carlo forecasting. Tellus, 35A, 100-118.

Homar, V., D.J. Stensrud, J.J. Levit and D.R. Bright, 2006. Value of Human-Generated Perturbations in Short-Range Ensemble Forecasts of Severe Weather. Wea. Forecasting, 21, 347–363.

Hou, D., E. Kalnay and K. K. Droegemeier, 2001. Objective verification of the SAMEX'98 ensemble forecasts. Mon. Wea. Rev., 129, 73-91.

Houtekamer, P. L., L. Lefaivre, J. Derome, H. Ritchie and H. L. Mitchell, 1996. Asystem simulation approach to ensemble prediction. Mon. Wea. Rev., 124, 1225-1242.

Houtekamer, P. L. and H. L. Mitchell, 1998. Data assimilation using an ensemble Kalman filter technique. Mon. Wea. Rev., 126, 196-811.

Im, J. S., K. Brill and E. Danaher, 2006. Confidence Interval Estimation for Quantitative Precipitation Forecasts (QPF) Using Short-Range Ensemble Forecasts (SREF). Wea. Forecasting, 21, 24–41.

Jankov, I., W.A. Gallus, M. Segal and S.E. Koch, 2007. Influence of Initial Conditions on the WRF–ARW Model QPF Response to Physical Parameterization Changes. Wea. Forecasting, 22, 501–519.

Jankov, I., W.A. Gallus, M. Segal, B. Shaw and S.E. Koch, 2005. The Impact of Different WRF Model Physical Parameterizations and Their Interactions on Warm Season MCS Rainfall. Wea. Forecasting, 20, 1048–1060.

Jolliffe, I. T. and D. B. Stephenson, 2003. Forecast verification: a practitioner’s guider in atmospheric science. John Wiley & Sons, Ltd., England, 240pp.

Jones, M.S., B.A. Colle and J.S. Tongue, 2007. Evaluation of a Mesoscale Short-Range Ensemble Forecast System over the Northeast United States. Wea. Forecasting, 22, 36–55.

Kalnay, E, 2003. Atmospheric modeling, data assimilation and predictability. Cambridge University Press, 368pp.

Kalnay, E., 2007. A talk at Arakawa Symposium of 2007 American Meteorological Society annual meeting, San Antonio, TX.

Keil, C. and G.C. Craig, 2007. A Displacement-Based Error Measure Applied in a Regional Ensemble Forecasting System. Mon. Wea. Rev., 135, 3248–3259.

Kong, F., K. K. Droegemeier and N.L. Hickmon, 2006. Multiresolution ensemble forecasts of an observed tornadic thunderstorm system. Part I: Comparsion of coarse- and fine-grid experiments. Mon. Wea. Rev., 134, 807–833.

Kong, F., K. K. Droegemeier and N.L. Hickmon, 2007. Multiresolution ensemble forecasts of an observed tornadic thunderstorm system, Part II. Mon. Wea. Rev., 135, 759-782.

Kong, F., M. Xue, D. Bright, M. C. Coniglio, K. W. Thomas, Y. Wang, D. Weber, J. S. Kain, S. J. Weiss and J. Du, 2007. Preliminary analysis on the real-time storm-scale ensemble forecasts produced as a part of the NOAA hazardous weather testbed 2007 spring experiment. 22nd Conf. WAF/18th Conf. NWP, Salt Lake City, Utah, Amer. Meteor. Soc., CDROM 3B.2.

Krishnamurti, T. N., C. M. Kishtawal, T. LaRow, D. Bachiochi, Z. Zhang, C. E. Williford, S. Gadhil and S. Surendran, 1999. Improved weather and seasonal climate forecasts from multimodel superensemble. Science, 285, 1548-1550.

Krishnamurti, T. N., C. M. Kishtawal, Z. Zhang, T. LaRow, D. Bachiochi and C. E. Williford, 2000. Multimodel ensemble forecasts for weather and seasonal climate. J. Climate, 13, 4196-4216.

Legg, T. E. and K. Mylne 2004. Early Warnings of Severe Weather from Ensemble Forecast Information. Wea. Forecasting, 19, 891-906.

Leith, C. E., 1974. Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev., 102, 409-418.

Li, Z. and D. Chen, 2002. The development and application of the operational ensemble prediction system at National Meteorological Center. J. of Appl. Meteor. Sci. (in Chinese), 13, 1-15.

Li, X., M. Charron, L. Spacek and G. Candille, 2007. A regional ensemble prediction system based on moist targeted singular vectors and stochastic parameter perturbations. Mon. Wea. Rev., (in press).

Lorenz, E. N., 1963. Deterministic nonperiodic flow. J. Atmos. Sci., 20, 130-141.

Lorenz, E. N., 1965. A study of the predictability of a 28-variable atmospheric model. Tellus, 17, 321-333.

Lorenz, E. N., 1993. The Essence of Chaos. University of Washington Press, Seattle, 240pp.

Lu, C., H. Yuan, B.E. Schwartz and S.G. Benjamin, 2007. Short-Range Numerical Weather Prediction Using Time-Lagged Ensembles. Wea. Forecasting, 22, 580–595.

Majumar, S. J., C. H. Bishop and B. J. Etherton, 2002. Adaptive sampling with ensemble transform Kalman filter. Part II: filed program implementation. Mon. Wea. Rev.

Martin, A., V. Homar, L. Fita, J. M., Gutierrez, M. A.. Rodriguez and C. Primo, 2007. Geometrid vs. classical breeding of vectors: application to hazardous weather in the Western Mediterranean. Geophysical Research Abstracts, 9, European Geosciences Union.

Marzban, C. and S. Sandgathe, 2006. Cluster Analysis for Verification of Precipitation Fields. Wea. Forecasting, 21, 824–838.

Matsueda M., M. Kyouda, H. L. Tanaka and T. Tsuyuki, 2007. Daily Forecast Skill of Multi-Center Grand Ensemble. SOLA, 3, 29-32.

Mclay, J. M., C. H. Bishop and C. A. Reynolds, 2007. The ensemble-transform scheme adapted for the generation of stochastic forecast perturbations. Quart. J. Roy. Meteor. Soc., 133, 1257-1266.

Mittermaier, M. P., 2007. Improving short-range high-resolution model precipitation forecast skill using time-lagged ensembles. Qart. J. Roy. Meteor. Soc., submitted.

Molteni F, R. Buizza, C. Marsigli, A. Montani, F. Nerozzi and T. Paccagnella, 2001. A strategy for high-resolution ensemble prediction. I: Definition of representative members and global-model experiments. Quart. J. Roy. Meteor. Soc., 127, 2069-2094.

Mu, M., 2000. Nonlinear singular vectors and nonlinear singular values. Scinece in China (D), 43, 375-385.

Mu M. and W. S. Duan, 2003. A new approach to study ENSO predictability: conditional nonlinear optimal perturbation. Chinese Sci. Bull., 48, 1045-1047.

Mu, M., W. S. Duan and B. Wang, 2003. Conditional nonlinear optimal perturbation and its applications. Nonlinear Processes in Geophysics, 10, 493-501.

Mu M. and Z. Zhang, 2006. Conditional nonlinear optimal perturbations of a two-dimensional quasi-geostrophic model. J. Atmos. Sci., 63, 1587-1604.

Mullen, S. L. and D. P. Baumhefner, 1994. Monte Carlo simulation of explosive cyclogenesis. Mon. Wea. Rev., 122, 1548-1567.

Mullen S. L. and R. Buizza, 2001. Quantitative precipitation forecasts over the United States by the ECMWF Ensemble Prediction System. Mon. Wea. Rev., 129, 638–663.

Mullen, S. L. and R. Buizza, 2002. The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF ensemble prediction system. Wea. Forecasting, 17, 173-191.

Mylne, K. R., Evans, R. E., and Clark, R. T., 2002: Multi-model multi-analysis ensembles in quasi-operational medium-range forecasting. Quart. J. Roy. Meteor. Soc., 128, 361-384.

Mullen, S. L. and J. Du, 1994. Monte Carlo forecasts of explosive cyclogenesis with a limited-area, mesoscale model. Preprints, 10th Confrence on Numerical Weather Prediction, Portland, Oregon, July 18-22, 1994, Amer. Meteor. Soc., 638-640.

Mullen, S. L., J. Du and F. Sanders, 1999. The dependence of ensemble dispersion on analysis forecast system: implications to short-range ensemble forecasting of precipitation. Mon. Wea. Rev., 127, 1674-1686.

Mureau, R., F. Molteni, and T. N. Parmer, 1993: Ensemble Prediction Using dynamically conditioned perturbations. Quart. J. Roy. Meteor. Soc., 119, 299-323.

Murphy, A. H., 1969: On the “ranked probability score.” J. Appl. Meteor., 8, 988–989.

Murphy, A. H., 1971: A note on the ranked probability score. J. Appl. Meteor., 10, 155–156.

Murphy, A. H, 1985. Decision making and the value of forecasts in generalized model of the cost-loss ratio situation. Mon. Wea. Rev., 113, 362-369.

Mylne, K. R., Evans, R. E. and Clark, R. T., 2002. Multi-model multi-analysis ensembles in quasi-operational medium-range forecasting. Quart. J. Roy. Meteor. Soc., 128, 361-384.

National Research Council (NRC), 2006. Completing the forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts, National Academy Press, 112pp.

Nicolis, C., 2004. Dynamics of Model Error: The Role of Unresolved Scales Revisited. J. Atmos. Sci., 61, 1740–1753.

Nutter, P., D. Stensrud and M. Xue, 2004a. Effects of Coarsely Resolved and Temporally Interpolated Lateral Boundary Conditions on the Dispersion of Limited-Area Ensemble Forecasts. Mon. Wea. Rev., 132, 2358–2377.

Nutter, P., M. Xue and D. Stensrud, 2004b. Application of Lateral Boundary Condition Perturbations to Help Restore Dispersion in Limited-Area Ensemble Forecasts. Mon. Wea. Rev., 132, 2378–2390.

Oortwijn, J. and J. Barkmeijer, 1995. Perturbations that optimally trigger weather regimes. J. Atmos. Sci., 52, 3952-3944.

Ott, E., B. R. Hunt, I. Szunyogh, A. V. Zimin, E. J. Kostelich, M. Corazza, E. Kalnay, D. J. Patil and J. A. Yorke, 2004. A local ensemble Kalman filter for atmospheric data assimilation. Tellus, 56A, 415-428.

Palmer, T. N., R. Gelaro, J. Barkmeijer and R. Buizza, 1998. Singular vectors, metrics and adaptive observations. J. Atmos. Sci., 55, 633-653.

Palmer, T. and R. Hagedorn (edited), 2006. Predictability of weather and climate. Cambridge University Press, 718pp.

Pu Z.-X. and E. Kalnay, 1999. Targeting observation with the quasi-inverse linear and adjoint NCEP global models: Performance during FASTEX. Quart. J. Roy. Meteor. Soc., 125, 3329-3338.

Raftery, A.E., T. Gneiting, F. Balabdaoui and M. Polakowski, 2005. Using Bayesian Model Averaging to Calibrate Forecast Ensembles. Mon. Wea. Rev., 133, 1155–1174.

Reynolds, C.A. and R.M. Errico, 1999. Convergence of Singular Vectors toward Lyapunov Vectors. Mon. Wea. Rev., 127, 2309–2323.

Richardson, D. S., 2000. Skill and relative economic value of the ECMWF ensemble prediction system. Quart. J. Royal. Meteor. Soc., 126, 649-668.

Roebber, P.J., D.M. Schultz, B.A. Colle and D.J. Stensrud, 2004. Toward Improved Prediction: High-Resolution and Ensemble Modeling Systems in Operations. Wea. Forecasting, 19, 936–949.

Roulston, M. S., 2005. A comparison of predictors of the error of weather forecasts. Nonlinear Processes in Geophysics, 12, 1021-1032.

Roulston, M. S. and L. A. Smith, 2002. Evaluating probabilistic forecasts using information theory. Mon. Wea. Rev., 130, 3297-3319.

Rouston, M. S. and L. A. Smith, 2003. Combining dynamical and statistical ensembles. Tellus A, 55, 16–30.

Saha, S., S. Nadiga, C. Thiaw, J. Wang, W. Wang, Q. Zhang, H.M. Van den Dool, H.L. Pan, S. Moorthi, D. Behringer, D. Stokes, M. Peņa, S. Lord, G. White, W. Ebisuzaki, P. Peng and P. Xie, 2006. The NCEP Climate Forecast System. J. Climate, 19, 3483–3517.

Schaake, J., K. Franz, A.Bradley and R. Buizza, 2006: The Hydrologic Enemble Prediction Experiment (HEPEX). Hydrology and Earth System Science, hessd-2006-0095.

Shutts, G., 2004. A stochastic kinetic energy backscatter algorithm for use in ensemble prediction systems. Technical Memorandum 449, ECMWF.

Sloughter J. M., A. E. Raftery, T. Gneiting and C. Fraley, 2007. Probabilistic Quantitative Precipitation Forecasting Using Bayesian Model Averaging. Mon. Wea. Rev., 135, 3209-3220.

Smith, L. A. and J. A. Hansen, 2004. Extending the limits of ensemble forecast verification with the minimum spanning tree. Mon. Wea. Rev., 132, 1522-1528.

Stensrud, D. J, 2007. Parameterization schemes: keys to understanding numerical weather models. Cambridge University Press, 479pp.

Stensrud, D. J., J. W. Bao and T. T. Warner, 2000. Using initial condition and model physics perturbations in short-range ensemble. Mon. Wea. Rev., 128, 2077-2107.

Stensrud, D. J., H. E. Brooks, J. Du, M. S. Tracton and E. Rogers, 1999. Using ensembles for short-range forecasting. Mon. Wea. Rev., 127, 433-446.

Stensrud D. J. and N. Yussouf, 2003. Short-Range Ensemble Predictions of 2-m Temperature and Dewpoint Temperature over New England. Mon. Wea. Rev., 131, 2510-2524.

Stensrud, D. J. and N. Yussouf, 2007. Reliable Probabilistic Quantitative Precipitation Forecasts from a Short-Range Ensemble Forecasting System. Wea. Forecasting, 22, 3–17.

Stensrud, D. J., N. Yussouf, M. E. Baldwin, J. T. McQueen, J. Du, B. Zhou, B. Ferrier, G. Manikin, F. M. Ralph, J. M. Wilczak, A. B. White, I. Djlalova, J. W. Bao, R. J. Zamora, S. G. Benjamin, P. A. Miller, T. L. Smith, T. Smirnova and M. F. Barth, 2006. The New England High-Resolution Temperature Program. Bull. Amer. Meteor. Soc., 87, 491–498.

Sutton, C., T. M. Hamill and T. T. Warner, 2006. Will Perturbing Soil Moisture Improve Warm-Season Ensemble Forecasts? A Proof of Concept. Mon. Wea. Rev., 134, 3174–3189.

Szungyogh, I., E. J. Kostelich, G. Gyarmati, B. R. Hunt, A. V. Zimin, E. Kalnay, D. J. Patil and J. A. York, 2004. A local ensemble Kalman filter for the NCEP GFS model. AMS annual meeting, Seattle, WA, Jan. 11-15.

Szunyogh, I, Z. Toth, R. E. Moss, S. J. Majumdar, B. J. Etherton and C. H. Bishop, 2000. The effect of targeted dropsonde observation during the 1999 winter storm reconnaissance program. Mon. Wea. Rev., 128, 3520-3537.

Talagrand, O., R. Vautard and B. Strauss, 1997. Evaluation of probabilistic prediction systems. Proceedings, ECMWF Workshop on Predictability, ECMWF, 1–25. [Available from ECMWF, Shinfield Park, Reading, Berkshire RG2 9AX, United Kingdom.]

Teixeira, J. and C. Reynolds, 2006. The stochastic nature of physical parameterization in ensemble prediction: a stochastic convection approach. Geophysical Research Abstracts, 8, 01548, European Geosciences Union.

Teixeira, J., C. Reynolds and K. Judd, 2007. Time-step sensitivity of nonlinear atmospheric models: numerical convergence, truncation error growth and ensemble design. J. Atmos. Sci., 64, 175-189.

Tennant, W.J., Z. Toth and K.J. Rae, 2007. Application of the NCEP Ensemble Prediction System to Medium-Range Forecasting in South Africa: New Products, Benefits, and Challenges. Wea. Forecasting, 22, 18–35.

Thompson, P. D., 1957. Uncertainty of initial state as a factor in the predictability of large scale atmospheric flow patterns. Tellus, 9, 275-295.

Tippett, M. K., J. L. Anderson, C. H. Bishop, T. Hamill and J. S. Whitaker, 2003. Ensemble squared root filters. Mon. Wea. Rev., 131, 1485-1490.

Toth, Z. and E. Kalnay, 1993. Ensemble forecasting at NCEP: the generation of perturbations. Bull. Amer. Meteor. Soc., 74, 2317-2330.

Toth, Z. and E. Kalnay, 1997. Ensemble forecasting at NCEP: the breeding method. Mon. Wea. Rev., 125, 3297-3318.

Tracton M. S., J. Du, Z. Toth and H. Juang, 1998: Short-range ensemble forecasting (SREF) at NCEP/EMC. Preprints, 12th Conf. on Numerical Weather Prediction, Phoenix, Amer. Meteor. Soc., 269-272.

Tracton, M. S. and E. Kalnay, 1993. Ensemble forecasting at NMC: practical aspects. Wea. Forecasting, 8, 379-398.

Unger, D. A., 1985. A method to estimate the continuous ranked probability score. Preprints, Ninth Conf. on Probability and Statistics in Atmospheric Sciences, Virginia Beach, VA, Amer. Meteor. Soc., 206–213.

Vannitsem, S. and Z. Toth, 2002. Short-term dynamics of model errors. J. Atmos. Sci., 59, 2594-2604.

Wandishin, M.S., M.E. Baldwin, S.L. Mullen and J.V. Cortinas, 2005. Short-Range Ensemble Forecasts of Precipitation Type. Wea. Forecasting, 20, 609–626.

Wang, X. and C. H. Bishop, 2003. A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 1140-1158.

Wang, X., C. H. Bishop and S. J. Julier, 2004. Which is better, an ensemble of positive/negative pairs or a centered spherical simplex ensemble? Mon. Wea. Rev., 132, 1590-1605.

Wang, X., T.M. Hamill, J.S. Whitaker and C.H. Bishop, 2007. A Comparison of Hybrid Ensemble Transform Kalman Filter–Optimum Interpolation and Ensemble Square Root Filter Analysis Schemes. Mon. Wea. Rev., 135, 1055–1076.

Wang, Y. and A. Kann, 2005. ALADIN-LAEF (Limited Area Ensemble Forecasting) at ZAMG: Status and Plan. 15th ALADIN Workshop [available online: www.cnrm.meteo.fr/aladin/meetings/Wk2005/WANG.pdf].

Warner, T. T., R. A. Perterson and R. E. Treadon, 1997: A tutorial on lateral boundary conditions as a basis and potential serious limitation to regional numerical weather prediction. Bull. Amer. Meteor. Soc., 78, 2599-2617.

Warner, T. T., R-S. Sheu, J. F. Bowers, R. I. Sykes, G. C. Dodd and D. S. Henn, 2002. Ensemble simulations with coupled atmospheric dynamic and dispersion models: Illustrating uncertainties in dosage simulations. J. Appl. Meteor., 41, 488–504.

Wei, M., Z. Toth, R. Wobus and Y. Zhu, 2007. Initial perturbations based on the ensemble transform (ET) technique on the NCEP global operational forecast system. Tellus, (in press).

Wei, M., Z. Toth, R. Wobus, Y. Zhu, C. Bishop and X. Wang, 2006. Ensemble Transform Kalman Filter-based ensemble perturbations in an operational global prediction system at NCEP. Tellus, 58A, 28-44.

Weiss, S. J., J. S. Kain, D. R. Bright, J. J. Levit, G. W. Carbin, M. E. Pyle, Z. I. Janjic, B. S. Ferrier, J. Du, M. L. Weisman and M. Xue, 2007. The NOAA Hazardous Weather Testbed: collaborative testing of ensemble and convective-allowing WRF models and subsequent transfer to operations at the Storm Prediction Center. 22nd Conf. on Wea. Analysis and Forecasting and 18th Conf. on Numerical Wea. Prediction, Park City, UT, 6B.4.

Whittaker, J. S. and T. M. Hamill, 2002. Ensemble data assimilation without perturbed observation. Mon. Wea. Rev., 130, 1913-1924.

Whitaker, J. S. and A. F. Loughe, 1998. The relationship between ensemble spread and ensemble mean skill. Mon. Wea. Rev., 126, 3292-3302.

Wilks, D.S., 2004. The Minimum Spanning Tree Histogram as a Verification Tool for Multidimensional Ensemble Forecasts. Mon. Wea. Rev., 132, 1329–1340.

Wilks, D. S., 2006. Statistical methods in the atmospheric sciences. Academic Press, 627 pp.

Wilson, L.J., S. Beauregard, A.E. Raftery and R. Verret, 2007. Calibrated Surface Temperature Forecasts from the Canadian Ensemble Prediction System Using Bayesian Model Averaging. Mon. Wea. Rev., 135, 1364–1385.

Wobus, R. and E. Kalnay, 1995. Three years of operational prediction of forecast skill. Mon. Wea. Rev., 123, 2132-2148.

Woodcock, F. and C. Engel, 2005. Operational consensus forecasts, Wea. Forecasting, 20, 101-111.

Yang, S-C., M. Cai, E. Kalnay, M. Rienecker, G. Yuan and Z. Toth, 2006. ENSO bred vector in coupled ocean-atmospheric general circulation models. J. Climate, 19, 1422-1436.

Yuan, H., J. Du, J. McGinley, P. Schultz, B. Zhou, C. Lu, Z. Toth and G. DiMego, 2007a. Postprocessing of Precipitation Forecasts for New Configured NCEP Short-Range Ensemble Forecasting (SREF) System, 22nd WAF/18th NWP Conference, Park City, Utah, June 25-29, 2007, Amer. Meteor. Soc., 6B.2.

Yuan, H., X. Gao, S. L. Mullen, S. Sorooshian, J. Du and H. H Juang, 2007b. Calibration of Probabilistic Quantitative Precipitation Forecasts with an Articial Neural Network. Wea. and Forecasting (in press).

Yuan, H., S.L. Mullen, X. Gao, S. Sorooshian, J. Du and H.M.H. Juang, 2005. Verification of Probabilistic Quantitative Precipitation Forecasts over the Southwest United States during Winter 2002/03 by the RSM Ensemble System. Mon. Wea. Rev., 133, 279–294.

Yuan, H., S. L. Mullen, X. Gao, S. Sorooshian, J. Du and H. H Juang, 2007c. Short-Range Probabilistic Quantitative Precipitation Forecasts over the Southwest United States by the RSM Ensemble System. Mon. Wea. Rev., 135, 1685-1698.

Yussouf, N. and D. J. Stensrud, 2006. Prediction of Near-Surface Variables at Independent Locations from a Bias-Corrected Ensemble Forecasting System. Mon. Wea. Rev., 134, 3415-3424.

Yussouf, N., D.J. Stensrud and S. Lakshmivarahan, 2004. Cluster Analysis of Multimodel Ensemble Data over New England. Mon. Wea. Rev., 132, 2452–2462.

Zhang, F., 2005. Dynamics and Structure of Mesoscale Error Covariance of a Winter Cyclone Estimated through Short-Range Ensemble Forecasts. Mon. Wea. Rev., 133, 2876–2893.

Zhang, F., C. Snyder and J. Sun, 2004. Impacts of initial estimate and observation availability on convective-scale data assimilation with an ensemble Kalman filter. Mon. Wea. Rev., 132, 1238–53.

Zhang, Z. and T. N. Krishnamurti, 1999. A perturbation method for hurricane ensemble prediction. Mon. Wea. Rev., 127, 447-469.

Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002. The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 73-83.