The Global Tropics Hazards and Benefits Outlook product verification is below, segmented into four sections (1) the most recent forecast verification, (2) long term rainfall area verification from 2010 to the present, (3) long term tropical cyclone verification from 2007-2012 and (4) an appendix where details of the verification procedures are described.
Most Recent Week-1 and Week-2 Verification
The four panel plots below show verification for the most recent Week-1 (left) and Week-2 (right) forecasts available. These plots are updated daily and will change as the observational data acculumlate each day as the current 7-day proceeds. The verifying period is from 0000 UTC Wednesday through 2359 Tuesday. The top panel is the percentile for weekly average OLR, top middle panel is the categorization of the weekly OLR data into one of the three categories (above-, near- or below-average), the bottom middle panel is the gridded GTH forecast shapes (above or below average rainfall forecast) and the bottom panel is verification of each grid point of highlighted shapes where green is a "hit" and red is a "miss". Heidke skill score is also provided. See Appendix for additional details.
Longterm Rainfall Area Verification
Verification of areas of enhanced/suppressed rainfall are possible by gridding outlook areas using GIS utilities and comparing with Outgoing Longwave Radiation (OLR) data. Tercile cutoffs were defined and wet (dry) area forecasts are correct if the weekly average OLR resides in the appropriate above (below) tercile categories respectively. See appendix for additional details. The plots below show a time series of Heidke skill scores for Week-1 and Week-2 outlooks from 2010 to the present. The first row are skill scores for areas in which a forecast was made only, while the second row are scores adjusted by forecast coverage (typically 10-20% across the global Tropics). These scores are much lower for this reason but also allow more equal comparison and limit the volatility in cases where coverage is very low. Positive values indicate skillful forecasts and value added and scores of zero and negative indicate no forecast skill. A four week running average (green) is applied to the weekly scores (thin blue) and horizontal lines show the average over the entire period (red), boreal winter months only (thick blue), boreal summer months only (maroon) and for active MJO periods (yellow). No forecast skill is shown as a horizontal black line.
Heidke skill scores as a function of geography over the period are also shown.
Tropical Cyclone Area Verification
Tropical cyclone verification is given below. A tropical cyclone must form during the weekly period in the region highlighted to be considered a correct forecast. Existing cyclones at time of forecast are not included in these statistics. Refer to the 2x2 contingency table below in Appendix for explanations.
1. Rainfall hazard areas:
Additional details for the rainfall verification procedure are given here. Outgoing Longwave Radiation (OLR) data are used since OLR is a generally good proxy for rainfall in the Tropics and also since there is a reliable, consistent and lengthy base period record in which to determine anomalies. The OLR data is 1x1 and weekly averages are calculated at each grid point for all julian days over the 30 year period of record. OLR value limits are then determined for a number of percentiles, including tercile cutoffs of 33% and 66%. The realtime weekly mean OLR data is then categoried into one of three categories above-, near- or below-average weekly mean OLR at each grid point. Weekly mean values below or equal to the 33% cutoff would verify as wet and those greater than or equal to the 66% cutoff would verify as dry. Only grid points in which a forecast was made (i.e. a green or yellow area designated in the outlook) are verified. As an example, forecasts that indicated yellow on the outlook map would be credited with a "hit" if the corresponding observations also ranked in the upper third of the historical distribution, while those grid points that ranked in the middle or lower part of the distribution were credited as a "miss". Vice versa for green forecasted areas.
The Heidke skill score (HSS) is a hit-based skill score and is calculated through the following relationship that compares the number of correct and incorrect hits along with the number of expected hits from random chance:
HSS (%) = 100 * (H - E) / (T - E)
where H = Number of correct forecasts, E = Expected number of correct forecasts (1/3 of total), and T = total number of forecasts.
The HSS is a measure of how well a forecast did relative to a randomly selected forecast. A score of 0 means that the forecast did no better than what would be expected by chance. A score of 100 depicts a "perfect" forecast and a score of -50 depicts the "worst possible" forecast. The second set of time series plots are obtained by simply multiplying the forecast coverage by the above skill score value which weights more heavily outlooks with greater forecast coverage.
2. Tropical cyclone hazard areas:
Using the 2x2 contingency table below, the "Hit Rate" is the proportion of correct "yes" and correct "no" forecasts: (a+d)/n. The "Success Ratio" is the proportion of correct "yes" forecasts: a/(a+b). The "Critcal Success Index" or threat score is a/(a+b+c). The Probability of Detection (POD) and False Alarm Rate (FAR) are calculated by a/(a+c) and b/(a+b) respectively. The total number of forecasts, n, is given by n=(a+b+c+d).
The HSS for the tropical cyclone outlooks is calculated based on the 2x2 contingency table with the following formula:
HSS = 2*(ad-bc) / [(a+c)(c+d) + (a+b)(b+d)]