The best forecast

My interest in forecast evaluation (saying how good forecasts are) goes back to about 1998 when a ginormous El Nino was threatening California and Arizona with floods. My Masters thesis was on how water providers and emergency managers used those forecasts from September to say what might happen that coming winter.

I did long interviews with key people in Arizona. I went some strange places, particularly emergency management offices. I saw the "big board" at the state emergency center. They really do have black helicopters at the Phoenix bunker (yes, a bunker with zigzagged hallways set up for nuclear explosions).

Before the start of the winter, a late season Pacific hurricane came up the west cost and passed through Yuma and California/Arizona border. That hurricane, and the images of the raging floods during the most-recent-ginormous-El-Nino in 1983, were enough to put the fear of god in everybody. Seriously, I'd go into flood managers' offices and at reception they'd have a massive photo of roiling waves and houses washing away from 1983- you would think that would be fresh in their mind. It wasn't this picture, but this was the event:

Anyhow, you didn't want to be that guy that everybody warned but you didn't do anything and then it happened and jeepers, what do we even pay you for anyway?!?

After the event, many people were wondering, there's small and medium El Ninos going on all the time, is this kind of warning something we could use all the time? How good are the forecasts? Will they ever bite us?

Well, it turns out, to a user, "how good are they" is a very complicated question that depends on where you are, what you do, how much risk you can handle and a host of other things. But for a forecaster, interestingly, there's only a few ways to be right and a few things to strive for to be good.

In my opinion, the best writer on "the goodness of forecasts" was Allan Murphy. I never met him, but in grad school my copy of his collected works was dog-eared and tattered. I hope to weave some of his ideas into my work this year.

Tom