In weather prediction, forecast models are used to predict future states of the atmosphere, based on how the climate system evolves with time from an initial state. While the forecast models are quite complex (and do require strong scientific and engineering capabilities), the science of analyzing those forecast models, and verifying their accuracy, by comparing actual empirical observations to predicted values, is quite straightforward.

For the domain of Air Quality, just like for weather prediction, it is possible to define models used to predict the future set of atmospheric pollution. There are actually plenty of such models, often referred as Atmospheric Dispersion Modeling. And just like weather prediction, the same concept of accuracy analysis can be applied to Atmospheric pollution predictions. This article is the first of a series on Air Quality forecasting.
--

PM_2.5 air pollution forecast is already available on the World Air Quality Index project for Asia, Europe as well as the whole world. But the data used to feed the forecast models is mostly based on satellite observation (see this article) rather than terrestrial stations readings. Using satellite data has the advantage of being able to cover any part of the globe, including oceans, provided there are no cloud. But, on the other hand, satellite data is also inherently less accurate, and only available twice a day, compared to 24 times (every hour) for terrestrial observations. Considering the dynamics of Air Pollution in Asia, having only two readings a day might introduced a significant true forecast error in the prediction, following Rosanne Cole's classification^[2]:

An observed forecast error may contain data errors of two kinds: (1) measurement errors in the data used to construct the forecast and (2) measurement error in the realized value. Data errors of the first kind will be a component of the true forecast error

Error of type 2 are related to the dispersion model used for the forecast. Since different models are used for the different countries and continents (this is currently the case for the World Air Quality Index project), the accuracy analysis needs to be done for each of the model. So, to start with, this article will focus on the model used for the Asian continent. In later post, we will extend the analysis to more continents.
--

Back to the initial question about the forecast accuracy, one last item to be considered in the analysis in the how far in advance the forecast in computer. The less in the advance, the more accurate the model is likely to be. So, just to start with, the following analysis graphs are based on "day +1" forecast (e.g. forecast for the next day, or if you are a Tuesday, then the forecast is for the Wednesday).

There are several ways of representing the accuracy, the most obvious one being a simple number representing percentage of forecast matching the actual observation. But because we do believe that graphic visualization are much more powerful than numbers, we prefer to present the superposed forecast/observation matching for several cities in Asia. The squares at the top are the empirical observations and the one at the bottom the foretasted values.


By checking all the graphs below, one can notice quite disappointing results for Guangzhou, Chengdu and South Korea... to the extent that the model in use for Asia could almost be disqualified for public usage. This is something that we will be writing in the second post of this series on Air Quality Forecasting.


--

Forecast advance: Day +1 Day +2 Day +3 Day +4 Day +5


--

Some interesting links for those interested to read about about forecast accuracy:

• [1] Accuracy of three major weather forecasting services - by Randy Olson
• [2] Data Errors and Forecasting Accuracy (pdf) - by Rosanne Cole
• [3] How Accurate is the Weather Forecast? - by Mike Hammer