And there’s another, very troubling observation that modellers will be familiar with. If you keep adding possible variables into the mix for your historical testing, you tend to get a better and better “fit” to past data, but worse and worse predictive power for the future.
Some of the best predictive models turn out to be simple (with perhaps only one or two variables included), even though the historical “fit” is not that good.
Good predictive power tends to come where the modellers are genuinely on top of the theory as well as the data. So, for example, modellers of airflow over an aircraft wing will build a model to predict what angle and speed combination will induce a stall in an aircraft, and these models are amazingly reliable (thank goodness).
But that’s because the modellers really understand the limited influences on airflow over a wing, and include only those variables which are genuinely relevant. This is not a complex system.
But if the system that the forecaster is modelling is complex – by which I mean has unknowable numbers of influences and information, and/or randomness inherent in it – then forecasting models become little better than guesses.
And typically the forecast that models produce in these circumstances often reflect the interests and biases of the forecasters themselves.
Examples of complex systems include: modern economies; stock market and currency prices; as well as the climate. In modelling systems like these, the modeller will never be able to accurately reflect what is going on.
Let me give you a real-life example. There are numerous models of the climate. These use historical data of the many influences on the climate: CO2 concentration; methane concentration; solar activity; water vapour; cloud cover; aerosols and so on.
But most climate modellers work for institutions that are concerned about global warming under the influence of human-induced CO2 emissions. Hence modellers are under subtle (and sometimes not-so-subtle) pressure to produce forecasts of future warming that look alarming.
