"Superforecasting : the art and science of Prediction", by Dan Gardner and Philip Tetlock

Hi Steemians!

This week I read the book "Superforecasting: The Art and Science of Prediction", written by Philip Tetlock and Dan Gardner in 2015. After a careful reading, I thought that reading could provide useful lessons in our uncertain environment.                       

General overview 

In our modern societies, the forecasts hold a central place. It is rare to consider the future without having previously projected the data from the past into a prospective model. They impact all our strategies on a daily basis: from the most mundane questions to the most elaborate. We rely on the opinion of the experts, and we are sometimes reluctant to question the word of these modern prophets. 

However, a large number of people give and follow forecasts that are too vague and difficult to verify. And this for one reason: The field of foresight is sorely lacking rigorous evaluations. The forecasts involve dangers and benefits for its followers. 

It is difficult to understand the results of a study and to be able to analyze them correctly without relying on quantitative data or assumptions with clearly defined terms. A good forecaster, as described in this book, is rigorous in its method of assessment and he is able to consider a multitude of perspectives. 

What can be learned from this book?  

Philip Tetlock, the author of this book, conducted experiments whose results seem curious: the majority of the predictions he evaluated were no more relevant than those obtained by the fruit of chance. This leads to questioning the relevance of the forecast: are the forecasts really reliable?  

The author avoids caricature leading to fallacies. The future is not necessarily a mystery or the fruit of chance. There is a close correlation between the temporal length of the prediction and the error rate of the forecast. So the more you try to see far back in time, the more you risk being wrong. The forecasts must be studied with great rigour, while considering the possible margin of error. 

In addition, forecasting can be a key factor in carrying out a strategy. People's thinking skills are based on key information. The forecasts underline the trajectories to follow for the individuals. This is the major risk: these trajectories can mislead them, if they are followed blindly. In response to the uncertainties of the forecast, we can only check them by controlling them and constantly refining them. 

Indeed, the accuracy of the forecasts is often neglected by the clocked pace of information. And it is common to see predictions become acquired trajectories. However, forecast are objectives (or warnings), and not facts acquired. We are taught by the authors that we need to be lucid about the possible margin of error. This goes through a more methodical approach and an increased requirement in the assessment of the forecasts. 

The conclusions of Dan Gardner and Philip Tetlock present two types of forecasts:  

• The first category includes forecasts guided by an oriented ideology. It is based on particular assumptions and information. Nevertheless, the results of these forecasts tend to be biased by this oriented ideology.  
• The second category brings together forecasts based on a variety of sources and information. Their assessments are more nuanced, and their results can be counter-intuitive compared to the oriented ideology. However, they are less accessible to the public.

The control must take into account the methods of the forecasts, but also their results. A more nuanced forecast should not justify the weakness of these results in its primary role: to assess the expectation of an event.  

In fact, the probability calculation is paramount in the forecasting processes. Thinking in mathematical terms requires above all precision and rigour. The accuracy of the figures is directly associated with the fineness of the judgment: an external point of view can bring elements that can significantly improve the initial forecast. As mentioned by the authors, accuracy is strongly correlated with precision.   

Discussion about this book 

In an uncertain environment, the information became the new gold. They give a certain advantage to businesses, and some influence to its holders. We rely blindly on the opinion of experts at the collective level. And this lack of clarity can cause serious damage as recent economic crises demonstrate. 

When the teachings of this book are brought closer to new practices such as "Lean management", the importance of evaluation should be emphasized in order to continually improve forecasting errors. This error factor must be taken into account in the forecasting method. 

 These authors consider that it is important to consider all the factors that can influence the predictions before and during the evaluation. Thus, the evaluation of the forecasts is dynamic, in order to be able to take into account the changes of the environment. Otherwise, it will be very difficult to measure the impact without being fooled by unconsidered external factors. 

Predictions are the image of a future projection in relation to a particular position. However, we are more able to defend measures based on an ideological position, than uncertain but more relevant solution. The cognitive biases explain how some forecasts are biased by oriented ideological assumptions. 

A particular attention in reading the forecasts must be taken by the follower. For example, a probability of 90% for an event implies that the second option have 10% of chance to be realized: the second part of the information is too often forgotten in order to highlight the first part. Then, how to know the relevance of a forecast? 

You have to know how to take the forecast with tweezers. Depending on the field and the situation, the forecast is more or less difficult: the accuracy of a meteorologist and a political analyst cannot be compared. And it is not because the weather forecast for the evening is wrong for the next day that all weather forecast must be rejected. We are in a constantly evolving environment where the slightest change in a variable of the initial model can affect the final forecast. Forecasters need to be prepared to give up a forecast for its re-evaluation. 

The authors show us the need to take several different angles to analyze a forecast. However, it seems that Philip Tetlock and Dan Gardner have forgotten to consider the Simpson paradox: trends observed on different samples can reverse on the macro-observation of all the samples. Thus, the methodological framework of the forecast should be kept in mind when evaluating the latter. 

Indeed, the first idea strongly marks the spirit: it is rarely radically diverted. Other angles of analysis nevertheless allow to deconstruct ideological biases to detect the weaknesses of the forecast. We never finished learning and digging his own reasoning to improve his method and predictions. 

It seems that no forecast is definitive. A good forecaster must have the audacity to update his assessment when new information comes to him. The awareness of uncertainty is a mark of lucidity indispensable for the success of a strategy based on predictions. 

And in a response to an uncertain and irregular environment, this lucidity is a major advantage to knowing how to react quickly and making decisions without procrastination, and sticking to it, unless circumstances force us to change our plan.  

And you? Did you read this book? What do you think about forecast, experts and other predictions?