Quantifying uncertainties of Machine Learning models is crucial to improve their reliability, accurately assess risks and make more robust decisions. By quantifying and understanding uncertainty, we can build more reliable and trustworthy systems.

Imagine we have a model that predicts whether or not a CT scan contains a tumour: traditional approaches tend to provide binary predictions, while not providing information on the model’s confidence in each prediction.

Conformal Prediction (CP) is a framework for uncertainty quantification that offers an estimate of the confidence in the model’s predictions: instead of providing just a point estimate, it provides a set of possible outcomes (prediction set), together with a measure of confidence in each outcome. These prediction sets come with a (mathematical!) guarantee of coverage of the true outcome, ensuring that they will detect at least a pre-fixed percentage of true values. CP is a model-agnostic paradigm, requiring no retraining of the model and making no major assumptions about the distribution of the data.

We Humans, when faced with uncertainty, tend to express indecision and offer alternatives. We will see that CP can be a key tool to include a human in the decision-making loop, once the ‘humanised’ machine is able to express its uncertainty.

CP therefore offers a robust framework that allows stakeholders to make more informed decisions, even more so in high-risk sectors such as healthcare, finance and autonomous systems.