Can financial models truly be dangerous? As complexity blossoms in the world’s financial systems, coping with irreducible modeling uncertainty is a key challenge to good decision making. Generations have wrestled with minimizing risk through portfolio design and diversification, but important threats persist. They stem from the inevitable possibility that our models are in fact wrong.
Jon Hill[1] reminds us of two well-known examples:
1998: Long-Term Capital Management (LTC) assumed correlations based on history were a stationary process (Gaussian copulas, no tail contagion, etc.). They weren’t, and the subsequent melt-down of their highly leveraged positions took LTCM down and almost the entire banking system with it.
2004-2007: Subprime mortgage bundlers assumed North American housing prices would keep rising forever, or at least be stable based on history. They didn’t and they weren’t. Massive global market meltdown resulted.
The U.S. Office of the Comptroller of the Currency[2] (OCC) defines model risk as “the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” Banking regulators have issued guidelines that models should be inventoried and validated, and that model risk warrants quantification and provision through reserves or margins.
Even the most carefully constructed models are subject to what actuaries call “parameter risk,” the irreducible uncertainty in a predicted distribution. Is parameter risk a species of model risk? The OCC thinks so: “An important outcome of effective [model risk management] is a banking organization’s demonstrated understanding of and accounting for such uncertainty. Accounting for model uncertainty can include applying … adjustments to model output.”
So what is to be done if uncertainty remains despite our best modeling efforts? There are in fact two practical responses within reach of model practitioners, management, and industry analysts. The first response is to quantify the uncertainty, and the second is to design a strategy for robust decision-making that is relatively insensitive to the potential model error.
Let’s call the model we use and believe the “baseline.” We suspect it might not be exactly right, meaning that the relative probabilities it assigns to potential outcomes can differ from reality. Our initial approach is to conduct an uncertainty audit, where we develop standard errors or confidence intervals. This is a fine first step, but we haven’t provided much actionable guidance to the decision maker.
The Bayesian approach is to modify the baseline by averaging (actually, mixing) over the model uncertainty. The modified model is then used to make decisions. In this way we treat uncertainty as another form of risk, and obtain concrete guidance for ranking decisions.
Since Ellsberg,[3] however, we know that most people are “ambiguity averse.” To them, uncertainty and risk are different phenomena and should not be conflated. The neo-Bayesian approach, exemplified by Gilboa & Schmeidler,[4] is to locate all plausible models sufficiently “near” the baseline. We then make decisions that are robust: The likely outcomes are least painful with respect to the worst case model in the plausible set. Specifically, one constructs utility functions to represent the value of decisions. Map each decision option to the plausible model that possesses the worst expected utility. Then choose the decision option that offers the best mapped utility, i.e., the maximin.*
The maximin approach requires a way to measure the plausibility of alternative models and a specific threshold value. One popular metric is relative entropy.[5],[6] If p(x) is the baseline model’s probability density and q(x) is the alternative, then the relative entropy is defined as the expected value of (q(x)/p(x)) × ln(q(x)/p(x)). If the plausible set is defined by a bound on relative entropy, the most adverse model for utility U(x) takes the form q(x)=C×exp(-k×U(x))×p(x). This expression is the familiar Esscher transform.
However, there are problems. The usual interpretation of entropy is information, and it is difficult to gain or communicate intuition about a plausibility bound denominated in bits. Also, plausible alternatives can include models putting zero or near-zero probability on scenarios that the baseline considers quite possible.
More fundamentally, there is the problem that one must choose a bound at all. It ends up being arbitrary, like the choice of VaR thresholds in capital requirement standards.
A solution to these problems is presented in our recently released working paper, “The Most Dangerous Model: A Natural Benchmark for Assessing Model Risk.”[7] If we know (or assume) the number of observations upon which the baseline model was built and treat it as if it were a perfect representation of the data, we can define plausibility in terms of the likelihood ratio. The interpretation is then straightforward: Alternative q is considered plausible if the underlying data is at least y% as likely to have been generated by q as by the baseline.
The worst plausible models are no longer defined by the Esscher transform, but the solution is just as easy to compute: q(x)=C×p(x)/(k+U(x)).
But there’s more! If we further assume that p and any alternative q were considered equally plausible before observing the empirical data, we have a 50-50 prior, which, when combined with the likelihood connection between p and q, permits computation of Bayesian posterior expected utility. Essentially, we compute a credibility-weighted average between the baseline model p and any (single) alternative q. The question then becomes: Which alternative q has the worst posterior expected utility? This is the worst credible model, a.k.a. the Most Dangerous Model (MDM). It is a naturally occurring object, not requiring the specification of an arbitrary plausibility threshold.
Finding the most dangerous model is a simple one-dimensional search problem, and it can be organized in a spreadsheet. When financial decisions rely on estimates marked to model, identifying the MDM can be a critical exercise in quantifying uncertainty and designing robust strategy.
Details and a reinsurance purchase case study can be found in the working paper.
References
[1] Hill: http://ev1053.eventive.incisivecms.co.uk/digital_assets/6243/Jon_Hill.pdf
[2] OCC: http://www.occ.treas.gov/news-issuances/bulletins/2011/bulletin-2011-12a.pdf
[3] Ellsberg: www.jstor.org/stable/1884324
[4] Gilboa & Schmeidler: http://www.sciencedirect.com/science/article/pii/0304406889900189
[5] Friedman: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1725561
[6] Glasserman & Xu: https://www0.gsb.columbia.edu/faculty/pglasserman/Other/RobustRiskMeasurement_published.pdf
[7] Major, Wang, & Woolstenhulme http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2611806
The authors work for Guy Carpenter & Company, LLC. John Major, ASA, is director of actuarial research and Micah Woolstenhulme, FCAS, is managing director of ERM advisory.
* The maximum of a set of minima.
