In the final installment of our three-part series, I will explore the practical aspects of dealing with model failures and the limitations of statistical tests. Understanding these elements is crucial for making the most of your models and maintaining their strategic value over time. I hope any leader will find value in this, regardless of familiarity with modeling and statistics, but buckle in: this is a nerdy one.
DON'T LET FAILED STATS TESTS SCUTTLE A USEFUL MODEL
During our reviews of banks' modeling and analytics suites, we often identify areas where rigid adherence to statistical testing impedes practical progress. While statistical tests aim to provide evidence that models and data behave in specific ways, their practical utility can be limited.
Academically, these tests are important. However, in real-world applications, they can sometimes be more of a hindrance than a help. When choosing between a less useful but “clean” model and a more useful model with some statistical imperfections, it’s often better to collaborate with Model Risk Management to get the valuable model into production quickly.
For those who are particular about statistics, consider this: many models fail due to minor issues like non-normal residuals or insufficient observations to prove cointegration between deposit and treasury rates. Additionally, not segmenting a mortgage application conversion model by key business segments because it’s only 92% statistically significant is not always a valid reason to reject it. These examples illustrate why practical utility should sometimes outweigh strict statistical adherence.
ALL YOUR MODELS WILL FAIL. PLAN FOR IT.
Models inherently assume that unrecognized drivers will remain constant, which is rarely the case in dynamic business environments. Significant changes, such as the strategic shifts businesses underwent due to COVID-19, can render pre-2020 trained models ineffective today.
To address this, we plan in two ways:
- Robust Model Development: Strive to include as many important drivers as possible during model creation. If a modeler adds a “Q3 2022 flag,” it indicates an unexplained event impacting performance. Investing the effort to understand such anomalies before deployment ensures the model is more responsive to future occurrences.
- Proactive Model Management: Recognize that models start to degrade the day they are implemented. Establish a proactive plan for evaluating model performance drift, set triggers for full redevelopment, and develop tactics to keep the model useful until its next overhaul.
THE STRATEGIC VALUE OF CLEAR BUSINESS INSIGHTS
While models are excellent for predictive decision-making, their strategic value significantly relies on clear business insights. At Nomis, our team leverages over 20 years of industry experience combined with advanced analytics to navigate the limitations of models in dynamic environments. We work closely with our customers to adapt strategies, ensuring sustained relevance and competitive performance.
In this series, we’ve examined the realities of business decision-making without complex models, the importance of asking the right questions, and the power of integrating business insights into modeling. By understanding the limitations of statistical tests and proactively planning for model failures, businesses can maximize the strategic value of their analytics.
For a comprehensive exploration of these insights and to learn more about how innovative analytics can redefine your business strategies, visit Nomis Solutions.
About the Author:
Wes West is the Chief Analytics Officer at Nomis Solutions, the foremost provider of end-to-end pricing analytics and execution technology. With a distinguished career at leading financial institutions, West brings unparalleled expertise and a track record of innovation to his role at Nomis. His extensive experience spans Retail, Strategy, Finance, and Treasury, offering unique insights into the world of financial analytics and strategic pricing. Connect with him on LinkedIn or reach out via email at wes.west@nomissolutions.com.
