Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don’t much care where.
The Cheshire Cat: Then it doesn’t much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you’re sure to do that, if only you walk long enough.
— Lewis Carroll, Alice in Wonderland
Model risk arises when a financial institution is highly dependent on internally developed or vendor models to perform certain critical functions. For many financial institutions, though, model risk management is sometimes analogous to Alice trying to make her way through Wonderland – they know they need to do something, but they are oftentimes unsure of the specifics. And, even more importantly, they don’t know the ultimate destination. So while they know they’ll end up somewhere, it just may not be the right or best place to get to. And unlike Alice, who ultimately wandered around aimlessly without deleterious consequence, financial institution boards, management, and examiners are unlikely to be as patient and understanding when the models end up going astray.
There are two specific areas of model risk management that are critical to banks and of heightened focus for financial institution regulators – anti-money laundering compliance and loan credit risk management. Models designed to estimate and quantify customer risk or transactional risk are the essential tools used for satisfying the requirements of the Bank Secrecy Act (BSA)/Anti-Money Laundering (AML) provisions. Similarly, credit risk models drive the fundamental assessment of the bank’s loan portfolio. As a result of pending changes to accounting standards, financial institutions are in the process of developing new quantitative credit models for estimating Current Expected Credit Losses (CECL).
Yet, in both of these areas, there is a lack of clear guidance and an abundance regulatory scrutiny. In general, the BSA/AML class of models represents an example of a mature class of models within a bank’s model inventory, while CECL implementation is expected to require a significant model development effort. We summarize the lessons learned from our experience with BSA/AML applications and loan credit modeling to offer recommendations to practitioners. Accordingly, herein, we provide an overview of the model risk management guidance and make several practical, implementable, and simple recommendations that can help any financial institution’s model risk management program from slipping down a never-ending regulatory rabbit hole.
In 2011, the Office of the Comptroller of the Currency (OCC) together with the Board of Governors of the Federal Reserve (Federal Reserve) released the Supervisory Guidance on Model Risk Management. The guidance was designed to aid financial institutions by describing the elements of a sound model risk management program. In this framework, a model is defined as: “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” Specifically, the guidance aims to help ensure that financial institutions appropriately develop, implement, validate, update, calibrate, optimize, document, assess, and report on their models.
Whether a model is designed to assess operational risk (e.g., BSA/AML), estimate Credit Risk (e.g., CECL), or measure some other type of risk, the key tenants of a sound governance framework include the following:
Within the BSA/AML framework, the use of risk management models is increasing. Banks are becoming more reliant on these models to identify higher-risk activity and support key business decisions. Some of these BSA/AML-related models commonly used by financial institutions include the following:
There are many benefits of having effective BSA/AML models in place. For example, effective models can help identify, minimize, and mitigate risk, increase productivity and efficiency, improve customer due diligence processes, address elevated risks proactively, reduce staffing costs by automating solutions, help avoid fines and penalties, and improve business decisions.
When a bank designs its AML models, it should leverage heavily from its BSA/AML Risk Assessment to identify and inventory its AML risks. At a high level, there are four main sources of risks:
Once a bank develops a structured approach to identify risks, it must then determine appropriate testing and define the parameters that will comprehensively validate the models. Broadly speaking, there are four components to validation: conceptual design, system, data, and process.
The AML model validation space is increasingly being scrutinized by regulators. Banks may face fines or other penalties from regulators if their AML model validation is not properly executed. Censure from regulators may occur if a bank’s model validation does not have appropriate testing or parameters, such as if it uses a conceptual framework that is inconsistent with regulatory expectations, a logic or methodology that is not commensurate with the risks of the organization, a reliance on unidentified models, inaccurate customer or transaction risk measurement, fundamental logic errors, or an inconsistent approach/lack of detailed documentation to support model risk management activities.
Banks may also get in trouble over model management, such as unclear lines of authority or a lack of appropriate resources deemed necessary to effectively manage risk activities. Regulators also focus on whether a model is up to date, and the failure of a model to identify changes in an organization’s activities that have an impact on risk can be a large issue. Examples of institutions that did not have proper AML models in place include:
To avoid running afoul of regulators, banks should ensure that their AML models and validation efforts have the proper guardrails in place. First and foremost, banks should involve senior management in AML compliance programs and ensure that experienced and independent personnel conduct model validation. The models should have adequate internal controls designed to ensure compliance with the BSA and its implementing regulations. Banks should also utilize a risk-based approach in designing and implementing AML models to properly focus resources and efforts commensurate to the level of risk posed. To accelerate their validation projects, they should identify frequent deficiencies and make use of regulator findings with reference to a model’s validation, which will help predict the examiners’ expectations. Finally, banks should have a plan documented for the validation process by defining what needs to be accomplished through the model validation effort. This documentation should demonstrate that all model validation policies and procedures have been completed successfully.
Following the global economic crisis, the Financial Accounting Standards Board (FASB) began to revisit how banks estimate credit risk in the allowance for loan and lease losses (ALLL) calculation. The legacy accounting framework is an impairment- based concept that is largely driven by incurred losses. Under this approach, loans are recognized as impaired when management determines that it is probable that future cash flows will be not be collected in full as per the contracted terms. This approach will be replaced by the new CECL framework in which credit losses are estimated for the life of the loan at origination and for subsequent reporting periods. In contrast to the ALLL system where credit losses are recorded based on a probable impairment criterion, the CECL approach mandates that financial institutions develop a forward-looking estimate for credit losses for all loans and covered financial assets irrespective of any adverse change in loan performance or fundamental financial metrics. As the CECL implementation will in many cases require the development or onboarding of new credit risk models, we offer recommendations for successful implementation by keeping the requirements of validation front of mind during model development. In sum, the core purpose of a CECL model is to generate estimated losses on the loan portfolio (output) using relevant sources of data (inputs), including bank historical loss experience, peer/industry loss history, loan risk metrics, national and regional economic forecasts, and interest rate and credit market expectations.
The first step in the model process involves an introspective review of the institution’s analytical modelling capabilities and the existing credit risk management culture. The scope of the CECL model development project is expected to vary considerably across the spectrum of financial institutions consisting of small community banks, regional banks, super-regional banks and complex or specialized banks, and large national bank platforms. The smallest institutions are expected to utilize third-party software vendors or consultants, and they typically do not have an internal model validation staff. At the other end of the spectrum, large national banks will be able to leverage certain data, systems, and processes from internally developed CCAR/DFAST predictive stress testing models. While regulators have signaled that CECL expectations will vary according to bank size and complexity, they note that there is no bright line to delineate smaller or less complex banks. Similarly, CECL does not prescribe specific model approaches or methodologies, nor does the guidance set expectations or ranges for the model output (expected credit losses) or the resulting change in the allowance levels from the ALLL framework.
Taking inventory of a bank’s resources involves recognizing that establishing a strong model risk management framework requires a diverse set of skills. In the case of CECL implementation, this involves quantitative model fluency and programming skills, fundamental credit risk management, project management capabilities, regulatory expertise, loan product specialization, technical writing and documentation skills, and internal audit and testing design experience.
As part of this internal assessment, banks should evaluate the current culture of model risk management at the institution by addressing the following questions:
The challenges posed by CECL can also represent an opportunity to integrate model risk management considerations into the model development process. In a broader sense, the CECL design and implementation process may provide the impetus for growing regional banks to align credit risk management with related internal accounting and market risk functions to develop a robust enterprise risk platform. We recommend incorporating validation requirements into the de novo model design to ensure that the CECL model not only serves its core purpose, but that the model can be seamlessly integrated into a model risk management framework. The starting point to this process is to allow the model validation team/stakeholders to participate in the design project. This facilitates communication throughout the process so that the validation team is kept apprised of developments during model creation and the design team is cognizant of how the model will be documented and tested.
Additional recommendations for collaboration include the following:
Finally, below we highlight common pitfalls to avoid as applied to CECL model implementation.
Much like the Cheshire Cat helped guide Alice in Wonderland, financial institutions can and should leverage third-party independent consultants to perform reviews of their models.
Third-party consultants and external model validation resources can act to facilitate informal reviews and comment prior to a formal (pass/fail) test or regulatory review. External consultants can provide fresh insights gathered from a variety of projects serving multiple peer institutions. Consultants can also contribute skills and experiences obtained from multiple functional roles, including business users, model developers, model reviewers, and C-Suite or management oversight.
From an operational perspective, consultants can provide flexibility with staffing and resource constraints to meet internally and externally imposed deadlines. Third-party advisors who have experience with multiple vendor systems can assist with the vendor-selection process and evaluate third-party models and sources of data. Finally, external validators are independent from the banks and work outside of the functional units within the institutions (e.g., internal auditing, model risk management, external auditing). In this vein, independent consultants can offer valuable insights to management, boards of directors, and shareholders. And, of course, third-party consultants and independent model testing and validation can help keep financial institution employees and their regulators from going mad.
Stout Analyst Lauren Rosenberg contributed to the development of this article.