Alice: Would you tell me, please, which way I ought to go from here?

The Cheshire Cat: That depends a good deal on where you want to get to.

Alice: I don’t much care where.

The Cheshire Cat: Then it doesn’t much matter which way you go.

Alice: ...So long as I get somewhere.

The Cheshire Cat: Oh, you’re sure to do that, if only you walk long enough.

— Lewis Carroll, Alice in Wonderland

Model risk arises when a financial institution is highly dependent on internally developed or vendor models to perform certain critical functions. For many financial institutions, though, model risk management is sometimes analogous to Alice trying to make her way through Wonderland – they know they need to do something, but they are oftentimes unsure of the specifics. And, even more importantly, they don’t know the ultimate destination. So while they know they’ll end up somewhere, it just may not be the right or best place to get to. And unlike Alice, who ultimately wandered around aimlessly without deleterious consequence, financial institution boards, management, and examiners are unlikely to be as patient and understanding when the models end up going astray.

There are two specific areas of model risk management that are critical to banks and of heightened focus for financial institution regulators – anti-money laundering compliance and loan credit risk management. Models designed to estimate and quantify customer risk or transactional risk are the essential tools used for satisfying the requirements of the Bank Secrecy Act (BSA)/Anti-Money Laundering (AML) provisions. Similarly, credit risk models drive the fundamental assessment of the bank’s loan portfolio. As a result of pending changes to accounting standards, financial institutions are in the process of developing new quantitative credit models for estimating Current Expected Credit Losses (CECL).

Yet, in both of these areas, there is a lack of clear guidance and an abundance regulatory scrutiny. In general, the BSA/AML class of models represents an example of a mature class of models within a bank’s model inventory, while CECL implementation is expected to require a significant model development effort. We summarize the lessons learned from our experience with BSA/AML applications and loan credit modeling to offer recommendations to practitioners. Accordingly, herein, we provide an overview of the model risk management guidance and make several practical, implementable, and simple recommendations that can help any financial institution’s model risk management program from slipping down a never-ending regulatory rabbit hole.

Model Risk Management Regulatory Background

In 2011, the Office of the Comptroller of the Currency (OCC) together with the Board of Governors of the Federal Reserve (Federal Reserve) released the Supervisory Guidance on Model Risk Management.^[1] The guidance was designed to aid financial institutions by describing the elements of a sound model risk management program. In this framework, a model is defined as: “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.”^[2] Specifically, the guidance aims to help ensure that financial institutions appropriately develop, implement, validate, update, calibrate, optimize, document, assess, and report on their models.

Whether a model is designed to assess operational risk (e.g., BSA/AML), estimate Credit Risk (e.g., CECL), or measure some other type of risk, the key tenants of a sound governance framework include the following:

• The model must have a clear purpose and be aligned with its intended use.
• The model’s design, theory, and logic should be well-documented and supported by research and industry practice.
• There should be a rigorous assessment of data quality and relevance.
• There should be detailed and regular testing of the model to help ensure that the model is operating as intended.
• The model must be validated on a periodic basis.
• The model validation must be performed by a party that is independent from those who developed and/or use the model.
• The validation must be performed by those who have the requisite knowledge, skill, and expertise.
• Those who perform the validation must have the authority to challenge the developers and model users.
• The board and senior management have the ultimate responsibility over the development and oversight of model risk management, which includes helping to ensure that there are adequate policies and procedures, defined roles and responsibilities, internal audit involvement, involvement of independent resources, a model inventory, and sufficient documentation.

Applications for Anti-Money Laundering

Within the BSA/AML framework, the use of risk management models is increasing. Banks are becoming more reliant on these models to identify higher-risk activity and support key business decisions.^[3] Some of these BSA/AML-related models commonly used by financial institutions include the following:

• Customer onboarding and retention
• Customer risk rating
• Monitoring transactions for potentially suspicious activity
• Sanctions violation detection

There are many benefits of having effective BSA/AML models in place. For example, effective models can help identify, minimize, and mitigate risk, increase productivity and efficiency, improve customer due diligence processes, address elevated risks proactively, reduce staffing costs by automating solutions, help avoid fines and penalties, and improve business decisions.

When a bank designs its AML models, it should leverage heavily from its BSA/AML Risk Assessment to identify and inventory its AML risks. At a high level, there are four main sources of risks:

• Customer risk – includes those individuals who are on the sanctions list, politically exposed persons, and entities that are in higher-risk lines of business
• Products/Services risk – includes identifying products and services offered by the institution that have a higher risk of misuse for criminal activity, including private banking, wire transfers, and trade finance
• Geographical risk – refers to locations such as sanctioned countries, countries with high rates of corruption, countries with weak AML laws and enforcement, and domestic areas such as border areas, high-intensity drug trafficking areas, and high-intensity financial crime areas
• Regulatory risk – encompasses risk of sanctions, lawsuits, fines/penalties/settlements, enforcement actions, loss of license, and imprisonment

Once a bank develops a structured approach to identify risks, it must then determine appropriate testing and define the parameters that will comprehensively validate the models. Broadly speaking, there are four components to validation: conceptual design, system, data, and process.

• Conceptual design validation – evaluates the logic and design of the model. This is done by reviewing relevant bank documentation of the model, including policies and procedures, results of prior testing and validation, and organizational charts. Banks need to interview key personnel, analyze and test underlying model assumptions to establish validity, review access controls, and perform a qualitative analysis of the current rule set.
• System validation – ensures that the system is properly designed to perform. Both the conceptual and technical designs of the system should be tested to ensure that they are adequate in mitigating AML risks. Banks should review recent risk assessments, determine whether the model encompasses appropriate risks, review rationale behind existing rules and update if needed, perform a threshold and parameter assessment, review a sample of transactions, and evaluate management reports.
• Data validation – checks that accurate and complete information is captured by a system to execute an AML model. A system can be professionally designed and implemented to achieve its objective; however, it can end up failing due to issues with data integrity. To complete this validation, a bank must gain an understanding of its systems, assess the data requirements of the model, determine whether the requirements are fully met, determine whether any information appears lost or corrupted, assess the quality of controls in place, and review sample data.
• Process validation – helps ensure the adequate design and ongoing sustainability of the processes and administration of the AML system and model. Banks should review the process documented for the AML model and ensure that no gaps are identified. Steps in process validation include assessing data requirements, identifying key trigger events and metrics, evaluating the rules and parameters used to generate reports, and reviewing the most recent relevant internal audit testing performed over the BSA/AML program.

The AML model validation space is increasingly being scrutinized by regulators. Banks may face fines or other penalties from regulators if their AML model validation is not properly executed. Censure from regulators may occur if a bank’s model validation does not have appropriate testing or parameters, such as if it uses a conceptual framework that is inconsistent with regulatory expectations, a logic or methodology that is not commensurate with the risks of the organization, a reliance on unidentified models, inaccurate customer or transaction risk measurement, fundamental logic errors, or an inconsistent approach/lack of detailed documentation to support model risk management activities.

Banks may also get in trouble over model management, such as unclear lines of authority or a lack of appropriate resources deemed necessary to effectively manage risk activities. Regulators also focus on whether a model is up to date, and the failure of a model to identify changes in an organization’s activities that have an impact on risk can be a large issue. Examples of institutions that did not have proper AML models in place include:

• Rabobank, N.A., which exempted certain accounts when its activity triggered the transaction monitoring system.
• Western Union, which whistleblowers alleged failed to identify or ignored higher-risk vendors in its model.
• U.S. Bank, which conducted below-threshold testing that resulted in SARs filings, yet the bank did not change its thresholds to meet its below-threshold testing findings.
• Alpine Securities, which had information about customers’ prior criminal histories but did not incorporate that risk into its model.

To avoid running afoul of regulators, banks should ensure that their AML models and validation efforts have the proper guardrails in place. First and foremost, banks should involve senior management in AML compliance programs and ensure that experienced and independent personnel conduct model validation. The models should have adequate internal controls designed to ensure compliance with the BSA and its implementing regulations. Banks should also utilize a risk-based approach in designing and implementing AML models to properly focus resources and efforts commensurate to the level of risk posed. To accelerate their validation projects, they should identify frequent deficiencies and make use of regulator findings with reference to a model’s validation, which will help predict the examiners’ expectations. Finally, banks should have a plan documented for the validation process by defining what needs to be accomplished through the model validation effort. This documentation should demonstrate that all model validation policies and procedures have been completed successfully.

Applications for Loan Credit Modeling

Following the global economic crisis, the Financial Accounting Standards Board (FASB) began to revisit how banks estimate credit risk in the allowance for loan and lease losses (ALLL) calculation. The legacy accounting framework is an impairment- based concept that is largely driven by incurred losses. Under this approach, loans are recognized as impaired when management determines that it is probable that future cash flows will be not be collected in full as per the contracted terms. This approach will be replaced by the new CECL framework in which credit losses are estimated for the life of the loan at origination and for subsequent reporting periods. In contrast to the ALLL system where credit losses are recorded based on a probable impairment criterion, the CECL approach mandates that financial institutions develop a forward-looking estimate for credit losses for all loans and covered financial assets irrespective of any adverse change in loan performance or fundamental financial metrics.^[4] As the CECL implementation will in many cases require the development or onboarding of new credit risk models, we offer recommendations for successful implementation by keeping the requirements of validation front of mind during model development. In sum, the core purpose of a CECL model is to generate estimated losses on the loan portfolio (output) using relevant sources of data (inputs), including bank historical loss experience, peer/industry loss history, loan risk metrics, national and regional economic forecasts, and interest rate and credit market expectations.

The first step in the model process involves an introspective review of the institution’s analytical modelling capabilities and the existing credit risk management culture. The scope of the CECL model development project is expected to vary considerably across the spectrum of financial institutions consisting of small community banks, regional banks, super-regional banks and complex or specialized banks, and large national bank platforms. The smallest institutions are expected to utilize third-party software vendors or consultants, and they typically do not have an internal model validation staff. At the other end of the spectrum, large national banks will be able to leverage certain data, systems, and processes from internally developed CCAR/DFAST^[5] predictive stress testing models. While regulators have signaled that CECL expectations will vary according to bank size and complexity, they note that there is no bright line to delineate smaller or less complex banks. Similarly, CECL does not prescribe specific model approaches or methodologies, nor does the guidance set expectations or ranges for the model output (expected credit losses) or the resulting change in the allowance levels from the ALLL framework.

Taking inventory of a bank’s resources involves recognizing that establishing a strong model risk management framework requires a diverse set of skills. In the case of CECL implementation, this involves quantitative model fluency and programming skills, fundamental credit risk management, project management capabilities, regulatory expertise, loan product specialization, technical writing and documentation skills, and internal audit and testing design experience.

As part of this internal assessment, banks should evaluate the current culture of model risk management at the institution by addressing the following questions:

• Where does model risk management fit currently in the bank’s culture?
• Does the bank currently have a robust (forward-looking) credit risk management culture?
• Is risk management a continual process, or is the risk management largely a compliance function driven by external requirements or financial reporting requirements?
• How is model risk management conducted? Is it a back-ended mechanical (check-the-box) process or a continual process embedded in the institution’s culture?
• How does the staffing and headcount of bank reflect the culture of credit risk and model risk?
• Are there key-person risks for models? How deep is the bench for model development, model implementation, and model risk management?

The challenges posed by CECL can also represent an opportunity to integrate model risk management considerations into the model development process. In a broader sense, the CECL design and implementation process may provide the impetus for growing regional banks to align credit risk management with related internal accounting and market risk functions to develop a robust enterprise risk platform. We recommend incorporating validation requirements into the de novo model design to ensure that the CECL model not only serves its core purpose, but that the model can be seamlessly integrated into a model risk management framework. The starting point to this process is to allow the model validation team/stakeholders to participate in the design project. This facilitates communication throughout the process so that the validation team is kept apprised of developments during model creation and the design team is cognizant of how the model will be documented and tested.

Additional recommendations for collaboration include the following:

• Allow input from internal or external model validation teams into the model development process
• Budget time for model risk management in the model development project plan
• Facilitate discussions around model conceptual framework and appropriateness of modeling choices
• Ensure that stakeholders from multiple function areas are involved throughout the process, including data gathering, portfolio segmentation, model design, and identification of model constituents (owners, reviewers, users of the model output)
• Avoid back-ended validation processes where model testing is solely compliance driven and occurs only after the model is complete
• Consider opportunities to rotate roles, e.g., model developers provide input on how the model will be tested and document the process, and model risk management are invited to participate in the model development process

Finally, below we highlight common pitfalls to avoid as applied to CECL model implementation.

• Use of inappropriate loan performance data sets (without adjustments) because they are easily available
• Reluctance to abandon legacy credit estimation models that are no longer fit for purpose
• Inappropriate use of historical performance data without considering changes to underwriting, portfolio composition, current economic conditions, and the future market outlook
• Lack of understanding of mathematical models, implicit assumptions, and source data used by third-party vendors’ credit modelling software
• Overreliance on peer bank performance data, or alternatively, ignoring relevant out-of-sample information
• Limited institutional understanding of credit risk models outside of a small group of bank professionals
• Imbalance in skills and resources between business owners, model developers, and model reviewers and a lack of integration across loan portfolio segments or business lines
• Insufficient documentation of the model framework and model review process. For example, could an external party follow/replicate the model design, implementation, and modifications?
• Lack of a developed policy on when and how models should be updated, modified or replaced
• Difficulties integrating firm data sources and software systems (accounting, underwriting, IT, credit risk, market risk, disparate business units)

Third-Party Validation

Much like the Cheshire Cat helped guide Alice in Wonderland, financial institutions can and should leverage third-party independent consultants to perform reviews of their models.

Third-party consultants and external model validation resources can act to facilitate informal reviews and comment prior to a formal (pass/fail) test or regulatory review. External consultants can provide fresh insights gathered from a variety of projects serving multiple peer institutions. Consultants can also contribute skills and experiences obtained from multiple functional roles, including business users, model developers, model reviewers, and C-Suite or management oversight.

From an operational perspective, consultants can provide flexibility with staffing and resource constraints to meet internally and externally imposed deadlines. Third-party advisors who have experience with multiple vendor systems can assist with the vendor-selection process and evaluate third-party models and sources of data. Finally, external validators are independent from the banks and work outside of the functional units within the institutions (e.g., internal auditing, model risk management, external auditing). In this vein, independent consultants can offer valuable insights to management, boards of directors, and shareholders. And, of course, third-party consultants and independent model testing and validation can help keep financial institution employees and their regulators from going mad.

Stout Analyst Lauren Rosenberg contributed to the development of this article. 

1. OCC Letter 2011-12 (hereinafter “OCC Letter”); Federal Reserve SR Letter 11-7.
2. OCC Letter.
3. “AML model risk management and validation,” EY, 2013.
4. For further detail and guidance on regulatory implications, see: “Frequently Asked Questions on the New Accounting Standard on Financial Instruments – Credit Losses,” Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, National Credit Union Administration, Office of the Comptroller of the Currency, April 3, 2019.
5. Dodd-Frank Act stress testing (DFAST), Federal Reserve Comprehensive Capital Analysis and Review (CCAR).