Although the Board of Governors of the Federal Reserve System (Federal Reserve) and the Office of the Comptroller of the Currency (OCC) released regulatory guidance (hereinafter “the Guidance”) related to model risk management in 2011, there has been little to no additional guidance from regulators since. As a result, most Bank Secrecy Act/Anti-Money Laundering (BSA/AML) practitioners have been left scratching their heads when it comes to the practical interpretation and implementation of this Guidance in BSA/AML-related models. Many financial institutions are unsure of regulators’ expectations with regards to model risk management, model validation, and, in some instances, they are not even sure what qualifies as a “model.” Indeed, even regulatory enforcement actions, which have been a traditionally reliable source of information for BSA and compliance officers about what not to do, often providing cautionary tales (i.e., following the George Costanza theory that, if what is currently being done is wrong, then the opposite must be right), when it comes to model risk management specifically, regulators have been generally silent – until recently.
On January 30, 2020, M.Y. Safra Bank, FSB (Safra or the Bank) entered into a Consent Order with the OCC that, among other things, requires the Bank to:
Herein, we provide a very brief overview of the Guidance with a focus on model validation requirements. In addition, we offer BSA/AML and compliance practitioners some practical recommendations and lessons based on the Guidance, on our experience working with models and regulators, and on the Safra Consent Order.
According to the Guidance, a “model” is “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” BSA/AML and sanctions automated systems use complex algorithms, behavioral monitoring analysis, anomaly detection, statistical theories, and even artificial intelligence to detect potentially suspicious activity or OFAC sanctions matches. These systems are considered “models,” and as such, they are subject to the Guidance. Specifically, the Guidance aims to ensure that financial institutions develop, implement, validate, update, calibrate, optimize, document, assess, and report on their models in an appropriate manner.
Within the context of AML and sanctions, the most commonly used models are:
Various sections within the Guidance put out requirements, which ask financial institutions to:
Model validation requirements are contained in Section V of the Guidance, which defines model validation as “the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses.” An effective validation framework should include three core elements:
Additionally, financial institutions subject to supervision by the New York Department of Financial Services are required to comply with the provisions of Regulation Part 504, which require that models are commensurate with a bank’s risk, and that they maintain adequate documentation and support, including model testing, validation, and tuning.
Given how important AML and sanctions models are – and the intense regulatory scrutiny these models are subject to, as evidenced in the Safra Consent Order – we wanted to share some practical suggestions on sound model governance and risk management practices, and on how to get the most out of the model validation process.
The board and senior management should have ultimate responsibility over the development and oversight of models, including ensuring adequate frameworks, policies and procedures, and internal controls for managing the models.
It is considered best practice for banks to perform regular model validation, ideally every 12-18 months. Banks may also conduct model validation following certain trigger events, such as a significant change to the AML or sanctions risk, changes to the model or rules due to a change in risk profile, or after a significant model version upgrade. Some banks perform “limited scope” validations, for example, data validations triggered by changes to the source systems that provide customer data and transactional data. Banks may also perform targeted threshold tuning or optimization engagements if they encounter significant shifts in model effectiveness or efficiency, or a surge in false positives.
Many institutions do not have the internal knowledge or expertise needed to perform effective validations. As such, external validation firms are often necessary. It is important to engage a validation firm that has experienced subject matter experts – those performing the validation must have the authority to challenge model developers and users, but they must also be independent. Validators should possess the requisite knowledge, skills, and expertise, as well as a significant degree of familiarity with the business in question and with the model’s intended use.
There are four broad components to the model validation process, outlined below. These aspects require a combination of vastly different skillsets.
The validation team must possess the right combination of skills and expertise, as there have been instances of regulators challenging banks to demonstrate that they conducted adequate due diligence based on those regulators examining the vendor experience and the biographies of the validation team. Those banks that were unable to prove adequate rigor and diligence have been censured.
Once the appropriate vendor has been selected, banks must insist on a well-defined model validation plan that clearly defines the scope, objective, timelines, key milestones, and deliverables for the process. This must include frequent validation status updates (i.e., weekly), to allow for any open items or questions to be discussed and to allow for the validators to share preliminary findings and discuss any alternate or mitigating controls that they may not previously have considered. This can help avoid any misunderstandings or surprises later in the process.
As noted above, there are four key components to model validation. Below, we discuss each of these components and offer suggestions on how banks can best use the results of their validation to enhance and refine their models.
During this stage, the validator assesses the conceptual soundness of the model and its design, as well as gauging whether the model aligns with the AML and sanctions risks of the institution. Validators will review documentation, including AML and sanctions policies and procedures, risk assessments, prior audits, examination and validation reports, and model documentation. The validator will also interview key personnel and assess how AML and sanctions risks (typically captured in a risk catalog or inventory) are addressed by the model’s design and configuration, in relation to the bank’s risk appetite. The bank must be able to demonstrate that the model documentation adequately explains the bank’s rationale for the design and development of the model, as well as its selection of detection rules and scenarios, model configurations, the basis for its parameters and threshold settings, the assumptions used in the model design, and any known limitations and mitigating factors. It must also address any customers or transactions that are excluded or exempt from the model.
The conceptual soundness assessment enables the bank to get an independent expert’s views on whether their model addresses the following key elements:
If necessary, the bank should carefully consider the validation team’s recommendations for enhancing risk coverage or implementing additional detection scenarios to cover unmitigated risks.
During this step, the validator will review if the model captures information that is accurate and complete enough to allow it to execute the transaction monitoring rules and sanctions screening algorithm effectively. Validators must gain an understanding of the source systems and the data requirements of the model, and verify the key data elements (KDEs) identified by the bank as necessary for proper functioning of the model. These typically include key fields such as customer names, originators and beneficiaries, transaction types, dates, amounts, countries, etc., which are critical elements for accurately triggering alerts based on detection scenarios. The bank must be able to demonstrate adequate control over the accuracy of data mapping from the core/source systems to the AML and sanctions models. Often, data are imported into these models after clean-up and enrichment via an intermediate data warehouse system; banks must adequately document and test the controls in the exchange, transform, load (ETL) process to ensure data integrity.
This component can be used by the bank to improve data mapping, and to enhance the controls related to the identification and enhancement of data quality for the relevant KDEs. Banks can use this assessment to identify and correct any inaccurate, invalid, or missing information in these key fields, which may have previously caused the model to miss some alerts.
This stage ensures that the system is properly designed and that it has been implemented appropriately for the detection of suspicious activity or potential sanctions matches. This is performed using outcome analysis and back-testing of rules and detection scenarios, by mirroring the rule logic, running this replicated rule logic against the bank’s historical transactions, and testing the alerts that are generated by comparing them to the historically generated alert output. This helps the bank identify any logic issues within the rules which may need to be corrected. In the case of sanctions models, functional testing includes testing using a set of “control names,” which are sanction names that have intentionally been altered in order to test the system’s ability to different sanction name variations.
The validator will also verify model performance using above-the-line (ATL) and below-the-line (BTL) testing, which involves the upward or downward adjustment of model parameters and thresholds in order to verify whether there is any unusual activity clustered either below or above the existing thresholds. The results allow banks to determine whether they need to further finetune their parameters and thresholds to optimize model effectiveness and efficiency.
This last component involves an assessment of the design and ongoing sustainability of the processes surrounding the administration of the AML and sanctions model. This includes evaluation of the controls related to rule scheduling and execution, to the generation and review of alerts, to the implementation of “four-eyes” reviews, and to the timeliness of alert escalation and dispositions as well as alert review and escalation when suspicious activity is identified. The validator will also assess data reconciliation processes and controls to ensure that daily data feeds are complete and accurate and that no customer or transactions records are being missed by the monitoring and screening processes. For sanctions models, the validator will also assess the timeliness and completeness of OFAC sanction list updates and the rationale and approval processes for the use of false positive mitigators, such as so-called “good guy” lists, in order to mitigate risk and help reduce false positive matches.
This component offers banks the opportunity to consider potential enhancements to their model risk management processes and operational controls environments.
The Safra Consent Order is a recent and cautionary tale of what can happen when financial institutions fail to manage their BSA/AML and sanctions models appropriately. Financial institutions, however, can stay out of regulators’ crosshairs if they follow these practical recommendations and if they ensure that senior management has ownership over and involvement in their model risk management programs. Finally, the knowledge and insights gained from the model validation process can help significantly enhance any bank’s AML and sanctions model effectiveness and efficiency.