Dear Director Blanco,

I am writing to provide comments on potential regulatory amendments to address Anti-Money Laundering Program Effectiveness. I have spent my career engaged in the global fight against illicit violence and coercion, including operational and policy-making tours in the DoD, DHS, the US Senate, industry and academia. I currently run a behavioral science-led Machine Learning company and teach future generations in the field of national security at Georgetown University in Washington, DC. I have developed familiarity with FinCEN over many years, and I congratulate the team, as well as the BSAAG AMLE WG, on the work represented by this ANPRM. 

I am writing to suggest that technology has not only changed the nature of financial crimes threats, as FinCEN identifies, but also the nature of the countermeasures available to government and industry. The latter point historically, I perceive, is underappreciated by regulators and therefore underutilized by regulated financial institutions (FIs). The dramatic evolution in technology over the past decade, specifically today’s easy access to Machine Learning (ML) and Artificial Intelligence (AI), changes the very way in which FinCEN, regulators, and regulated financial institutions (regardless of size and type) can measure effectiveness and efficiency. Specifically, ML allows us to share models while protecting data, and this sharing allows collaboration across industry and between industry and government. When viewed from the vantage point of understanding innovation, AML effectiveness requires FinCEN leadership to consider defining an “‘effective and reasonably designed’ AML program as one” comprised of three elements.

• Effective performance on government-communicated AML priorities, with this performance defined in ways that can be measured.
• Effectiveness demonstrated at stated levels of efficiency.
• Effective communication from government agencies in the form of training data and test data for measurement of performance on the first two bullets -- effectiveness at priorities and effectiveness at levels of efficiency.

Note:  Without training data from the government beneficiaries of FI-generated data sent to FinCEN, any new regulatory amendments will almost surely fail to produce the desired results. This assertion relates to the nature of Machine Learning and not the skills of any team of individuals.

The three elements identified in the ANPRM may be improved if modified based upon the above points. I elaborate on each point below.

First, the only reasonable way to define effectiveness is in relation to a standard. The current system of measuring SAR yield -- i.e. the number of SARs filed -- is one example of a misguided attempt to measure effectiveness. SARs are an input to a system whose output matters for public safety and security. More SARs filed does not represent progress year over year without qualitative evaluation of the output of the system. As I indicated in my testimony before the House Financial Services Committee on the AML reform bill in 2019, filing higher volumes of low-quality SARs at a lower cost seems like a poor excuse for a goal. I strongly endorse the third component of FinCEN’s proposed definition, indicating measurement of a “high degree of usefulness.” This phrase refers to the BSA; to be clear, my meaning is that increasing data quality means increasing the contribution of this data to national goals such as disrupted terrorist financing and convicted criminals such as drug and human traffickers.

For effectiveness, the government must measure the “output” of the system. Let’s call the successful identification of a criminal a “True Positive.” We want to measure an AML system’s ability to identify True Positives. Data from FIs such as SARs and CTRs constitute “inputs” and True Positives verified by the government constitute system “outputs.”

I suggest FinCEN consider output and not just input (i.e. law enforcement actions and not number of SARs filed), and adopt the following terminology for any regulatory changes to address the effectiveness and design of the AML system: true positive, true negative, false positive, and false negative. These terms are well known in academic literature and can be described or defined in any rule promulgated. For example NASA JPL’s Chriss Mattmann discusses the use of these terms in describing the terms accuracy, precision, and recall in his primer Machine Learning with TensorFlow.

For example, if an FI currently employs five people to clear 1,000 alerts to generate one True Positive, progress could mean generating two True Positives with the same resources (people and alerts), or the same one True Positive but with fewer resources. Discussions could take place over labor hours, alerts cleared, and must include qualitative evaluation of system output:  True Positives generated.

As correctly identified by FinCEN in the ANPRM, the “effectiveness” of the AML system must also measure performance against government-defined Strategic AML Priorities. In the context of this comment, not all True Positives have the same “degree of usefulness to government authorities.” This means that some True Positives must carry more weight than others when evaluating any FIs AML program. For example, if the US government prioritizes three crimes, then effectiveness must measure performance against these three priorities. Of significance but not stated explicitly in the ANPRM, this means that the financial institution can be allowed more False Negatives on lower priority financial crimes. When all crimes are a priority, then no crimes are a priority. By adding Strategic AML Priorities into a regulation, the government must understand that this implies that reporting many other crimes do not provide usefulness to the government. I support this evolution, but I believe that FinCEN should publicly acknowledge this fact that missing lower priority crimes is expected.

Second, measuring effectiveness requires a corresponding efficiency threshold. This measure compares True Positives with False Positives. In our work on GOST, for example, we like to think of an effort curve to identify effectiveness at different levels of work:  what percentage of cases must one review in order to identify a target percentage of threat. An effective and reasonably designed AML program must efficiently identify AML priorities as the priorities are set by the government. A financial institution cannot investigate 100% of customers every day to find 100% of the risk on the population. However, an FI may be able to screen 100% of the population on regular intervals, given certain thresholds of False Positives or hours worked clearing alerts. FinCEN and regulators should consider the efficiency of any AML system and the relationship between True Positive and False Positive results. This point certainly seems related to an FI’s risk profile, as identified in the ANPRM.

Third, the entire premise of FinCEN’s ANPRM, requires that the government provide financial institutions with AML priorities. This requires providing necessary training data. ML and AI systems are inductive by nature; they require examples in order to learn. Any FI, regardless of size and type, requires training data in order to build effective and efficient models of behavior to identify the prioritized criminal activity. Some law enforcement initiatives exist today that share “typologies” from law enforcement with banks; however, we can each build models and share the code. 

I compliment the FinCEN team and the BSAAG AMLE WG and look forward to supporting the progress of this initiative.