If you use a credit card, you probably know the feeling of having your card declined due to a suspected fraudulent transaction. An industry report from 2015 found that one out of every six legitimate cardholders experienced at least one declined transaction because of inaccurate fraud detection in the past year. That makes fraud detection an expensive problem for issuers: Those declined transactions lead to nearly $118 billion dollars in losses on an annual basis.
Even though numerous machine learning approaches have been developed in the past to address fraud, newly introduced data science automation platforms like Feature Labs give us a reason to revisit the problem. And now, any organization can see the power of automation for themselves using our just announced developer library, Featuretools.
Recently, teams at MIT and the multinational Spanish banking group BBVA took advantage of Featuretools to develop a new system to better predict credit card fraud. In their report, Solving the “false positives” problem in fraud prediction, they demonstrate how BBVA could reduce the number of false positives by 54% compared to their existing solution, thereby shrinking the number of customers affected by incorrectly blocked transactions. The report estimates the financial impact of the new model to be €190,000 per 2 million transactions evaluated. If it were applied to the approximately 300 million transactions that BBVA processes per year, this would have a huge impact.
So, how did they do it?
Finding the right data
In this particular use case, they had very granular transactional data from credit card purchases. Variables included the purchase amount, time of day, merchant code, the country that the transaction originated from, and many others.
Applying machine learning to create a baseline model with just these variables in hand did not outperform BBVA’s existing system. Instead, as is often the case, they needed to enrich the data with variables that describe the historical behavior of a given card, e.g. the most common country in which a card was used. This process of creating new variables based on the raw data is known as “feature engineering.”
Today, feature engineering is one of the key skills required for one to be a top data scientist, which makes it a crucial component of data science automation. According to Gartner, even if an “organization already has a data science team … it may need to be enhanced with even more specialized data science skills specific to machine learning, such as feature engineering and feature extraction.” To address this, Feature Labs developed the Deep Feature Synthesis algorithm for automated feature engineering and made it available through our hosted platform and open source library, Featuretools. With Deep Feature Synthesis it is possible to engineer and extract the features that capture the implicit predictive signals to make raw data “machine learning ready.”
An Algorithm to Automate Machine Learning
BBVA first came across Deep Feature Synthesis last year, calling it “the algorithm that will automate machine learning.” For this project, BBVA had the opportunity to apply the algorithm to their own data by using Featuretools.
With Featuretools, over 100 new historical behavior variables were automatically generated from BBVA’s data by applying Deep Feature Synthesis. By adding these variables to the model, they reported a twofold improvement in the machine learning performance metrics compared to the baseline model, which relied solely on transactional variables. This not only highlights the pivotal role that feature engineering plays in data science, it also showcases the sophistication of automation today.
Understanding Advantages of Automation
Data science is a quantitative process. The metrics being evaluated drive the solution that gets developed. When those metrics focus on the accuracy of a machine learning model, it is easy to lose sight of the actual business needs being considered in the first place.
For this case, the baseline model with only transactional features had outperformed the existing system in terms of precision and F1 score. But it would not have enabled BBVA to earn or save more money if it had been deployed into production.
To understand this, MIT and BBVA developed an economic model that accounted for two variables: the money saved from catching fraudulent transactions and the revenue increase from flagging fewer valid transactions as fraud. While the machine learning method cannot optimize for these figures directly, a thorough analysis should take them into account.
The main priority of the work was to optimize the financial impact of fraud detection on BBVA. Fortunately, feature engineering automation gave MIT and BBVA extra time to thoroughly develop an economic model to assess this impact. Ultimately, the €190k ROI benefit can be broken down into €55k in savings by more effectively blocking high value fraudulent transactions and a €135k increase in top line revenue from identifying fewer valid transactions as fraud.
Focusing on What Really Matters
As companies start using machine learning to accomplish their business goals, they will encounter two significant challenges. The first is that it is difficult to give data scientists, developers, and domain experts the tools they need to efficiently build a deployable solution. And the second is accurately evaluating the solution in terms of how it benefits the business.
Automation technology is crucial to both steps. It streamlines the data science process for everyone involved, giving them the time to figure out what really matters.
