Watch: Masters of Risk | Season 4 Episode 4 (Video): Bias, Blind Spots, and Bottom Lines: Reframing AI Risk in Financial Services

View Full Transcript

Stewart Webster   00:00:00

On this episode of Masters of Risk, I'm joined by Karim Saleh from FairPlay AI to explore one of the most important and often overlooked challenges in modern finance and technology, fairness. As artificial intelligence becomes more deeply embedded in financial decision-making from credit underwriting to fraud detection, the stakes are high. These systems don't just process data, they shape outcomes, improve access to capital and ultimately impact people's lives. In this conversation, we dig into where B actually enters AI systems, why the idea of objective algorithms often fall short and what organizations can do to build models that are not only effective but also fair and accountable. Karim, thanks for joining the podcast today.

Kareem Saleh   00:00:53

Thanks for having me, Stu. Delighted to be here.

Stewart Webster   00:00:55

Absolutely. And I know you already, but for our audience, can you start by talking about your background and what FairPlay AI does? And then take us back to the moment when you first realized that AI models, especially in financial services, were not as fair as many people assume because this is the company you founded in 2020, which is years before the average household understood AI. So this is something you've been envisioning for many years now.

Kareem Saleh   00:01:21

I spent my whole career on the problem of underwriting hard-to-score borrowers. I started doing that work in frontier emerging markets like Sub-Saharan Africa and Eastern Europe and Latin America, financing development-friendly projects alongside some of the most prestigious financial institutions in the world. And what shocked me was that even at the commanding heights of global finance, the underwriting methodologies were extremely primitive, usually the 20 to 50 variables and models built and excel. Almost all of those models exhibited disparities towards people of color, women and other historically underserved groups. It wasn't because the people building those models were people of bad faith. It was largely due to limitations in data and mathematics. Now I'm trained as a banking lawyer. So I knew that lending discrimination was both illegal and severely punishable. So I had this puzzle, how is it possible that on the one hand, discrimination in lending is severely punishable. And on the other hand, it's ubiquitous. It's everywhere. And what I found is what I jokingly refer to as the fair lending industrial complex, which is a cottage industry of consultancies and law firms hired to come up with clever statistical and legal justifications for disparities in models rather than actually fixing them. And so the aha moment came actually even earlier than we founded the company in 2017 that my co-founder, who was an early employee at Google, started experimenting with adversarial networks, and that's the same math that powers deep fakes. And he began applying them to credit underwriting, and we ran a pilot with a major mortgage originator, and we found that we could lift the approval rates for black consumers by about 10% with no increase in risk. for that mortgage originator, that translated into roughly $6 billion in additional credit originated and around 50,000 more families and homes. So good for profits, good for people and good for progress. And that's kind of when we realized that in the AI era of kind of more complex models and alternative data that bias testing, fairness testing was about to go from being a kind of low rigor, low-frequency compliance exercise to a high rigor, high-frequency engineering discipline. So today, FairPlay is the AI enablement company for the financial services and insurance industries. We help institutions test, tune, validate and monitor AI systems and agents from traditional credit models all the way up to agentic AI systems, handling things like sanction screening, -- some of the biggest names in financial services rely on us to find and fix the blind spots in their AI systems in ways that let them make more money and do more good.

Stewart Webster   00:04:39

Wow, that's fascinating. And this is a company you started 6 years ago. So I'm sure the idea has started way before then. So clearly, you have the credibility there. You have a strong background in credit underwriting, as you articulated. And then also, you're an attorney, too. So you understand the legal aspects and the regulatory concerns around AI. I guess in terms of my perspective, algorithms in finance are important because they help with cost minimization, profit maximization and they eliminate human error. So one of the sections of finance where we see a lot of this is around trading, right? And many individuals would exhibit these trading biases through cognitive errors or through different types of behavioral biases, right? And the AI or training algorithms helps eliminate that. So it seems great. And computers are also concerned with being optimal. They're concerned with being efficient. So many people believe algorithms eliminate human bias. But why does that assumption fall apart in real-world AI systems?

Kareem Saleh   00:05:43

Yes. It's a beautiful idea, and it's totally wrong, right? Algorithms work great if your credit data is present and correct. But if you're one of the 60 million Americans whose credit data is messy, missing or wrong or if you belong to a group that's been systematically excluded from or prayed upon by the financial system, the models can treat the absence of information as inherent riskiness. There's also a more subtle issue, which is that models overweight certain variables. There are variables that are perfectly predictive for some populations, but they can be actively harmful for others. So if you take a variable like consistency of employment, it sounds like a perfectly reasonable credit variable, but it's a much better signal for a man than it is for a woman because all things being equal, women are statistically more likely to take time out of the workforce to start a family or care for a loved one. It's a variable that looks neutral on its face, but actually encodes a disparity. Another reason algorithmic bias emerges in AI systems is that machines are capable of learning the wrong things. Once upon a time, when we were first starting out applying a complex machine learning techniques to AI, my co-founder and I worked at a lender where we built a credit underwriting algorithm that told us very confidently that we should make a bunch of loans in Arkansas. Now it just so happens that my co-founder is from Arkansas, and he happened to know that the regulatory regime in Arkansas made that particular lending product essentially impossible. So why was the algorithm so bullish on Arkansas? Well, because the training data had no loans from Arkansas in it. And so no loans from Arkansas meant the algorithm had never seen a defaulted loan in Arkansas from which the AI concluded that loans never went bad in Arkansas, right? So if you don't build safeguards that accommodate bad data and correct for it, then you encode the unfairness of the past. into the digital decisions that will govern our futures. Algorithms don't eliminate human bias unless you actively correct for it. In fact, they scale human bias and potentially launderer it through math.

Stewart Webster   00:08:18

Well, so clearly, there's some negative downstream effects when AI goes wrong, right? And you said a lot there. So let's spend some time unpacking where the AI biases are coming from because they could be quite detrimental, as you mentioned. And there have been many instances of AI biases negatively impacting people. Actually, recently, there was an experimental study led by Lehigh University, and they actually use real mortgage data, and they found AI systems will recommend higher interest rates and more denials for black applicants with identical financial profiles. So I guess, in the AI life cycle, where do you most often see unfairness introduced? Is it the historical data being used? Is it the model design? Is it deployment? Or is it something else?

Kareem Saleh   00:09:05

Yes. It's everywhere. We see it across the model development life cycle and across the customer journey. It starts with the data. seemingly fair variables can interact in ways that quietly encode sensitive information like gender or race or age. Now we were working recently with a used car lender that was using 2 innocent-looking variables in its credit model. One of the variables was mileage, what's the mileage on the car that's proposed to be purchased and the other was a state, specifically, do you live in Nevada? And mileage is a perfectly reasonable predictor of credit risk on a used car, right? Because the higher the mileage, the more likely the car is to break down, the more likely the car is to break down, the more likely the borrower is to miss work. The more likely the borrower is to miss work, the more likely that they'll lose their job. And if they lose their job, they miss their car payment, right? So totally rational variable. The second variable was, do you live in Nevada? And it turns out that Nevadans have a slightly elevated risk of default relative to the rest of the country. Maybe that's attributable to Nevada's long-time association with the gaming industry and maybe that makes Nevadans more risk-seeking. Maybe there's something going on macroeconomically in Nevada, whatever it is, I don't know, Nevada's default at higher rates. Now no, this is do you live in Nevada. It's not like red lining a particular ZIP code in Nevada. It's a state-level variable. So it's relatively innocent and it has a clear connection to credit risk. There's just one problem. If you're buying a high mileage car in Nevada, there's like a 70% probability that you're a person of color. So you think you're making a decision based on mileage and Nevada, but you're actually making a decision based on race. Now compounding that is even with clean data, the model itself can be unfair because models overfit to populations that are well represented. If you're an online lender, your model is probably biased against the elderly, not because anyone designed it that way, but because there are far fewer elderly people seeking loans online. right? And once you deploy, your fairness can degrade because the world changes, right? The knock-on door population shifts, the macro environment shifts, your training data becomes stale. Think about models trained on data from the pandemic when there was a moratorium on negative credit reporting, right? Those models have a distorted view of the world. Today's inflation environment and geopolitical environment is stress testing them in real time. Beyond the model itself, though, unfairness can enter anywhere in the customer journey. Think about marketing. Your underwriting model might be perfectly fair to women, but if you're only marketing in men's health magazines, you're going to get skewed outcomes anyway. So it turns out that you have to test for bias at every step. Is your marketing fair? Are your fraud screens fair? Is your identity verification, your income verification, your underwriting, your pricing, your line assignment, your account management, your collections, bias has the potential to creep in at every point of the customer journey, and you won't know it if you're not actively measuring it.

Stewart Webster   00:12:36

Well, that's a lot there and very interesting. And at S&P, we sell a ton of financial data. We like to say that data is gold. And from what I'm hearing from you is it's not about the amount of data that you have, the breadth of data that you have, but it's really about making sure that it makes sense, right, to put it in layman's term more directly. Going back to the models here, one of the issues I foresee is that unfair models or models with inherent biases could be profitable and correcting those models or making them more fair would result in lower profits or perhaps negative earnings, right? So how can AI models be made fair without sacrificing accuracy, without sacrificing profitability or risk performance. How do you guys do that in practice?

Kareem Saleh   00:13:21

Yes. So I think there's a widespread misconception that fairness comes at the expense of accuracy. In practice, we see the opposite more often than people expect. A lot of unfairness is actually just a symptom of information inefficiency, right? So take cash flow data, it's often a much better representation of someone's financial reality than a traditional credit file. So when you incorporate it, you're not just being far, you're being more accurate. In addition, a lot of these newer AI training techniques have as their animating goal, doing a better job of risk ranking populations that are poorly represented in the data. And so it turns out that like by incorporating information about the distributional properties of hard-to-score groups, you can add knowledge to the model, not sacrifice accuracy. And the numbers back this up, like we consistently see that something like 25% to 33% of the highest scoring applicants that get declined would have performed as well as the riskiest applicants who get approved. And that generally means that lenders can increase their approval rates by 10%, increase their take rates by 13% and increase inclusion by 20% without taking on additional risk. That's like good for the company, good for the consumer and good for the community.

Stewart Webster   00:14:48

No, absolutely. So I guess this next question, this is really about the bread and butter of Fair Play AI. So once the model is found to be unfair, what are the most effective ways to actually improve it rather than just documenting that there's an issue?

Kareem Saleh   00:15:02

Well, if you just like step back a moment and think about it, every AI system has to be given a target. right, an objective that it relentlessly optimizes for. And if you give a machine a single goal, it will pursue that goal relentlessly and sometimes in ways that you don't like or intend. Take the Facebook algorithm, right? The Facebook algorithm optimizes for engagement. So it shows you whatever keeps you engaged regardless of whether the things it's showing you to keep you engaged are good for your mental health or good for society, right? You didn't tell us to care about that stuff. You only told the machine to prioritize engagement. Now think about Tesla, right? What if Tesla gave the neural network that powers its self-driving cars, the mere objective of getting passengers from point A to point B. Well, the car might do that while driving the wrong way down a one-way street or blowing through red lights or causing mayhem to pedestrians. So what does Tesla have to do? It gives the neural network 2 targets, get the passenger from point A to point B and respect the rules of the road. And it turns out that we can do the same thing in financial services, right? Instead of telling the model, predict who's going to default, we tell it, predict who's going to default and minimize unjustified disparities. And the good news is it works. The models actually get sharper because you're forcing them to be thoughtful about variables they would otherwise overweight. And I think that is the key point. You're not weakening the model in the interest of fairness, you're making it more complete.

Stewart Webster   00:16:47

Now thinking about the C-level executive, many of our audience members are CEOs or CFOs or big fund managers. So they're concerned about the bottom line and maximizing return for shareholders. How do you explain the risk of unfair AI models to senior leaders who tend to think about risk primarily in terms of financial or regulatory terms?

Kareem Saleh   00:17:09

Yes. I don't advocate fairness for fairness's sake. There's objective evidence that it pays to be fair. There's a business rationale alongside the ethical one. So the pitch to a CEO or a CRO is straightforward. you're leaving money on the table, 10% to 20% of it, right? You are carrying undiagnosed model risk because unfair models tend to over rely on a small number of variables, which means they have blind spots that break at the edges or under stress. And those blind spots will leave you carrying regulatory and reputational risk, consent orders, exam findings, enforcement actions, viral headlines. And any one of those dwarfs the cost of doing this properly. So the line that I keep coming back to is you can't run decisioning in the AI age on fairness infrastructure from the stone age, right? You've got to be as sophisticated in your AI compliance and governance as you are in the rest of your business. Fairness infrastructure is not a drag on speed. It's actually the thing that lets you move fast.

Stewart Webster   00:18:24

And 20% is a huge number, increasing revenue or increasing earnings by 20% is music to it in executive's ears, right? And your business would double every 5 years if you had access to that 20%, and that's what you're articulating here. So very interesting findings there. One thing, if I were to turn on the news right now, I'd be hard-pressed not to see a segment about AI replacing jobs, right? That's a big concern amongst corporations and people losing their job potentially to AI. So for you, as an AI specialist and the founder of an AI company, right, so you understand these mechanics and the dynamics of AI. So as an executive in that space, how do you expect artificial intelligence to impact the labor force over the next decade in both terms of job displacement and job creation?

Kareem Saleh   00:19:12

Yes. It's too soon to tell, and there are sharply differing views. Let's like just start by remembering that like every new technology is accompanied by a moral panic. Before the traffic light, humans directed traffic. And when the traffic light arrived, the question was, what are we going to do with all these people who used to direct traffic? And we ended up with a whole new industry of people who fix traffic lights, right? So technology tends to reshape work more than it eliminates work outright. Like when video editing got cheaper, we didn't fire the editors. We got YouTube and TikTok and 1 billion creators. When design tools got better, companies didn't slash design budgets, they invested more. So my advice to young people trying to figure out the future of work is invest in skills with elastic demand, like take accounting. If AI made accounting 10x faster tomorrow, we wouldn't suddenly need 10x more balance sheets, right? The demand for accounting is inelastic. There's a natural ceiling on how much of it a society wants. But if software developers became 10x more productive, would we build fewer apps? No. We build 100x more features, more automation, entirely new product categories. The demand is elastic. And so the more supply we create, the more new demand appears. Now that said, AI is coming fast, probably faster than any prior technology wave. And so we face the risk that job displacement outruns society's ability to absorb AI. knowledge work, I think, is in the crosshairs in a way that it wasn't in prior waves of innovation, and we can't put our heads in the sand about that. That means investing in the things AI is worse at like judgment and taste and emotional intelligence and the ability to build trust and lead people through ambiguity. We're going to need to retrain, redeploy, redefine. AI may not necessarily replace work, but it will absolutely amplify industries where demand is limitless and shrink the ones where it isn't.

Stewart Webster   00:21:43

I think you brought up a good point in terms of AI and technology bringing up -- creating jobs that haven't existed before, right? If I took a time machine back to 1929, and I said, I'm a software engineer, I would be the only software engineer in the world, right? Because software and application didn't exist at that time. And I would say less than 10 years ago, software engineering was one of the best careers that you could have, right? But now with AI taking over, there aren't as many software engineer jobs, but that career is becoming evolved because of artificial intelligence. So there are a lot of dynamics at play. One of the things I've been seeing a lot lately is companies participating in the AI race. And they're encouraging employees to use AI in their daily workflow. They're creating new AI products. So there's a huge race for AI and everyone wants to be the winner. Looking ahead, what do you think organizations will wish they had done earlier when it comes to fairness and risk in AI?

Kareem Saleh   00:22:46

Companies are pouring billions into better models, better data, autonomous agents, but their compliance and governance systems are often manual and back-looking and underpowered and under-resourced. And that just doesn't work in a world of continuous AI-driven decisioning. If you want to move fast with AI, you need equally sophisticated fairness and governance infrastructure. Otherwise, governance becomes the bottleneck, right? So just as Google built search infrastructure for the Internet and Stripe built payments infrastructure for the Internet, we're going to need to build fairness infrastructure for the Internet to debias digital decisions in real time. And I think the winners in the next phase will be the ones who understand that fairness isn't friction. It's actually infrastructure for speed.

Stewart Webster   00:23:38

That was Karim from Fair Play AI, offering a clear and thought-provoking perspective on the risk and responsibilities that come with deploying artificial intelligence at scale. What stands out is that AI risk isn't just a technical issue, it's a business risk, a reputational risk and in many cases, a societal one. As organizations increasingly rely on algorithms to make critical decisions, the ability to understand, measure and improve fairness will become a defining capability. Karim's insights remind us that the future of AI won't be defined by how powerful our models become, but by how responsibly we choose to use them because in the end, the real risk isn't that machines make decisions, it's that we fail to question the decisions they make. If you enjoy this episode, don't forget to like, share and leave a review. Thanks for listening to Masters of Risk. I'm Stewart Webster, and we'll see you next time.