In the age of artificial intelligence, your chances of getting a loan are increasingly shaped not by a person but by an algorithm. Machine learning (ML) models are now extraordinarily powerful and accurate, able to detect patterns in financial behavior that a traditional loan officer might miss. But they come with a cost: they are remarkably opaque – and this opacity is more than an inconvenience; it’s a risk because even the model’s own developers may struggle to explain how a specific credit decision was made.
These systems can deny someone a loan based on complex patterns buried deep in the data without offering a clear justification. As Cynthia Rudin of Duke University has pointed out, this lack of transparency is especially dangerous in high-stakes areas like credit scoring, where decisions affect people’s access to financial security and future opportunities.
Indeed, the European Union’s Artificial Intelligence Act has formally classified credit scoring as a “high-risk” use case, putting pressure on financial institutions to ensure that automated decisions are not only effective but explainable. This raises a fundamental question for the future of AI in finance: How much accuracy (if any) are we willing to trade for transparency?
Machine learning models like XGBoost have become the go-to tools in credit scoring, thanks to their high accuracy and ability to model complex, non-linear patterns in large, high-dimensional datasets. But their “black box” nature makes them difficult to interpret. And that’s a growing concern for both regulators and consumers. Explainability is required not only by regulation but also by consumers seeking transparency in how their financial lives are evaluated.
Returning to the issue of opacity, this growing concern has led to the development of new techniques aimed at improving model transparency. Many institutions are playing catchup at the moment – understandable, given the current speed of AI adoption, but the impact can be significant. For example, the Bank of England and the Financial Conduct Authority conducted a survey of AI and machine learning in UK financial services and found that while 75% of firms are already using AI, with an additional 10% planning to do so over the next three years, 46% of these reported only a partial understanding of the technologies they use. That gap between usage and understanding highlights the urgent need for more interpretable systems.
In response, two different approaches have emerged. First, explainability or post-hoc interpretability techniques which work from the model. For instance, after any model is trained, we may use methods based on the permutation of the features to assess the extent to which they affect our outcome. One of the more promising techniques is Shapley Additive Explanations (SHAP), which draws on game theory to create a set of features’ coalitions to rank their local and global importance. These tools are powerful but have their own limitations. They often require high computing power and rely on approximations, meaning that they are missing the statistical rigor of traditional inference. They help open the black box, but they don’t provide a full understanding of it.
An alternative approach is to design machine learning models that are interpretable from the start. For instance, using monotonicity constraints as rules embedded into the model that ensure relationships between key variables and predicted outcomes follow consistent, economically sound logic. Typically, in a credit scoring setting, higher income should not lead to higher risk of default. However, free versions of XGBoost models might find correlations, or interactions that imply a non-monotonic (i.e.: constantly increasing or decreasing) relationship between income and probability of default.
But this introduces a new challenge, and one that motivated our research: what is the economic and/ or social cost of this clarity?
In a recent empirical study with our colleagues Jose Manuel Carbo of the Banco de España and Juan José Guillén García of the Universidad Politecnica de Madrid, we compared two versions of an XGBoost model trained on data from LendingClub, a major U.S. peer-to-peer lending platform. One version was unconstrained and optimized purely for predictive accuracy. The other applied monotonicity constraints to ensure that the relationships between variables follow financial intuition and supervisory expectations.
Predictably, the constrained model performed slightly worse in predictive accuracy. On the surface, the difference was in the second decimal. This may seem negligible, but in credit scoring, where misclassifying even a small percentage of loans can lead to substantial financial loss or unfair outcomes, it raises real concerns.
The path forward is not about choosing between innovation and accountability.
However, we found that the constraints significantly improved the model’s explainability without meaningfully disrupting the underlying logic of the predictions. The model became easier to interpret while still relying on the same key variables. In both versions, the top contributing factors (for example, loan term, installment size, income, and FICO score) measured using SHAP, remained largely consistent at global level.
On another side, at local level, one of the most striking effects we observed was how the constrained model altered individual credit predictions. Rather than producing scores across a wide spectrum of risk, the model tended to pull estimates toward the mean – “mean reversion”. High-risk scores were lowered while low-risk scores pushed higher.
This centralizing effect had two notable consequences. First, it reduced granularity. The model became less effective at identifying edge cases, such as those applicants who were truly risky or exceptionally safe. Second, it raised subtle fairness questions. By smoothing out the distribution of risk, the model appears more interpretable but may unintentionally shift scores in ways that disproportionately affect certain demographic and socioeconomic groups more than others.
For example, our findings show that some borrowers initially assessed as low risk – often younger, higher-income individuals – saw their predicted default probabilities increase. Conversely, some actual defaulters saw their risk scores decrease more sharply than those who repaid their loans. This helps explain the slight dip in overall model accuracy: the constrained version was less likely to flag borrowers at either end of the spectrum.
This then raised a second question: even if a model is easier to explain, how do we know those explanations are statistically reliable? To bridge the gap between transparency and rigor, we used a technique called Shapley regressions to test whether the model’s explanations held up under scrutiny.
This approach quantifies how much each variable contributes to a prediction, using p-values and confidence intervals – familiar tools in traditional statistical analysis. The results were promising: the variables most critical to model predictions were also those most statistically significant, and this remained true in both the constrained and unconstrained models. On other words, adding interpretability didn’t weaken the model’s data-driven foundation, it clarified it.
For financial institutions, that distinction is critical. It allows compliance teams, auditors, and regulators to ask not whether a model works but whether its reasoning is reliable, trustworthy – an important distinction in environments subject to increasing regulatory scrutiny. Overall, financial institutions must rethink how they evaluate AI model risk. Tools like Shapley regressions can help institutions verify that explainability is evidence-based – and when paired with “human-in-the-loop” oversight, it is more likely that automated decisions align with institutional values and social expectations.
The path forward is not about choosing between innovation and accountability. It’s about designing systems that optimize both, leveraging the well-known “multiplicity of good models” theory of machine learning. It’s possible to improve model transparency and maintain performance – if institutions rethink how they evaluate risk, measure fairness, and communicate AI-driven decisions.
Inherently interpretable models, supported by statistical validation tools like Shapley regressions, offer a new blueprint for responsible AI adoption, one that reinforces trust with customers, withstand regulatory scrutiny and helps financial institutions understand both the what and the why behind a model’s predictions.
The challenge is as cultural as it is technical. Financial organizations and fintech firms must make deliberate choices about how they design and govern their AI systems. That means building teams that combine data science and domain expertise in order to ensure the models used reflect real-world constraints. It also means prioritizing an oversight structure that keeps humans in the loop as a way to reinforce accountability.
These choices matter because early design decisions tend to become long-term practices. If AI-driven credit decisions remain opaque and inconsistent, if they are left in the black box, we risk automating inequality and bias. Trust in financial systems will eventually erode. By embracing explainability, it’s possible to develop credit systems that are a benefit to the individuals whose financial futures hang in the balance as well as for the long-term health of the system itself.
© IE Insights.
