The economic explainability of machine learning and standard econometric models-an application to the U.S. mortgage default risk
Abstract
This study aims to bridge the gap between two perspectives of explainability−machine learning and engineering, and economics and standard econometrics−by applying three marginal measurements. The existing real estate literature has primarily used econometric models to analyze the factors that affect the default risk of mortgage loans. However, in this study, we estimate a default risk model using a machine learning-based approach with the help of a U.S. securitized mortgage loan database. Moreover, we compare the economic explainability of the models by calculating the marginal effect and marginal importance of individual risk factors using both econometric and machine learning approaches. Machine learning-based models are quite effective in terms of predictive power; however, the general perception is that they do not efficiently explain the causal relationships within them. This study utilizes the concepts of marginal effects and marginal importance to compare the explanatory power of individual input variables in various models. This can simultaneously help improve the explainability of machine learning techniques and enhance the performance of standard econometric methods.
Keyword : mortgage loan, default risk, machine learning, explainability, marginal effect, partial dependence plot, SHAP
How to Cite
Kim, D.- sup, & Shin, S. (2021). The economic explainability of machine learning and standard econometric models-an application to the U.S. mortgage default risk. International Journal of Strategic Property Management, 25(5), 396–412. https://doi.org/10.3846/ijspm.2021.15129
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Archer, W. R., & Smith, B. C. (2013). Residential mortgage default: the roles of house price volatility, euphoria and the borrower’s put option. The Journal of Real Estate Finance and Economics, 46(2), 355–378. https://doi.org/10.1007/s11146-011-9335-y
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012
Athey, S., & Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27), 7353–7360.
https://doi.org/10.1073/pnas.1510489113
Athey, S., & Imbens, G. W. (2017). The state of applied econometrics: causality and policy evalua
tion. Journal of Economic Perspectives, 31(2), 3–32. https://doi.org/10.1257/jep.31.2.3
Bhardwaj, G., & Sengupta, R. (2010, October). Where’s the smoking gun? A study of underwriting standards for US subprime mortgages (Working Paper). Federal Reserve Bank of St. Louis. https://doi.org/10.2139/ssrn.1361139
Bracke, P., Datta, A., Jung, C., & Sen, S. (2019). Machine learning explainability in finance: an application to default risk analysis (Bank of England Working Paper No. 816). https://doi.org/10.2139/ssrn.3435104
Bücker, M., Szepannek, G., Gosiewska, A., & Biecek, P. (2020). Transparency, auditability and explainability of machine learning models in credit scoring. arXiv preprint arXiv:2009.13384.
Bussmann, N., Giudici, P., Marinelli, D., & Papenbrock, J. (2021). Explainable machine learning in credit risk management. Computational Economics, 57, 203–216. https://doi.org/10.1007/s10614-020-10042-0
Cameron, A. C., & Trivedi, P. K. (2005). Microeconometrics: methods and applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511811241
Campbell, J. Y., & Cocco, J. F. (2015). A model of mortgage default. The Journal of Finance, 70(4), 1495–1554. https://doi.org/10.1111/jofi.12252
Chen, J. (2018). Fair lending needs explainable models for responsible recommendation. arXiv preprint arXiv:1809.04684.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68.
https://doi.org/10.1111/ectj.12097
Chopra, A., & Bhilare, P. (2018). Application of ensemble models in credit scoring models. Business Perspectives and Research, 6(2), 129–141. https://doi.org/10.1177/2278533718765531
Dahiya, S., Handa, S. S., & Singh, N. P. (2017). A feature selection enabled hybrid‐bagging algorithm for credit risk evaluation. Expert Systems, 34(6), e12217. https://doi.org/10.1111/exsy.12217
Datta, A., Sen, S., & Zick, Y. (2016, May). Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In 2016 IEEE Symposium on Security and Privacy (SP) (pp. 598–617). IEEE.
https://doi.org/10.1109/SP.2016.42
Elul, R., Souleles, N. S., Chomsisengphet, S., Glennon, D., & Hunt, R. (2010). What “triggers” mortgage default? American Economic Review, 100(2), 490–494.
https://doi.org/10.1257/aer.100.2.490
Fahner, G. (2018). Developing transparent credit risk scorecardsmore effectively: an explainable artificial intelligence approach. In The 7th International Conference on Data Analytics (pp. 7–14), Athens, Greece.
Feldman, D., & Gross, S. (2005). Mortgage default: classification trees analysis. The Journal of Real Estate Finance and Economics, 30(4), 369–396. https://doi.org/10.1007/s11146-005-7013-7
Fitzpatrick, T., & Mues, C. (2016). An empirical comparison of classification algorithms for mortgage default prediction: evidence from a distressed mortgage market. European Journal of Operational Research, 249(2), 427–439.
https://doi.org/10.1016/j.ejor.2015.09.014
Foote, C. L., Gerardi, K., & Willen, P. S. (2008). Negative equity and foreclosure: theory and evidence. Journal of Urban Economics, 64(2), 234–245. https://doi.org/10.1016/j.jue.2008.07.006
Fuster, A., Goldsmith-Pinkham, P., Ramadorai, T., & Walther, A. (2020, October 1). Predictably unequal? The effects of machine learning on credit markets (Working paper).
Galindo, J., & Tamayo, P. (2000). Credit risk assessment using statistical and machine learning: basic methodology and risk modeling applications. Computational Economics, 15(1), 107– 143. https://doi.org/10.1023/A:1008699112516
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015). Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics, 24(1), 44–65.
https://doi.org/10.1080/10618600.2014.907095
Kabul, I. K. (2017, December 18). Interpretability is crucial for trusting AI and machine learning. The SAS Data Science Blog. https://blogs.sas.com/content/subconsciousmusings/2017/12/18/interpretability-crucial-trusting-ai-machine-learning/
Koutanaei, F. N., Sajedi, H., & Khanbabaei, M. (2015). A hybrid data mining model of feature selection algorithms and ensemble learning classifiers for credit scoring. Journal of Retailing and Consumer Services, 27, 11–23.
https://doi.org/10.1016/j.jretconser.2015.07.003
Kvamme, H., Sellereite, N., Aas, K., & Sjursen, S. (2018). Predicting mortgage default using convolutional neural networks. Expert Systems with Applications, 102, 207–217. https://doi.org/10.1016/j.eswa.2018.02.029
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765–4774.
Mullainathan, S., & Spiess, J. (2017). Machine learning: an applied econometric approach. Journal of Economic Perspectives, 31(2), 87–106. https://doi.org/10.1257/jep.31.2.87
Nanni, L., & Lumini, A. (2009). An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications, 36(2), 3028–3033. https://doi.org/10.1016/j.eswa.2008.01.018
Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3), 54–60. https://doi.org/10.1145/3241036
Pławiak, P., Abdar, M., & Acharya, U. R. (2019). Application of new deep genetic cascade ensemble of SVM classifiers to predict the Australian credit scoring. Applied Soft Computing, 84, 105740. https://doi.org/10.1016/j.asoc.2019.105740
Preece, A., Harborne, D., Braines, D., Tomsett, R., & Chakraborty, S. (2018). Stakeholders in explainable AI. arXiv preprint arXiv:1810.00184.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
https://doi.org/10.1038/s42256-019-0048-x
Sealand, J. C. (2018). Short-term prediction of mortgage default using ensembled machine learning models [Unpublished doctoral dissertation]. Slippery Rock University.
Shead, S. (2020, August 21). How a computer algorithm caused a grading crisis in British schools. CNBC. https://www.cnbc.com/2020/08/21/computer-algorithm-caused-a-grading-crisis-in-british-schools.html
Sirignano, J., Sadhwani, A., & Giesecke, K. (2016). Deep learning for mortgage risk. SSRN Electronic Journal.
Steinwart, I., Hush, D., & Scovel, C. (2009). Learning from dependent observations. Journal of Multivariate Analysis, 100(1), 175–194. https://doi.org/10.1016/j.jmva.2008.04.001
Štrumbelj, E., & Kononenko, I. (2011, April). A general method for visualizing and explaining black-box regression models. In International Conference on Adaptive and Natural Computing Algorithms (pp. 21–30). Springer. https://doi.org/10.1007/978-3-642-20267-4_3
Torrent, N. L., Visani, G., & Bagli, E. (2020). PSD2 explainable AI model for credit scoring. arXiv preprint arXiv:2011.10367.
Tsai, C. F., & Wu, J. W. (2008). Using neural network ensembles for bankruptcy prediction and credit scoring. Expert Systems with Applications, 34(4), 2639–2649. https://doi.org/10.1016/j.eswa.2007.05.019
Wallace, N. (2005). Innovations in mortgage modeling: an introduction. Real Estate Economics, 33(4), 587–594.
https://doi.org/10.1111/j.1540-6229.2005.00131.x
Wang, G., Ma, J., Huang, L., & Xu, K. (2012). Two credit scoring models based on dual strategy ensemble trees. KnowledgeBased Systems, 26, 61–68. https://doi.org/10.1016/j.knosys.2011.06.020
Wang, J., Wiens, J., & Lundberg, S. (2021, March). Shapley flow: a graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics (pp. 721–729). PMLR.
Zhao, Q., & Hastie, T. (2021). Causal interpretations of blackbox models. Journal of Business & Economic Statistics, 39(1), 272–281. https://doi.org/10.1080/07350015.2019.1624293
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012
Athey, S., & Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, 113(27), 7353–7360.
https://doi.org/10.1073/pnas.1510489113
Athey, S., & Imbens, G. W. (2017). The state of applied econometrics: causality and policy evalua
tion. Journal of Economic Perspectives, 31(2), 3–32. https://doi.org/10.1257/jep.31.2.3
Bhardwaj, G., & Sengupta, R. (2010, October). Where’s the smoking gun? A study of underwriting standards for US subprime mortgages (Working Paper). Federal Reserve Bank of St. Louis. https://doi.org/10.2139/ssrn.1361139
Bracke, P., Datta, A., Jung, C., & Sen, S. (2019). Machine learning explainability in finance: an application to default risk analysis (Bank of England Working Paper No. 816). https://doi.org/10.2139/ssrn.3435104
Bücker, M., Szepannek, G., Gosiewska, A., & Biecek, P. (2020). Transparency, auditability and explainability of machine learning models in credit scoring. arXiv preprint arXiv:2009.13384.
Bussmann, N., Giudici, P., Marinelli, D., & Papenbrock, J. (2021). Explainable machine learning in credit risk management. Computational Economics, 57, 203–216. https://doi.org/10.1007/s10614-020-10042-0
Cameron, A. C., & Trivedi, P. K. (2005). Microeconometrics: methods and applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511811241
Campbell, J. Y., & Cocco, J. F. (2015). A model of mortgage default. The Journal of Finance, 70(4), 1495–1554. https://doi.org/10.1111/jofi.12252
Chen, J. (2018). Fair lending needs explainable models for responsible recommendation. arXiv preprint arXiv:1809.04684.
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68.
https://doi.org/10.1111/ectj.12097
Chopra, A., & Bhilare, P. (2018). Application of ensemble models in credit scoring models. Business Perspectives and Research, 6(2), 129–141. https://doi.org/10.1177/2278533718765531
Dahiya, S., Handa, S. S., & Singh, N. P. (2017). A feature selection enabled hybrid‐bagging algorithm for credit risk evaluation. Expert Systems, 34(6), e12217. https://doi.org/10.1111/exsy.12217
Datta, A., Sen, S., & Zick, Y. (2016, May). Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In 2016 IEEE Symposium on Security and Privacy (SP) (pp. 598–617). IEEE.
https://doi.org/10.1109/SP.2016.42
Elul, R., Souleles, N. S., Chomsisengphet, S., Glennon, D., & Hunt, R. (2010). What “triggers” mortgage default? American Economic Review, 100(2), 490–494.
https://doi.org/10.1257/aer.100.2.490
Fahner, G. (2018). Developing transparent credit risk scorecardsmore effectively: an explainable artificial intelligence approach. In The 7th International Conference on Data Analytics (pp. 7–14), Athens, Greece.
Feldman, D., & Gross, S. (2005). Mortgage default: classification trees analysis. The Journal of Real Estate Finance and Economics, 30(4), 369–396. https://doi.org/10.1007/s11146-005-7013-7
Fitzpatrick, T., & Mues, C. (2016). An empirical comparison of classification algorithms for mortgage default prediction: evidence from a distressed mortgage market. European Journal of Operational Research, 249(2), 427–439.
https://doi.org/10.1016/j.ejor.2015.09.014
Foote, C. L., Gerardi, K., & Willen, P. S. (2008). Negative equity and foreclosure: theory and evidence. Journal of Urban Economics, 64(2), 234–245. https://doi.org/10.1016/j.jue.2008.07.006
Fuster, A., Goldsmith-Pinkham, P., Ramadorai, T., & Walther, A. (2020, October 1). Predictably unequal? The effects of machine learning on credit markets (Working paper).
Galindo, J., & Tamayo, P. (2000). Credit risk assessment using statistical and machine learning: basic methodology and risk modeling applications. Computational Economics, 15(1), 107– 143. https://doi.org/10.1023/A:1008699112516
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015). Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics, 24(1), 44–65.
https://doi.org/10.1080/10618600.2014.907095
Kabul, I. K. (2017, December 18). Interpretability is crucial for trusting AI and machine learning. The SAS Data Science Blog. https://blogs.sas.com/content/subconsciousmusings/2017/12/18/interpretability-crucial-trusting-ai-machine-learning/
Koutanaei, F. N., Sajedi, H., & Khanbabaei, M. (2015). A hybrid data mining model of feature selection algorithms and ensemble learning classifiers for credit scoring. Journal of Retailing and Consumer Services, 27, 11–23.
https://doi.org/10.1016/j.jretconser.2015.07.003
Kvamme, H., Sellereite, N., Aas, K., & Sjursen, S. (2018). Predicting mortgage default using convolutional neural networks. Expert Systems with Applications, 102, 207–217. https://doi.org/10.1016/j.eswa.2018.02.029
Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765–4774.
Mullainathan, S., & Spiess, J. (2017). Machine learning: an applied econometric approach. Journal of Economic Perspectives, 31(2), 87–106. https://doi.org/10.1257/jep.31.2.87
Nanni, L., & Lumini, A. (2009). An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring. Expert Systems with Applications, 36(2), 3028–3033. https://doi.org/10.1016/j.eswa.2008.01.018
Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3), 54–60. https://doi.org/10.1145/3241036
Pławiak, P., Abdar, M., & Acharya, U. R. (2019). Application of new deep genetic cascade ensemble of SVM classifiers to predict the Australian credit scoring. Applied Soft Computing, 84, 105740. https://doi.org/10.1016/j.asoc.2019.105740
Preece, A., Harborne, D., Braines, D., Tomsett, R., & Chakraborty, S. (2018). Stakeholders in explainable AI. arXiv preprint arXiv:1810.00184.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
https://doi.org/10.1038/s42256-019-0048-x
Sealand, J. C. (2018). Short-term prediction of mortgage default using ensembled machine learning models [Unpublished doctoral dissertation]. Slippery Rock University.
Shead, S. (2020, August 21). How a computer algorithm caused a grading crisis in British schools. CNBC. https://www.cnbc.com/2020/08/21/computer-algorithm-caused-a-grading-crisis-in-british-schools.html
Sirignano, J., Sadhwani, A., & Giesecke, K. (2016). Deep learning for mortgage risk. SSRN Electronic Journal.
Steinwart, I., Hush, D., & Scovel, C. (2009). Learning from dependent observations. Journal of Multivariate Analysis, 100(1), 175–194. https://doi.org/10.1016/j.jmva.2008.04.001
Štrumbelj, E., & Kononenko, I. (2011, April). A general method for visualizing and explaining black-box regression models. In International Conference on Adaptive and Natural Computing Algorithms (pp. 21–30). Springer. https://doi.org/10.1007/978-3-642-20267-4_3
Torrent, N. L., Visani, G., & Bagli, E. (2020). PSD2 explainable AI model for credit scoring. arXiv preprint arXiv:2011.10367.
Tsai, C. F., & Wu, J. W. (2008). Using neural network ensembles for bankruptcy prediction and credit scoring. Expert Systems with Applications, 34(4), 2639–2649. https://doi.org/10.1016/j.eswa.2007.05.019
Wallace, N. (2005). Innovations in mortgage modeling: an introduction. Real Estate Economics, 33(4), 587–594.
https://doi.org/10.1111/j.1540-6229.2005.00131.x
Wang, G., Ma, J., Huang, L., & Xu, K. (2012). Two credit scoring models based on dual strategy ensemble trees. KnowledgeBased Systems, 26, 61–68. https://doi.org/10.1016/j.knosys.2011.06.020
Wang, J., Wiens, J., & Lundberg, S. (2021, March). Shapley flow: a graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics (pp. 721–729). PMLR.
Zhao, Q., & Hastie, T. (2021). Causal interpretations of blackbox models. Journal of Business & Economic Statistics, 39(1), 272–281. https://doi.org/10.1080/07350015.2019.1624293