Combined Feature Selection Scheme for Banking Modeling

Machine learning methods have been successful in various aspects of bank lending. Banks have accumulated huge amounts of data about borrowers over the years of application. On the one hand, this made it possible to predict borrower behavior more accurately, on the other, it gave rise to the problem a problem of data redundancy, which greatly complicates the model development. Methods of feature selection, which allows to improve the quality of models, are apply to solve this problem. Feature selection methods can be divided into three main types: filters, wrappers, and embedded methods. Filters are simple and time-efficient methods that may help discover one-dimensional relations. Wrappers and embedded methods are more effective in feature selection, because they account for multi-dimensional relationships, but these methods are resource-consuming and may fail to process large samples with many features. In this article, the authors propose a combined feature selection scheme (CFSS), in which the first stages of selection use coarse filters, and on the final — wrappers for high-quality selection. This architecture lets us increase the quality of selection and reduce the time necessary to process large multi-dimensional samples, which are used in the development of industrial models. Experiments conducted by authors for four types of bank modelling tasks (survey scoring, behavioral scoring, customer response to cross-selling, and delayed debt collection) have shown that the proposed method better than classical methods containing only filters or only wrappers.

1. Guyon I., Elisseeff A. An introduction to variable and feature selection. Journal of Machine Learning Research. 2003;3(7–8):1157–1182. DOI: 10.1162/153244303322753616

2. Hamon J. Optimisation combinatoire pour la sélection de variables en régression en grande dimension: Application en génétique animale. Docteur en Informatique Thèse. Lille: Université des Sciences et Technologie de Lille; 2013. 160 p. URL: https://core.ac.uk/download/pdf/51213307.pdf

3. Shen C., Zhang K. Two-stage improved Grey Wolf optimization algorithm for feature selection on highdimensional classification. Complex & Intelligent Systems. 2022;8(4):2769–2789. DOI: 10.1007/s40747–021–00452–4

4. Basak H., Das M., Modak S. RSO: A novel reinforced swarm optimization algorithm for feature selection.arXiv:2107.14199. URL: https://arxiv.org/pdf/2107.14199.pdf

5. Roffo G., Melzi S. Features selection via eigenvector centrality. In: Proc. 5th Int. workshop on new frontiers in mining complex patterns (NFMCP2016). (Riva del Garda, 19 September, 2016). Cham: Springer-Verlag; 2017. (Lecture Notes in Computer Science. Vol. 10312). URL: https://www.researchgate.net/publication/305918391_Feature_Selection_via_Eigenvector_Centrality

6. Hall M. A. Correlation-based feature selection for machine learning. PhD thesis. Hamilton: The University of Waikato; 1999. 198 p. URL: https://www.lri.fr/~pierres/donn%E9es/save/these/articles/lpr-queue/hall99correlationbased.pdf

7. James G., Witten D., Hastie T., Tibshirani R. An introduction to statistical learning: With applications in R. 8th ed. New York, NY: Springer Science+Business Media; 2017. 440 p. (Springer Texts in Statistics).

8. Janitza S., Celik E., Boulesteix A.-L. A computationally fast variable importance test for random forests for highdimensional data. Advances in Data Analysis and Classification. 2018;12(4):885–915. DOI: 10.1007/s11634–016–0276–4

9. Магнус Я. Р., Катышев П. К., Пересецкий А. А. Эконометрика. М.: Дело; 2004. 576 с.

10. Magnus Ya.R., Katyshev P. K., Peresetskii A. A. Econometrics. Moscow: Delo; 2004. 576 p. (In Russ.).

11. Pearson K. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science. 1901;2(11):559–572. DOI: 10.1080/14786440109462720

12. Айвазян С. А., Бухштабер В. М., Енюков И. С., Мешалкин Л. Д. Прикладная статистика. Классификация и снижение размерности. М.: Финансы и статистика; 1989. 607 с.

13. Aivazyan S. A., Bukhshtaber V. M., Enyukov I. S., Meshalkin L. D. Applied statistics. Classification and dimensionality reduction. Moscow: Finansy i statistika; 1989. 607 p. (In Russ.).

14. Zhang Y., Dong Z., Phillips P., Wang S., Ji G., Yang J., Yuan T.-F. Detection of subjects and brain regions related to Alzheimer’s disease using 3D MRI scans based on eigenbrain and machine learning. Frontiers in Computational Neuroscience. 2015;9:66. DOI: 10.3389/fncom.2015.00066

15. Hocking R. R. The analysis and selection of variables in linear regression. Biometrics. 1976;32(1):1–49. DOI: 10.2307/2529336

16. Mirjalili S., Mirjalili S. M., Lewis A. Grey wolf optimizer. Advances in Engineering Software. 2014;69:46–61. DOI: 10.1016/j.advengsoft.2013.12.007

17. Flom P. L., Cassell D. L. Stopping stepwise: Why stepwise and similar selection methods are bad, and what you should use. In: Northeast SAS Users Group 2007 (NESUG 2007). (Baltimore, 11–14 November, 2007). URL: https://www.lexjansen.com/pnwsug/2008/DavidCassell-StoppingStepwise.pdf

18. Eberhart R., Kennedy J. A new optimizer using particle swarm theory. In: Proc. 6th Int. symp. on micro machine and human science (MHS’95). (Nagoya, 04–06 October, 1995). Piscataway, NJ: IEEE; 1995:39–43. DOI: 10.1109/MHS.1995.494215

19. Schott J. R. Fault tolerant design using single and multicriteria genetic algorithm optimization. PhD thesis. Cambridge, MA: Massachusetts Institute of Technology; 1995. 201 p. URL: https://dspace.mit.edu/handle/1721.1/11582

20. Karaboga D. An idea based on honey bee swarm for numerical optimization. Technical Report. 2005;(06). URL: https://abc.erciyes.edu.tr/pub/tr06_2005.pdf

21. Altmann A., Toloşi L., Sander O., Lengauer T. Permutation importance: A corrected feature importance measure. Bioinformatics. 2010;26(10):1340–1347. DOI: 10.1093/bioinformatics/btq134

22. Hapfelmeier A., Ulm K. A new variable selection approach using random forests. Computational Statistics & Data Analysis. 2013;60:50–69. DOI: 10.1016/j.csda.2012.09.020

23. Louzada F., Ara A., Fernandes G. B. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science. 2016;21(2):117–134. DOI: 10.1016/j.sorms.2016.10.001

24. Santosa F., Symes W. W. Linear inversion of band-limited refl ection seismograms. SIAM Journal on Scientifi c and Statistical Computing. 1986;7(4):1307–1330. DOI: 10.1137/0907087

25. Hilt D. E., Seegrist D. W. Ridge: A computer program for calculating ridge regression estimates. USDA Forest Service Research Note. 1977;(236). URL: https://ia803007.us.archive.org/23/items/ridgecomputerpro236hilt/ridgecomputerpro236hilt.pdf

26. Тихонов А. Н. О решении некорректно поставленных задач и методе регуляризации. Доклады Академии наук СССР. 1963;151(3):501–504.

27. Tikhonov A. N. Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics. Doklady. 1963;(4):1035–1038. (In Russ.: Doklady Akademii nauk SSSR. 1963;151(3):501–504.).

28. Воронцов К. В. Лекции по алгоритмам восстановления регрессии. 21 декабря 2007 г. URL: http://www.ccas.ru/voron/download/Regression.pdf

29. Vorontsov K. V. Lectures on regression recovery algorithms. December 21, 2007. URL: http://www.ccas.ru/voron/download/Regression.pdf (In Russ.).

30. Zou H., Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society. Series B: Statistical Methodology. 2005;67(2):301–320. DOI: 10.1111/j.1467–9868.2005.00503.x

31. Брюс П., Брюс Э. Разведочный анализ данных. Практическая статистика для специалистов Data Science. Пер. с англ. СПб.: БХВ-Петербург; 2018:19–58.

32. Bruce P., Bruce A. Exploratory data analysis. In: Practical statistics for data scientists: 50 essential concepts. Beijing: O’Reilly Media; 2017;1–46. (Russ. ed.: Bruce P., Bruce A. Razvedochnyi analiz dannykh. Prakticheskaya statistika dlya spetsialistov Data Science. St. Petersburg: BHV-Peterburg; 2018:19–58.).

33. Afanasiev S., Smirnova A. Predictive fraud analytics: B-tests. Journal of Operational Risk. 2018;13(4):17–46. DOI: 10.21314/JOP.2018.213

34. Lin J. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory. 1991;37(1):145–151. DOI: 10.1109/18.61115

35. Kolmogorov A. Sulla determinazione empirica di una legge di distribuzione. Giornale dell’Istituto Italiano degli Attuari. 1933;4:83–91.

36. Harris D., Harris S. Digital design and computer architecture. 2nd ed. San Francisco, CA: Morgan Kaufmann; 2012. 720 p.

37. Kutner M. H., Nachtsheim C. J.; Neter J. Applied linear regression models. 4th ed. New York, NY: McGraw-Hill/Irwin; 2004. 701 p.