python评分卡建模作用 python 评分卡

转载

mob64ca14040d22 2023-09-06 14:30:24

文章标签 python评分卡建模作用 python 机器学习缺失值数据 文章分类 Python 后端开发

评分卡之数据预处理：重复值、填补缺失值、异常值与数据不平衡

在银行借贷场景中，评分卡是一种以分数形式来衡量一个客户的信用风险大小的手段，它衡量向别人借钱的人（受信人，需要融资的公司）不能如期履行合同中的还本付息责任，并让借钱给别人的人（授信人，银行等金融机构）造成经济损失的可能性。一般来说，评分卡打出的分数越高，客户的信用越好，风险越小。

Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 7.6.1 -- An enhanced Interactive Python.

1.导入库并获取数据

%matplotlib inline
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression as LR

data = pd.read_csv(r"H:\程志伟\python\rankingcard.csv",index_col=0)

2.数据探索与数据预处理
#观察数据类型

data.head()
Out[3]:
SeriousDlqin2yrs ... NumberOfDependents
1 1 ... 2.0
2 0 ... 1.0
3 0 ... 0.0
4 0 ... 0.0
5 0 ... 0.0

[5 rows x 11 columns]

#观察数据结构

data.shape
Out[4]: (150000, 11)

data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 150000 entries, 1 to 150000
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 150000 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 150000 non-null float64
2 age 150000 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 150000 non-null int64
4 DebtRatio 150000 non-null float64
5 MonthlyIncome 120269 non-null float64
6 NumberOfOpenCreditLinesAndLoans 150000 non-null int64
7 NumberOfTimes90DaysLate 150000 non-null int64
8 NumberRealEstateLoansOrLines 150000 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 150000 non-null int64
10 NumberOfDependents 146076 non-null float64
dtypes: float64(4), int64(7)
memory usage: 13.7 MB

数据字段描述

SeriousDlqin2yrs 出现 90 天或更长时间的逾期行为（即定义好坏客户）
RevolvingUtilizationOfUnsecuredLines 贷款以及信用卡可用额度与总额度比例
age 借款人借款年龄
NumberOfTime30-59DaysPastDueNotWorse 过去两年内出现35-59天逾期但是没有发展得更坏的次数
DebtRatio 每月偿还债务，赡养费，生活费用除以月总收入
MonthlyIncome 月收入
NumberOfOpenCreditLinesAndLoans 开放式贷款和信贷数量
NumberOfTimes90DaysLate 过去两年内出现90天逾期或更坏的次数
NumberRealEstateLoansOrLines 抵押贷款和房地产贷款数量，包括房屋净值信贷额度
NumberOfTime60-89DaysPastDueNotWorse 过去两年内出现60-89天逾期但是没有发展得更坏的次数
NumberOfDependents 家庭中不包括自身的家属人数（配偶，子女等）

2.1 去重数据
#去除重复值

data.drop_duplicates(inplace=True)

data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 149391 entries, 1 to 150000
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 145563 non-null float64
dtypes: float64(4), int64(7)
memory usage: 13.7 MB

#删除之后千万不要忘记，恢复索引，标红部分发生变化

data.index = range(data.shape[0])

data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 145563 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB

2.2 填补缺失值

查看每列缺失值的情况

data.isnull().sum()
Out[10]:
SeriousDlqin2yrs 0
RevolvingUtilizationOfUnsecuredLines 0
age 0
NumberOfTime30-59DaysPastDueNotWorse 0
DebtRatio 0
MonthlyIncome 29221
NumberOfOpenCreditLinesAndLoans 0
NumberOfTimes90DaysLate 0
NumberRealEstateLoansOrLines 0
NumberOfTime60-89DaysPastDueNotWorse 0
NumberOfDependents 3828
dtype: int64

#查看缺失值的比例

data.isnull().sum()/data.shape[0]
Out[11]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.025624
dtype: float64

data.isnull().mean()
Out[12]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.025624
dtype: float64

使用均值填补“家属人数”

data["NumberOfDependents"].fillna(int(data["NumberOfDependents"].mean()),inplace=True)

data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 120170 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 149391 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB

#填补之后在观测每列的数据缺失比例

data.isnull().sum()/data.shape[0]
Out[15]:
SeriousDlqin2yrs 0.000000
RevolvingUtilizationOfUnsecuredLines 0.000000
age 0.000000
NumberOfTime30-59DaysPastDueNotWorse 0.000000
DebtRatio 0.000000
MonthlyIncome 0.195601
NumberOfOpenCreditLinesAndLoans 0.000000
NumberOfTimes90DaysLate 0.000000
NumberRealEstateLoansOrLines 0.000000
NumberOfTime60-89DaysPastDueNotWorse 0.000000
NumberOfDependents 0.000000
dtype: float64

使用随机森林填补一个特征的缺失值的函数

def fill_missing_rf(X,y,to_fill):
"""
使用随机森林填补一个特征的缺失值的函数
参数：
X：要填补的特征矩阵
y：完整的，没有缺失值的标签
to_fill：字符串，要填补的那一列的名称

X_train 特征T不缺失的值：

Y_train 特征T缺失的值对应的其他n-1个特征 + 本来的标签：

X_test 特征T缺失的值：未知，

我们需要预测的Y_test

"""
#构建我们的新特征矩阵和新标签
df = X.copy()
fill = df.loc[:,to_fill]
df = pd.concat([df.loc[:,df.columns != to_fill],pd.DataFrame(y)],axis=1)

#找出我们的训练集和测试集
Ytrain = fill[fill.notnull()]
Ytest = fill[fill.isnull()]
Xtrain = df.iloc[Ytrain.index,:]
Xtest = df.iloc[Ytest.index,:]

#用随机森林回归来填补缺失值
from sklearn.ensemble import RandomForestRegressor as rfr
rfr = rfr(n_estimators=100)
rfr = rfr.fit(Xtrain, Ytrain)
Ypredict = rfr.predict(Xtest)

return Ypredict

将参数导入函数，产出结果：

X = data.iloc[:,1:]

y = data["SeriousDlqin2yrs"]

X.shape
Out[20]: (149391, 10)

y_pred = fill_missing_rf(X,y,"MonthlyIncome")

y_pred.shape
Out[22]: (29221,)

data.loc[data.loc[:,"MonthlyIncome"].isnull(),"MonthlyIncome"].shape
Out[23]: (29221,)

将预测数据填回到数据集中

data.loc[data.loc[:,"MonthlyIncome"].isnull(),"MonthlyIncome"] = y_pred

再次查看数据的缺失情况

data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149391 entries, 0 to 149390
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149391 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149391 non-null float64
2 age 149391 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149391 non-null int64
4 DebtRatio 149391 non-null float64
5 MonthlyIncome 149391 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149391 non-null int64
7 NumberOfTimes90DaysLate 149391 non-null int64
8 NumberRealEstateLoansOrLines 149391 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149391 non-null int64
10 NumberOfDependents 149391 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB

2.3 描述性统计处理异常值

data.describe([0.01,0.1,0.25,.5,.75,.9,.99]).T
Out[26]:
count ... max
SeriousDlqin2yrs 149391.0 ... 1.0
RevolvingUtilizationOfUnsecuredLines 149391.0 ... 50708.0
age 149391.0 ... 109.0
NumberOfTime30-59DaysPastDueNotWorse 149391.0 ... 98.0
DebtRatio 149391.0 ... 329664.0
MonthlyIncome 149391.0 ... 3008750.0
NumberOfOpenCreditLinesAndLoans 149391.0 ... 58.0
NumberOfTimes90DaysLate 149391.0 ... 98.0
NumberRealEstateLoansOrLines 149391.0 ... 54.0
NumberOfTime60-89DaysPastDueNotWorse 149391.0 ... 98.0
NumberOfDependents 149391.0 ... 20.0

[11 rows x 12 columns]

#异常值也被我们观察到，年龄的最小值居然有0，这不符合银行的业务需求，即便是儿童账户也要至少8岁，我们可以
(data["age"] == 0).sum()
Out[27]: 1

#发现只有一个人年龄为0，可以判断这肯定是录入失误造成的，可以当成是缺失值来处理，直接删除掉这个样本
data = data[data["age"] != 0]

data.shape
Out[29]: (149390, 11)

"""
另外，有三个指标看起来很奇怪：
"NumberOfTime30-59DaysPastDueNotWorse"
"NumberOfTime60-89DaysPastDueNotWorse"
"NumberOfTimes90DaysLate"
这三个指标分别是“过去两年内出现35-59天逾期但是没有发展的更坏的次数”，“过去两年内出现60-89天逾期但是没
有发展的更坏的次数”,“过去两年内出现90天逾期的次数”。这三个指标，在99%的分布的时候依然是2，最大值却是
98，看起来非常奇怪。一个人在过去两年内逾期35~59天98次，一年6个60天，两年内逾期98次这是怎么算出来的？
我们可以去咨询业务人员，请教他们这个逾期次数是如何计算的。如果这个指标是正常的，那这些两年内逾期了98次的
客户，应该都是坏客户。在我们无法询问他们情况下，我们查看一下有多少个样本存在这种异常：
"""

data[data.loc[:,"NumberOfTimes90DaysLate"] > 90].count()
Out[30]:
SeriousDlqin2yrs 225
RevolvingUtilizationOfUnsecuredLines 225
age 225
NumberOfTime30-59DaysPastDueNotWorse 225
DebtRatio 225
MonthlyIncome 225
NumberOfOpenCreditLinesAndLoans 225
NumberOfTimes90DaysLate 225
NumberRealEstateLoansOrLines 225
NumberOfTime60-89DaysPastDueNotWorse 225
NumberOfDependents 225
dtype: int64

#有225个样本存在这样的情况，并且这些样本，我们观察一下，标签并不都是1，他们并不都是坏客户。因此，我们基
本可以判断，这些样本是某种异常，应该把它们删除。

data = data[data.loc[:,"NumberOfTimes90DaysLate"] < 90]

data.index = range(data.shape[0])

data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 149165 entries, 0 to 149164
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 SeriousDlqin2yrs 149165 non-null int64
1 RevolvingUtilizationOfUnsecuredLines 149165 non-null float64
2 age 149165 non-null int64
3 NumberOfTime30-59DaysPastDueNotWorse 149165 non-null int64
4 DebtRatio 149165 non-null float64
5 MonthlyIncome 149165 non-null float64
6 NumberOfOpenCreditLinesAndLoans 149165 non-null int64
7 NumberOfTimes90DaysLate 149165 non-null int64
8 NumberRealEstateLoansOrLines 149165 non-null int64
9 NumberOfTime60-89DaysPastDueNotWorse 149165 non-null int64
10 NumberOfDependents 149165 non-null float64
dtypes: float64(4), int64(7)
memory usage: 12.5 MB

2.4 样本不均衡问题

X = data.iloc[:,1:]
y = data.iloc[:,0]

y.value_counts()
Out[35]:
0 139292
1 9873
Name: SeriousDlqin2yrs, dtype: int64

可以看出，样本严重不均衡

n_sample = X.shape[0]
n_1_sample = y.value_counts()[1]
n_0_sample = y.value_counts()[0]
print('样本个数：{}; 1占{:.2%}; 0占{:.2%}'.format(n_sample,n_1_sample/n_sample,n_0_sample/n_sample))

样本个数：149165; 1占6.62%; 0占93.38%

使用上采样方法来平衡样本

import imblearn

from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=42) #实例化
X,y = sm.fit_sample(X,y)
n_sample_ = X.shape[0]
pd.Series(y).value_counts()
Out[38]:
1 139292
0 139292
dtype: int64

n_1_sample = pd.Series(y).value_counts()[1]
n_0_sample = pd.Series(y).value_counts()[0]
print('样本个数：{}; 1占{:.2%}; 0占{:.2%}'.format(n_sample_,n_1_sample/n_sample_,n_0_sample/n_sample_))
样本个数：278584; 1占50.00%; 0占50.00%

我们就实现了样本平衡，样本量也增加了

2.5 分训练集和测试集

from sklearn.model_selection import train_test_split
X = pd.DataFrame(X)
y = pd.DataFrame(y)
X_train, X_vali, Y_train, Y_vali = train_test_split(X,y,test_size=0.3,random_state=420)

model_data = pd.concat([Y_train, X_train], axis=1)
model_data.index = range(model_data.shape[0])
model_data.columns = data.columns

vali_data = pd.concat([Y_vali, X_vali], axis=1)
vali_data.index = range(vali_data.shape[0])
vali_data.columns = data.columns

model_data.to_csv(r"H:\程志伟\python\model_data.csv")

vali_data.to_csv(r"H:\程志伟\python\\vali_data.csv")

作为评分卡的数据预处理完成。

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。