Scikit-Learn集成机器学习：boosting（4万字详解，收藏）

目录¶

• 介绍

• GradientBoostingRegressor

• GradientBossting分类器

• AdaBoost 回归器

• AdaBoost分类器

• 参考

  
  

介绍¶

Boosting 是一种集成学习，我们按顺序训练估计器，而不是并行训练所有估计器。我们尝试创建一些快速简单（弱但比随机猜测更好）的模型，然后结合所有弱估计器的结果来做出最终预测。我们已经在bagging & random forests教程中讨论了另一种集成学习方法。如果您想了解它，请随时浏览它。

Scikit-learn 为分类和回归问题提供了两种不同的增强算法：

• 梯度树提升（梯度提升决策树） - 它迭代地构建学习器，弱学习器在预测错误的样本错误上进行训练。它最初从一个学习者开始，然后迭代地添加学习者。它试图通过迭代地添加新树来最小化损失。它使用决策树是弱估计器。Scikit-learn 提供了两个类，用于实现​​Gradient Tree Boosting​​分类和回归问题。

  
  

  • 梯度提升分类器​GradientBoostingClassifier​

  • ​梯度提升回归器​GradientBoostingRegressor

    
    

• Adaptive Boost - 它在修改后的数据上迭代地拟合弱估计器列表。然后它根据加权投票组合所有估计器的结果以生成最终结果。在每次迭代中，将高权重分配给在前一次迭代中预测错误的样本，并为那些在前一次迭代中预测正确的样本减少权重。这使模型能够专注于出错的样本。最初，所有样本都被分配相同的权重（1/ n_samples）。它让我们指定用于该过程的估计器。Scikit-learn 提供了两个类，用于实现​​Adaptive Boosting​​分类和回归问题。

  
  

  • AdaBoost分类器

  • AdaBoost 回归器

这结束了我们对 Boosting 过程的简短介绍。我们现在将从编码部分开始。

我们将从导入必要的库开始。

import numpy as np
import pandas as pd
import sklearn
import warnings

warnings.filterwarnings("ignore")
np.set_printoptions(precision=3)
%matplotlib inline

加载数据集¶

为了我们的目的，我们将加载下面提到的两个。

• 数字数据集：我们将使用具有数字大小图像的数字数据​​8x8​​集​​0-9​​。我们将在下面的分类任务中使用数字数据。

• 波士顿住房数据集：我们将使用波士顿住房数据集，其中包含有关各种房屋属性的信息，例如平均房间数量、城镇人均犯罪率等。我们将使用它进行回归任务。

  
  

Sklearn 提供这两个数据集作为​​datasets​​​模块的一部分。我们可以通过调用​​load_digits()​​​和​​load_boston()​​​方法来加载它们。它返回类似字典的对象​​BUNCH​​，可用于检索特征和目标。

from sklearn.datasets import load_boston, load_digits
digits = load_digits()
X_digits, Y_digits = digits.data, digits.target
print('Dataset Size : ',X_digits.shape, Y_digits.shape)

数据集大小 : (1797, 64) (1797,)

boston = load_boston()
X_boston, Y_boston = boston.data, boston.target
print('Dataset Size : ',X_boston.shape, Y_boston.shape)

Dataset Size :  (506, 13) (506,)

GradientBoostingRegressor¶

可​​GradientBoostingRegressor​​​  作为​​ensemble​​sklearn 模块的一部分使用。我们将使用波士顿住房数据训练默认模型，然后通过尝试各种超参数设置来调整模型以提高其性能。我们还将它与其他回归估计器进行比较，以检查其相对于其他机器学习模型的性能。

from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_boston, Y_boston, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

训练/测试大小：(404, 13) (102, 13) (404,) (102,)

from sklearn.ensemble import GradientBoostingRegressor
grad_boosting_regressor = GradientBoostingRegressor()
grad_boosting_regressor.fit(X_train, Y_train)

GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,
                          learning_rate=0.1, loss='ls', max_depth=3,
                          max_features=None, max_leaf_nodes=None,
                          min_impurity_decrease=0.0, min_impurity_split=None,
                          min_samples_leaf=1, min_samples_split=2,
                          min_weight_fraction_leaf=0.0, n_estimators=100,
                          n_iter_no_change=None, presort='auto',
                          random_state=None, subsample=1.0, tol=0.0001,
                          validation_fraction=0.1, verbose=0, warm_start=False)

Y_preds = grad_boosting_regressor.predict(X_test)

print(Y_preds[:15])
print(Y_test[:15])

print('Test R^2 Score : %.3f'%grad_boosting_regressor.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training R^2 Score : %.3f'%grad_boosting_regressor.score(X_train, Y_train))

[33.731 26.108 48.711 18.784 31.065 43.077 25.474  9.03  18.201 29.294

22.577 19.06  15.871 24.611 19.605]

[15.  26.6 45.4 20.8 34.9 21.9 28.7  7.2 20.  32.2 24.1 18.5 13.5 27.

23.1]

Test R^2 Score : 0.812

Training R^2 Score : 0.979

​​GradientBoostingRegressor​​的重要属性

以下是一些重要属性，​​GradientBoostingRegressor​​一旦模型经过训练，这些属性就可以提供重要信息。

• ​​feature_importances_​​- 它返回一个浮点数组，表示数据集中每个特征的重要性。
• ​​estimators_​​- 它返回训练有素的估计器。
• ​​oob_improvement_​​- 它返回大小数组 (n_estimators,)。数组中的每个值都表示袋外样本相对于前一次迭代的损失有所改善。
• ​​loss_​​- 它返回损失函数作为对象。

print("Feature Importances : ", grad_boosting_regressor.feature_importances_)

Feature Importances :  [1.551e-02 3.064e-04 1.025e-03 4.960e-05 3.208e-02 4.883e-01 1.295e-02

5.908e-02 1.602e-03 1.187e-02 2.461e-02 5.472e-03 3.472e-01]

print("Estimators Shape: ", grad_boosting_regressor.estimators_.shape)
grad_boosting_regressor.estimators_[:2]

Estimators Shape:  (100, 1)

array([[DecisionTreeRegressor(criterion='friedman_mse', max_depth=3, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort='auto',

random_state=RandomState(MT19937) at 0x7F0EF00E7780,

splitter='best')],

[DecisionTreeRegressor(criterion='friedman_mse', max_depth=3, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort='auto',

random_state=RandomState(MT19937) at 0x7F0EF00E7780,

splitter='best')]], dtype=object)

print("Loss : ", grad_boosting_regressor.loss_)

Loss :  <sklearn.ensemble._gb_losses.LeastSquaresError object at 0x7f0eac9d4b00>

通过对各种超参数进行网格搜索来微调模型¶

以下是需要调整以获得最适合我们数据的常见超参数列表。我们将尝试对训练/测试数据的各种拆分进行各种超参数设置，以找出最佳拟合，这对于训练和测试数据集具有几乎相同的精度，或者在精度之间的差异非常小。

• learning_rate - 它缩小了每棵树的贡献。learning_rate 和 n_estimatros 之间存在权衡。
• n_estimators - 其结果将被组合以产生最终预测的基本估计器的数量。 ​​default=100​​
• max_depth - 单个树的最大深度。我们需要找到最佳价值。​​default=3​​
• min_samples_split - 拆分内部节点所需的样本数。它接受​​int(0-n_samples)​​,​​float(0.0-0.5]​​值。Float 采用 ceil(min_samples_split * n_samples) 特征。​​default=2​​
• min_samples_leaf - 叶节点所需的最小样本数。它接受​​int(0-n_samples)​​,​​float(0.0-0.5]​​值。Float 采用 ceil(min_samples_leaf * n_samples) 特征。​​default=1​​
• 标准- 我们算法试图最小化的成本函数。目前它支持 mse（均方误差）和 mae（平均绝对误差）。​​default=friedman_mse​​
• max_features - 进行拆分时要考虑的特征数。它接受​​int(0-n_features)​​, ​​float(0.0-0.5]​​,​​string(sqrt, log2, auto) or None​​作为值。​​default=None​​

• None - 如果提供 None，则 n_features 用作值。
• sqrt - sqrt(n_features) 特征用于分割。
• auto - sqrt(n_features) 特征用于分割。
• log2 - log2(n_features) 特征用于分割。

validation_fraction - 它是指用于早期停止验证的训练数据的比例。它接受​​float(0.0,1.0)​​​ ​​default=0.1​​

我们将在下面尝试上述超参数的各种值，通过对数据进行 3 折交叉验证来找到我们数据集的最佳估计器。

%%time

from sklearn.model_selection import GridSearchCV

n_samples = X_boston.shape[0]
n_features = X_boston.shape[1]

params = {'n_estimators': np.arange(100, 301, 50),
          'max_depth': [None, 3, 5,],
          'min_samples_split': [2, 0.3, 0.5, n_samples//2, ],
          'min_samples_leaf': [1, 0.3, 0.5, n_samples//2, ],
          'criterion': ['friedman_mse', 'mae'],
          'max_features': [None, 'sqrt', 'auto', 'log2', 0.3, 0.7, n_features//2, ],
         }

grad_boost_regressor_grid = GridSearchCV(GradientBoostingRegressor(random_state=1), param_grid=params, n_jobs=-1, cv=3, verbose=5)
grad_boost_regressor_grid.fit(X_train,Y_train)

print('Train R^2 Score : %.3f'%grad_boost_regressor_grid.best_estimator_.score(X_train, Y_train))
print('Test R^2 Score : %.3f'%grad_boost_regressor_grid.best_estimator_.score(X_test, Y_test))
print('Best R^2 Score Through Grid Search : %.3f'%grad_boost_regressor_grid.best_score_)
print('Best Parameters : ',grad_boost_regressor_grid.best_params_)

Fitting 3 folds for each of 3360 candidates, totalling 10080 fits

[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.

[Parallel(n_jobs=-1)]: Done  10 tasks      | elapsed:   27.6s

[Parallel(n_jobs=-1)]: Done 134 tasks      | elapsed:   29.9s

[Parallel(n_jobs=-1)]: Done 854 tasks      | elapsed:   38.2s

[Parallel(n_jobs=-1)]: Done 1662 tasks      | elapsed:   44.9s

[Parallel(n_jobs=-1)]: Done 2958 tasks      | elapsed:   55.1s

[Parallel(n_jobs=-1)]: Done 4542 tasks      | elapsed:  1.1min

[Parallel(n_jobs=-1)]: Done 5503 tasks      | elapsed:  1.8min

[Parallel(n_jobs=-1)]: Done 6281 tasks      | elapsed:  2.8min

[Parallel(n_jobs=-1)]: Done 7186 tasks      | elapsed:  3.5min

[Parallel(n_jobs=-1)]: Done 8124 tasks      | elapsed:  4.2min

[Parallel(n_jobs=-1)]: Done 9301 tasks      | elapsed:  5.3min

[Parallel(n_jobs=-1)]: Done 10080 out of 10080 | elapsed:  5.9min finished

Train R^2 Score : 0.997

Test R^2 Score : 0.776

Best R^2 Score Through Grid Search : 0.891

Best Parameters :  {'criterion': 'friedman_mse', 'max_depth': None, 'max_features': None, 'min_samples_leaf': 1, 'min_samples_split': 0.3, 'n_estimators': 150}

CPU times: user 5.14 s, sys: 254 ms, total: 5.39 s

Wall time: 5min 53s

输出前几个交叉验证结果¶

cross_val_results = pd.DataFrame(grad_boost_regressor_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))

cross_val_results.head() ## Printing first few results.

Number of Various Combinations of Parameters Tried : 3360

比较梯度提升与 Bagging、随机森林、额外树、决策树和额外树的性能

from sklearn import ensemble, tree
## Gradient Boosting Regressor with Default Params
gb_regressor = ensemble.GradientBoostingRegressor(random_state=1)
gb_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_regressor.__class__.__name__,
                                                     gb_regressor.score(X_train, Y_train),gb_regressor.score(X_test, Y_test)))

## Above Hyper-perameter tuned Gradient Boosting Regressor
gb_regressor = ensemble.GradientBoostingRegressor(random_state=1, **grad_boost_regressor_grid.best_params_)
gb_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_regressor.__class__.__name__,
                                                     gb_regressor.score(X_train, Y_train),gb_regressor.score(X_test, Y_test)))

## Random Forest Regressor with Default Params
rforest_regressor = ensemble.RandomForestRegressor(random_state=1)
rforest_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(rforest_regressor.__class__.__name__,
                                                     rforest_regressor.score(X_train, Y_train),rforest_regressor.score(X_test, Y_test)))


## Extra Trees Regressor with Default Params
extra_forest_regressor = ensemble.ExtraTreesRegressor(random_state=1)
extra_forest_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_forest_regressor.__class__.__name__,
                                                     extra_forest_regressor.score(X_train, Y_train),extra_forest_regressor.score(X_test, Y_test)))

## Bagging Regressor with Default Params
bag_regressor = ensemble.BaggingRegressor(random_state=1)
bag_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(bag_regressor.__class__.__name__,
                                                     bag_regressor.score(X_train, Y_train),bag_regressor.score(X_test, Y_test)))


## Decision Tree with Default Parameters
dtree_regressor = tree.DecisionTreeRegressor(random_state=1)
dtree_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(dtree_regressor.__class__.__name__,
                                                     dtree_regressor.score(X_train, Y_train),dtree_regressor.score(X_test, Y_test)))

## Decision Tree with Default Parameters
extra_tree_regressor = tree.ExtraTreeRegressor(random_state=1)
extra_tree_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_tree_regressor.__class__.__name__,
                                                     extra_tree_regressor.score(X_train, Y_train),extra_tree_regressor.score(X_test, Y_test)))

GradientBoostingRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

GradientBoostingRegressor : Train Accuracy : 1.00, Test Accuracy : 0.78

RandomForestRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

ExtraTreesRegressor : Train Accuracy : 1.00, Test Accuracy : 0.83

BaggingRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

DecisionTreeRegressor : Train Accuracy : 1.00, Test Accuracy : 0.44

ExtraTreeRegressor : Train Accuracy : 1.00, Test Accuracy : 0.51

GradientBossting分类器

可​​GradientBosstingClassifier​​​  作为​​ensemble​​sklearn 模块的一部分使用。我们将使用数字数据训练默认模型，然后通过尝试各种超参数设置来调整模型以提高其性能。我们还将它与其他分类估计器进行比较，以检查其相对于其他机器学习模型的性能。

X_train, X_test, Y_train, Y_test = train_test_split(X_digits, Y_digits, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

Train/Test Sizes :  (1437, 64) (360, 64) (1437,) (360,)

from sklearn.ensemble import GradientBoostingClassifier

grad_boosting_classif = GradientBoostingClassifier()
grad_boosting_classif.fit(X_train, Y_train)

GradientBoostingClassifier(criterion='friedman_mse', init=None,

learning_rate=0.1, loss='deviance', max_depth=3,

max_features=None, max_leaf_nodes=None,

min_impurity_decrease=0.0, min_impurity_split=None,

min_samples_leaf=1, min_samples_split=2,

min_weight_fraction_leaf=0.0, n_estimators=100,

n_iter_no_change=None, presort='auto',

random_state=None, subsample=1.0, tol=0.0001,

validation_fraction=0.1, verbose=0,

warm_start=False)

Y_preds = grad_boosting_classif.predict(X_test)

print(Y_preds[:15])
print(Y_test[:15])

print('Test Accuracy : %.3f'%(Y_preds == Y_test).mean())
print('Test Accuracy : %.3f'%grad_boosting_classif.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training Accuracy : %.3f'%grad_boosting_classif.score(X_train, Y_train))

[3 3 4 4 1 3 1 0 7 4 0 6 5 1 6]

[3 3 4 4 1 3 1 0 7 4 0 0 5 1 6]

Test Accuracy : 0.956

Test Accuracy : 0.956

Training Accuracy : 1.000

​​GradientBoostingClassifier​​重要属性

​​GradientBoostingClassifier​​​具有与 相同的属性集​​GradientBoostingRegressor​​。

print("Feature Importances Shape: ", grad_boosting_classif.feature_importances_.shape)

grad_boosting_classif.feature_importances_[:10]

Feature Importances Shape:  (64,)

array([0.   , 0.001, 0.011, 0.003, 0.003, 0.059, 0.004, 0.001, 0.001,

0.002])

print("Estimators Shape : ", grad_boosting_classif.estimators_.shape)

Estimators Shape :  (100, 10)

print("Loss : ", grad_boosting_classif.loss_)

Loss :  <sklearn.ensemble._gb_losses.MultinomialDeviance object at 0x7f0eac97d4a8>

通过对各种超参数进行网格搜索来微调模型¶

​​GradientBoostingClassifier​​​几乎所有参数都与​​GradientBoostingRegressor​​

%%time

n_samples = X_digits.shape[0]
n_features = X_digits.shape[1]

params = {'n_estimators': [100, 200],
          'max_depth': [None, 2,5,],
          'min_samples_split': [2,0.5, n_samples//2, ],
          'min_samples_leaf': [1, 0.5, n_samples//2, ],
          'criterion': ['friedman_mse', 'mae'],
          'max_features': [None, 'sqrt', 'log2', 0.5, n_features//2,],
         }

grad_boost_classif_grid = GridSearchCV(GradientBoostingClassifier(random_state=1), param_grid=params, n_jobs=-1, cv=3, verbose=5)
grad_boost_classif_grid.fit(X_train,Y_train)

print('Train Accuracy : %.3f'%grad_boost_classif_grid.best_estimator_.score(X_train, Y_train))
print('Test Accuracy : %.3f'%grad_boost_classif_grid.best_estimator_.score(X_test, Y_test))
print('Best Accuracy Through Grid Search : %.3f'%grad_boost_classif_grid.best_score_)
print('Best Parameters : ',grad_boost_classif_grid.best_params_)

Fitting 3 folds for each of 540 candidates, totalling 1620 fits

[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.

[Parallel(n_jobs=-1)]: Done  10 tasks      | elapsed:    7.6s

[Parallel(n_jobs=-1)]: Done  64 tasks      | elapsed:   21.5s

[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed:   40.8s

[Parallel(n_jobs=-1)]: Done 280 tasks      | elapsed:  1.3min

[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  1.9min

[Parallel(n_jobs=-1)]: Done 640 tasks      | elapsed:  2.7min

[Parallel(n_jobs=-1)]: Done 874 tasks      | elapsed: 13.0min

[Parallel(n_jobs=-1)]: Done 1144 tasks      | elapsed: 30.8min

[Parallel(n_jobs=-1)]: Done 1450 tasks      | elapsed: 48.8min

[Parallel(n_jobs=-1)]: Done 1620 out of 1620 | elapsed: 59.1min finished

Train Accuracy : 1.000

Test Accuracy : 0.972

Best Accuracy Through Grid Search : 0.978

Best Parameters :  {'criterion': 'mae', 'max_depth': 5, 'max_features': 'log2', 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 200}

CPU times: user 50.9 s, sys: 236 ms, total: 51.2 s

Wall time: 59min 56s

输出前几个交叉验证结果¶

cross_val_results = pd.DataFrame(grad_boost_classif_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))

cross_val_results.head() ## Printing first few results.

Number of Various Combinations of Parameters Tried : 540

比较梯度提升与 Bagging、随机森林、额外树、决策树和额外树的性能¶

from sklearn import ensemble

## Gradient Boosting Regressor with Default Params
gb_classifier = ensemble.GradientBoostingClassifier(random_state=1)
gb_classifier.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_classifier.__class__.__name__,
                                                     gb_classifier.score(X_train, Y_train),gb_classifier.score(X_test, Y_test)))

## Above Hyper-perameter tuned Gradient Boosting Regressor
gb_classifier = ensemble.GradientBoostingClassifier(random_state=1, **grad_boost_classif_grid.best_params_)
gb_classifier.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_classifier.__class__.__name__,
                                                     gb_classifier.score(X_train, Y_train),gb_classifier.score(X_test, Y_test)))

## Random Forest Regressor with Default Params
rforest_classif = ensemble.RandomForestClassifier(random_state=1)
rforest_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(rforest_classif.__class__.__name__,
                                                     rforest_classif.score(X_train, Y_train),rforest_classif.score(X_test, Y_test)))


## Extra Trees Regressor with Default Params
extra_forest_classif = ensemble.ExtraTreesClassifier(random_state=1)
extra_forest_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_forest_classif.__class__.__name__,
                                                     extra_forest_classif.score(X_train, Y_train),extra_forest_classif.score(X_test, Y_test)))

## Bagging Regressor with Default Params
bag_classif = ensemble.BaggingClassifier(random_state=1)
bag_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(bag_classif.__class__.__name__,
                                                     bag_classif.score(X_train, Y_train),bag_classif.score(X_test, Y_test)))


## Decision Tree with Default Parameters
dtree_classif = tree.DecisionTreeClassifier(random_state=1)
dtree_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(dtree_classif.__class__.__name__,
                                                     dtree_classif.score(X_train, Y_train),dtree_classif.score(X_test, Y_test)))

## Decision Tree with Default Parameters
extra_tree_classif = tree.ExtraTreeClassifier(random_state=1)
extra_tree_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_tree_classif.__class__.__name__,
                                                     extra_tree_classif.score(X_train, Y_train),extra_tree_classif.score(X_test, Y_test)))

GradientBoostingClassifier : Train Accuracy : 1.00, Test Accuracy : 0.96

GradientBoostingClassifier : Train Accuracy : 1.00, Test Accuracy : 0.97

RandomForestClassifier : Train Accuracy : 1.00, Test Accuracy : 0.94

ExtraTreesClassifier : Train Accuracy : 1.00, Test Accuracy : 0.95

BaggingClassifier : Train Accuracy : 1.00, Test Accuracy : 0.94

DecisionTreeClassifier : Train Accuracy : 1.00, Test Accuracy : 0.83

ExtraTreeClassifier : Train Accuracy : 1.00, Test Accuracy : 0.83

AdaBoost 回归器¶

可​​AdaBoostRegressor​​​  作为​​ensemble​​sklearn 模块的一部分使用。我们将使用波士顿住房数据训练默认模型，然后通过尝试各种超参数设置来调整模型以提高其性能。我们还将它与其他回归估计器进行比较，以检查其相对于其他机器学习模型的性能。

X_train, X_test, Y_train, Y_test = train_test_split(X_boston, Y_boston, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

Train/Test Sizes :  (404, 13) (102, 13) (404,) (102,)

from sklearn.ensemble import AdaBoostRegressor

ada_boost_regressor = AdaBoostRegressor()
ada_boost_regressor.fit(X_train, Y_train)

AdaBoostRegressor(base_estimator=None, learning_rate=1.0, loss='linear',

n_estimators=50, random_state=None)

Y_preds = ada_boost_regressor.predict(X_test)

print(Y_preds[:15])
print(Y_test[:15])

print('Test R^2 Score : %.3f'%ada_boost_regressor.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training R^2 Score : %.3f'%ada_boost_regressor.score(X_train, Y_train))

[18.221 27.061 47.212 18.138 31.677 39.125 27.313 11.922 17.933 26.785

26.249 20.919 17.408 27.167 18.951]

[15.  26.6 45.4 20.8 34.9 21.9 28.7  7.2 20.  32.2 24.1 18.5 13.5 27.

23.1]

Test R^2 Score : 0.834

Training R^2 Score : 0.912

​AdaBoostRegressor的重要属性​

以下是一些重要属性，​​AdaBoostRegressor​​一旦模型经过训练，这些属性就可以提供重要信息。

• ​​base_estimator_​​- 它返回基估计器，从中创建由弱估计器组成的整个强估计器。

• ​​feature_importances_​​- 它返回一个浮点数组，表示数据集中每个特征的重要性。

• ​​estimators_​​- 它返回训练有素的估计器。

  
  

print("Base Estimator : ", ada_boost_regressor.base_estimator_)

Base Estimator :  DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort=False, random_state=None, splitter='best')

print("Feature Importances : ", ada_boost_regressor.feature_importances_)

Feature Importances :  [3.205e-02 0.000e+00 4.921e-03 3.564e-04 4.383e-02 2.726e-01 8.314e-03

1.300e-01 1.609e-02 5.684e-02 2.638e-02 4.679e-03 4.040e-01]

print("Estimators Shape : ", len(ada_boost_regressor.estimators_))

ada_boost_regressor.estimators_[:2]
Estimators Shape :  50

[DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort=False, random_state=280424452, splitter='best'),

DecisionTreeRegressor(criterion='mse', max_depth=3, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort=False, random_state=1540040155, splitter='best')]

通过对各种超参数进行网格搜索来微调模型¶

以下是需要调整以最适合我们的数据的常见超参数列表。我们将对训练/测试数据的各种拆分尝试各种超参数设置，以找出最佳拟合，这对于训练和测试数据集具有几乎相同的准确度，或者准确度之间的差异非常小。

• base_estimator - 它让我们指定将创建集成的基本估计器。它可以是任何其他机器学习估计器，例如 KNearestNeighbors、DecisionTree 等。默认是最大深度为 3 的决策树。
• learning_rate - 它缩小了每棵树的贡献。learning_rate 和 n_estimatros 之间存在权衡。
• n_estimators - 其结果将被组合以产生最终预测的基本估计器的数量。 ​​default=100​​

我们将在下面尝试上述超参数的各种值，通过对数据进行 3 折交叉验证来找到我们数据集的最佳估计器。

%%time

from sklearn.tree import DecisionTreeRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.linear_model import LinearRegression

n_samples = X_boston.shape[0]
n_features = X_boston.shape[1]

params = {
            'base_estimator':[None, DecisionTreeRegressor(), KNeighborsRegressor(), LinearRegression()],
            'n_estimators': np.arange(100, 350, 50),
            'learning_rate': [0.5, 0.8, 1.0, 2.0, ]
         }

ada_boost_regressor_grid = GridSearchCV(AdaBoostRegressor(random_state=1), param_grid=params, n_jobs=-1, cv=3, verbose=5)
ada_boost_regressor_grid.fit(X_train,Y_train)

print('Train R^2 Score : %.3f'%ada_boost_regressor_grid.best_estimator_.score(X_train, Y_train))
print('Test R^2 Score : %.3f'%ada_boost_regressor_grid.best_estimator_.score(X_test, Y_test))
print('Best R^2 Score Through Grid Search : %.3f'%ada_boost_regressor_grid.best_score_)
print('Best Parameters : ',ada_boost_regressor_grid.best_params_)

Fitting 3 folds for each of 80 candidates, totalling 240 fits

[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.

[Parallel(n_jobs=-1)]: Done  12 tasks      | elapsed:    0.9s

[Parallel(n_jobs=-1)]: Done 120 tasks      | elapsed:    8.2s

[Parallel(n_jobs=-1)]: Done 240 out of 240 | elapsed:   15.0s finished

Train R^2 Score : 1.000

Test R^2 Score : 0.848

Best R^2 Score Through Grid Search : 0.878

Best Parameters :  {'base_estimator': DecisionTreeRegressor(criterion='mse', max_depth=None, max_features=None,

max_leaf_nodes=None, min_impurity_decrease=0.0,

min_impurity_split=None, min_samples_leaf=1,

min_samples_split=2, min_weight_fraction_leaf=0.0,

presort=False, random_state=None, splitter='best'), 'learning_rate': 1.0, 'n_estimators': 150}

CPU times: user 512 ms, sys: 11.9 ms, total: 524 ms

Wall time: 15.3 s

输出前几个交叉验证结果¶

cross_val_results = pd.DataFrame(ada_boost_regressor_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))

cross_val_results.head() ## Printing first few results.

Number of Various Combinations of Parameters Tried : 80

比较 Ada Boost 与 Gradient Boosting、Bagging、随机森林、额外树、决策树和额外树的性能¶

from sklearn import ensemble
## Ada Boosting Regressor with Default Params
ada_regressor = ensemble.AdaBoostRegressor(random_state=1)
ada_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(ada_regressor.__class__.__name__,
                                                     ada_regressor.score(X_train, Y_train),ada_regressor.score(X_test, Y_test)))

## Above Hyper-perameter tuned Ada Boosting Regressor
ada_regressor = ensemble.AdaBoostRegressor(random_state=1, **ada_boost_regressor_grid.best_params_)
ada_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(ada_regressor.__class__.__name__,
                                                     ada_regressor.score(X_train, Y_train),ada_regressor.score(X_test, Y_test)))

## Gradient Boosting Regressor with Default Params
gb_regressor = ensemble.GradientBoostingRegressor(random_state=1)
gb_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_regressor.__class__.__name__,
                                                     gb_regressor.score(X_train, Y_train),gb_regressor.score(X_test, Y_test)))

## Random Forest Regressor with Default Params
rforest_regressor = ensemble.RandomForestRegressor(random_state=1)
rforest_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(rforest_regressor.__class__.__name__,
                                                     rforest_regressor.score(X_train, Y_train),rforest_regressor.score(X_test, Y_test)))


## Extra Trees Regressor with Default Params
extra_forest_regressor = ensemble.ExtraTreesRegressor(random_state=1)
extra_forest_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_forest_regressor.__class__.__name__,
                                                     extra_forest_regressor.score(X_train, Y_train),extra_forest_regressor.score(X_test, Y_test)))

## Bagging Regressor with Default Params
bag_regressor = ensemble.BaggingRegressor(random_state=1)
bag_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(bag_regressor.__class__.__name__,
                                                     bag_regressor.score(X_train, Y_train),bag_regressor.score(X_test, Y_test)))


## Decision Tree with Default Parameters
dtree_regressor = tree.DecisionTreeRegressor(random_state=1)
dtree_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(dtree_regressor.__class__.__name__,
                                                     dtree_regressor.score(X_train, Y_train),dtree_regressor.score(X_test, Y_test)))

## Decision Tree with Default Parameters
extra_tree_regressor = tree.ExtraTreeRegressor(random_state=1)
extra_tree_regressor.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_tree_regressor.__class__.__name__,
                                                     extra_tree_regressor.score(X_train, Y_train),extra_tree_regressor.score(X_test, Y_test)))

AdaBoostRegressor : Train Accuracy : 0.92, Test Accuracy : 0.79

AdaBoostRegressor : Train Accuracy : 1.00, Test Accuracy : 0.85

GradientBoostingRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

RandomForestRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

ExtraTreesRegressor : Train Accuracy : 1.00, Test Accuracy : 0.83

BaggingRegressor : Train Accuracy : 0.98, Test Accuracy : 0.81

DecisionTreeRegressor : Train Accuracy : 1.00, Test Accuracy : 0.44

ExtraTreeRegressor : Train Accuracy : 1.00, Test Accuracy : 0.51

AdaBoost分类器¶

​​AdaBoostClassifier​​​ 可作为​​ensemble​​sklearn 模块的一部分使用。我们将使用数字数据训练默认模型，然后通过尝试各种超参数设置来调整模型以提高其性能。我们还将它与其他分类估计器进行比较，以检查其相对于其他机器学习模型的性能。

X_train, X_test, Y_train, Y_test = train_test_split(X_digits, Y_digits, train_size=0.80, test_size=0.20, random_state=123)
print('Train/Test Sizes : ', X_train.shape, X_test.shape, Y_train.shape, Y_test.shape)

Train/Test Sizes :  (1437, 64) (360, 64) (1437,) (360,)

from sklearn.ensemble import AdaBoostClassifier

ada_boosting_classif = AdaBoostClassifier()
ada_boosting_classif.fit(X_train, Y_train)

AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,

n_estimators=50, random_state=None)

Y_preds = ada_boosting_classif.predict(X_test)

print(Y_preds[:15])
print(Y_test[:15])

print('Test Accuracy : %.3f'%(Y_preds == Y_test).mean())
print('Test Accuracy : %.3f'%ada_boosting_classif.score(X_test, Y_test)) ## Score method also evaluates accuracy for classification models.
print('Training Accuracy : %.3f'%ada_boosting_classif.score(X_train, Y_train))

[8 8 8 8 8 8 8 9 8 8 0 9 6 6 6]

[3 3 4 4 1 3 1 0 7 4 0 0 5 1 6]

Test Accuracy : 0.369

Test Accuracy : 0.369

Training Accuracy : 0.337

​​AdaBoostClassifier​​​的重要属性​

​​AdaBoostClassifier​​​所有属性都与 ​​AdaBoostRegressor相同​​。

print("Base Estimator : ", ada_boosting_classif.base_estimator_)

Base Estimator :  DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=1,

max_features=None, max_leaf_nodes=None,

min_impurity_decrease=0.0, min_impurity_split=None,

min_samples_leaf=1, min_samples_split=2,

min_weight_fraction_leaf=0.0, presort=False,

random_state=None, splitter='best')

print("Feature Importances : ", ada_boosting_classif.feature_importances_)

Feature Importances :  [0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.

0.   0.   0.   0.   0.   0.   0.   0.02 0.   0.   0.   0.   0.   0.

0.48 0.   0.   0.   0.   0.   0.   0.   0.5  0.   0.   0.   0.   0.

0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.   0.

0.   0.   0.   0.   0.   0.   0.   0.  ]

print("Estimators Shape : ", len(ada_boosting_classif.estimators_))

ada_boosting_classif.estimators_[:2]

Estimators Shape :  50

[DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=1,

max_features=None, max_leaf_nodes=None,

min_impurity_decrease=0.0, min_impurity_split=None,

min_samples_leaf=1, min_samples_split=2,

min_weight_fraction_leaf=0.0, presort=False,

random_state=777627829, splitter='best'),

DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=1,

max_features=None, max_leaf_nodes=None,

min_impurity_decrease=0.0, min_impurity_split=None,

min_samples_leaf=1, min_samples_split=2,

min_weight_fraction_leaf=0.0, presort=False,

random_state=429440265, splitter='best')]

通过对各种超参数进行网格搜索来微调模型¶

​​AdaBoostClassifier​​​几乎所有参数都与​​AdaBoostRegressor​​

%%time

from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression

n_samples = X_digits.shape[0]
n_features = X_digits.shape[1]

params = {
            'base_estimator':[None, DecisionTreeClassifier(), SVC(), LogisticRegression()],
            'n_estimators': np.arange(100, 350, 100),
            'learning_rate': [0.5, 1.0, 2.0, ]
         }


ada_boost_classif_grid = GridSearchCV(AdaBoostClassifier(random_state=1, algorithm='SAMME'), param_grid=params, n_jobs=-1, cv=3, verbose=5)
ada_boost_classif_grid.fit(X_train,Y_train)

print('Train Accuracy : %.3f'%ada_boost_classif_grid.best_estimator_.score(X_train, Y_train))
print('Test Accuracy : %.3f'%ada_boost_classif_grid.best_estimator_.score(X_test, Y_test))
print('Best Accuracy Through Grid Search : %.3f'%ada_boost_classif_grid.best_score_)
print('Best Parameters : ',ada_boost_classif_grid.best_params_)

Fitting 3 folds for each of 36 candidates, totalling 108 fits

[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.

[Parallel(n_jobs=-1)]: Done  10 tasks      | elapsed:    1.3s

[Parallel(n_jobs=-1)]: Done 108 out of 108 | elapsed: 11.2min finished

Train Accuracy : 0.985

Test Accuracy : 0.956

Best Accuracy Through Grid Search : 0.953

Best Parameters :  {'base_estimator': LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,

intercept_scaling=1, l1_ratio=None, max_iter=100,

multi_class='warn', n_jobs=None, penalty='l2',

random_state=None, solver='warn', tol=0.0001, verbose=0,

warm_start=False), 'learning_rate': 2.0, 'n_estimators': 300}

CPU times: user 32.7 s, sys: 17.6 s, total: 50.3 s

Wall time: 11min 33s

输出前几个交叉验证结果¶

cross_val_results = pd.DataFrame(ada_boost_classif_grid.cv_results_)
print('Number of Various Combinations of Parameters Tried : %d'%len(cross_val_results))

cross_val_results.head() ## Printing first few results.

Number of Various Combinations of Parameters Tried : 36

比较 Ada Boost 与 Gradient Boosting、Bagging、随机森林、额外树、决策树和额外树的性能¶

from sklearn import ensemble

## Gradient Boosting Regressor with Default Params
ada_classifier = ensemble.AdaBoostClassifier(random_state=1)
ada_classifier.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(ada_classifier.__class__.__name__,
                                                     ada_classifier.score(X_train, Y_train),ada_classifier.score(X_test, Y_test)))

## Above Hyper-perameter tuned Gradient Boosting Regressor
ada_classifier = ensemble.AdaBoostClassifier(random_state=1, **ada_boost_classif_grid.best_params_)
ada_classifier.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(ada_classifier.__class__.__name__,
                                                     ada_classifier.score(X_train, Y_train),ada_classifier.score(X_test, Y_test)))

## Gradient Boosting Regressor with Default Params
gb_classifier = ensemble.GradientBoostingClassifier(random_state=1)
gb_classifier.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(gb_classifier.__class__.__name__,
                                                     gb_classifier.score(X_train, Y_train),gb_classifier.score(X_test, Y_test)))


## Random Forest Regressor with Default Params
rforest_classif = ensemble.RandomForestClassifier(random_state=1)
rforest_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(rforest_classif.__class__.__name__,
                                                     rforest_classif.score(X_train, Y_train),rforest_classif.score(X_test, Y_test)))


## Extra Trees Regressor with Default Params
extra_forest_classif = ensemble.ExtraTreesClassifier(random_state=1)
extra_forest_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_forest_classif.__class__.__name__,
                                                     extra_forest_classif.score(X_train, Y_train),extra_forest_classif.score(X_test, Y_test)))

## Bagging Regressor with Default Params
bag_classif = ensemble.BaggingClassifier(random_state=1)
bag_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(bag_classif.__class__.__name__,
                                                     bag_classif.score(X_train, Y_train),bag_classif.score(X_test, Y_test)))


## Decision Tree with Default Parameters
dtree_classif = tree.DecisionTreeClassifier(random_state=1)
dtree_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(dtree_classif.__class__.__name__,
                                                     dtree_classif.score(X_train, Y_train),dtree_classif.score(X_test, Y_test)))

## Decision Tree with Default Parameters
extra_tree_classif = tree.ExtraTreeClassifier(random_state=1)
extra_tree_classif.fit(X_train, Y_train)
print("%s : Train Accuracy : %.2f, Test Accuracy : %.2f"%(extra_tree_classif.__class__.__name__,
                                                     extra_tree_classif.score(X_train, Y_train),extra_tree_classif.score(X_test, Y_test)))

AdaBoostClassifier : Train Accuracy : 0.34, Test Accuracy : 0.37

AdaBoostClassifier : Train Accuracy : 0.75, Test Accuracy : 0.74

GradientBoostingClassifier : Train Accuracy : 1.00, Test Accuracy : 0.96

RandomForestClassifier : Train Accuracy : 1.00, Test Accuracy : 0.94

ExtraTreesClassifier : Train Accuracy : 1.00, Test Accuracy : 0.95

BaggingClassifier : Train Accuracy : 1.00, Test Accuracy : 0.94

DecisionTreeClassifier : Train Accuracy : 1.00, Test Accuracy : 0.83

ExtraTreeClassifier : Train Accuracy : 1.00, Test Accuracy : 0.83

欢迎关注​​《python金融风控评分卡模型和数据分析(加强版)》​​，学习评分卡,集成树，神经网络算法等系统化风控建模内容。

​版权声明：文章来自公众号(python风控模型),未经许可，不得抄袭。遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。​