解决的问题是：每个足球运动员在转会市场都有各自的价码。本次数据练习的目的是根据球员的各项信息和能力（63项属性）值来预测该球员的市场价值。数据来源：FIFA2018。
http://sofasofa.io/competition.php?id=7#c4

借助一些优秀博主分享对单一xgboost的经验已经可以把预测结果提高到前30名，MSE约为18.XX左右。
https://www.jianshu.com/p/2f803f449567?utm_campaign=haruki&utm_content=note&utm_medium=reader_share&utm_source=weixin

今天与其他文章侧重点不太一样的地方关于 集成学习。其目的是在xgboost的基础上进一步提高预测准度，目前名次是20。

1.数据处理

训练集中共有10441条样本，预测集中有7000条样本。每条样本代表一位球员，数据中每个球员有63项属性。数据中含有缺失值。在数据集中增加一列是否为门将的特征，以这个特征可以将数据样本分类，然后预测结果。一是对该特征进行独热编码，产生是门将和非门将两个特征，不切分预测；二是根据该特征对数据样本进行分类，将是门将的样本归为一个数据集，非门将的样本归为另一个数据集，然后对这两个数据集进行预测。

# 获得球员年龄
today = pd.to_datetime(date(2018, 4, 15))

train['birth_date'] = pd.to_datetime(train['birth_date'])
train['age'] = (today - train['birth_date']).apply(lambda x: x.days) / 365.

test['birth_date'] = pd.to_datetime(test['birth_date'])
test['age'] = (today - test['birth_date']).apply(lambda x: x.days) / 365.


# 获得球员最擅长位置上的评分
positions = ['rw', 'rb', 'st', 'lw', 'cf', 'cam', 'cm', 'cdm', 'cb', 'lb', 'gk']

train['best_pos'] = train[positions].max(axis=1)
test['best_pos'] = test[positions].max(axis=1)


# 计算球员的身体质量指数(BMI)
train['BMI'] = 10000. * train['weight_kg'] / (train['height_cm'] ** 2)
test['BMI'] = 10000. * test['weight_kg'] / (test['height_cm'] ** 2)


# 判断一个球员是否是守门员
train['is_gk'] = train['gk'] > 0
test['is_gk'] = test['gk'] > 0
#独热编码
colunms_to_enconding=['work_rate_att','work_rate_def','preferred_foot','is_gk']
all_data=pd.get_dummies(all_data,columns=colunms_to_enconding)

#切分数据集
test['pred'] = 0
train=train.drop(train[positions],axis=1)
train=train.drop('id',axis=1)
train1=train.drop(['y','is_gk'],axis=1)
used_feat=train1.columns
train_no_gk=train[train['is_gk'] == False][used_feat].copy()
y_train_no_gk= train[train['is_gk'] == False]['y'].copy()
test_no_gk=test[test['is_gk'] == False][used_feat].copy()
train_is_gk=train[train['is_gk'] == True][used_feat].copy()
y_train_is_gk= train[train['is_gk'] == True]['y'].copy()
test_is_gk=test[test['is_gk'] == True][used_feat].copy()

2.特征选择

特征的选择有两种方法：
第一，人工选择。参考了一下网上的链接，由人工经验来取舍特征，似乎也得到不错的效果（MSE排名前20）。比如nationality这个特征，一个球员的价值跟他的国籍关系大不大，可以根据人工经验选择。
第二，使用遗传算法对特征进行筛选。用遗传算法筛选特征虽然在线下得到非常好的效果，但是在线上提交得出的结果并不理想。使用PCA降维，由于在特征处理中，将分类特征都进行的独热编码，造成了维度增大，所以考虑用PCA降维，效果也不是很好；最后用lasso去fit数据集，然后根据lasso的系数的绝对值大小作为特征重要性的判断，然后将系数接近0的特征进行剔除；使用手动降维，通过对数据集的判断，来手动删除特征，，在切分是否为门将的数据集中，在非门将的样本数据集中关于门将的特征可以进行剔除。

for col in ('id','lb', 'cb', 'cdm', 'cm','cam','cf','lw','st','rb','rw','birth_date','gk',
            'gk_diving','gk_handling','gk_kicking','gk_positioning','gk_reflexes'
            ,'nationality'):
   all_data=all_data.drop(col,axis=1)

3.模型集成

代码中选择了5个模型，最后互相组合后发现LGB、GBoost、XGboost这些在kaggle中大杀器模型进行集成效果最好。大概的思想是先把最强的所有模型预测结果求平均，然后最强平均的给一个适合的权重，在配上2个单个最强的模型。
如以高考为例，我把全国排名前5的考生平均答案作为70%的参考依据，然后加上15%第一名（XGboost）的结果，再加上15%第二名（LGB）的结果。
通过数据验证，组合出来的MSE结果，要比第一名（XGboost）效果要好。

def rmsle_cv(model):
    kf = KFold(n_folds, shuffle=True, random_state=42).get_n_splits(train.values)
    rmse= np.sqrt(-cross_val_score(model, train.values, y_train, scoring="neg_mean_squared_error", cv = kf))
    return(rmse)
def rmsle(y_train, y_pred):
    return np.sqrt(mean_squared_error(y_train, y_pred))
GBoost = GradientBoostingRegressor(max_depth=45,
                         learning_rate = 0.05,
                         alpha=0.05,                     
                         n_estimators=100,                             
                         verbose = 0)
model_xgb = xgb.XGBRegressor(max_depth=45,
                         learning_rate = 0.1,                         
                         n_estimators=650)
model_lgb = lgb.LGBMRegressor(objective='regression',
                         max_depth=25,
                         learning_rate = 0.1,
                         num_leaves = 25,
                         n_estimators=8000,
                         metric='rmse', 
                         verbose = 0,)
reg_ngk = RandomForestRegressor( max_depth=75,
                         min_samples_split = 2,                       
                         min_samples_leaf = 2,
                         n_estimators=450, 
                         verbose = 0)
lasso = make_pipeline(RobustScaler(), Lasso(alpha =0.0005, random_state=1))
class StackingAveragedModels(BaseEstimator, RegressorMixin, TransformerMixin):
    def __init__(self, base_models, meta_model, n_folds=5):
        self.base_models = base_models
        self.meta_model = meta_model
        self.n_folds = n_folds
   
    # We again fit the data on clones of the original models
    def fit(self, X, y):
        self.base_models_ = [list() for x in self.base_models]
        self.meta_model_ = clone(self.meta_model)
        kfold = KFold(n_splits=self.n_folds, shuffle=True, random_state=156)
        
        # Train cloned base models then create out-of-fold predictions
        # that are needed to train the cloned meta-model
        out_of_fold_predictions = np.zeros((X.shape[0], len(self.base_models)))
        for i, model in enumerate(self.base_models):
            for train_index, holdout_index in kfold.split(X, y):
                instance = clone(model)
                self.base_models_[i].append(instance)
                instance.fit(X[train_index], y[train_index])
                y_pred = instance.predict(X[holdout_index])
                out_of_fold_predictions[holdout_index, i] = y_pred
                
        # Now train the cloned  meta-model using the out-of-fold predictions as new feature
        self.meta_model_.fit(out_of_fold_predictions, y)
        return self
   
    #Do the predictions of all base models on the test data and use the averaged predictions as 
    #meta-features for the final prediction which is done by the meta-model
    def predict(self, X):
        meta_features = np.column_stack([
            np.column_stack([model.predict(X) for model in base_models]).mean(axis=1)
            for base_models in self.base_models_ ])
        return self.meta_model_.predict(meta_features)
stacked_averaged_models = StackingAveragedModels(base_models = (reg_ngk, GBoost, model_lgb, model_xgb),
                                                 meta_model = lasso)
score = rmsle_cv(stacked_averaged_models)
print("Stacking Averaged models score: {:.4f} ({:.4f})".format(score.mean(), score.std()))
stacked_averaged_models.fit(train.values, y_train)
stacked_train_pred = stacked_averaged_models.predict(train.values)
stacked_pred = stacked_averaged_models.predict(test.values)
print(rmsle(y_train, stacked_train_pred))


model_xgb.fit(train.values, y_train)
xgb_train_pred = model_xgb.predict(train.values)
xgb_pred = model_xgb.predict(test.values)
print(rmsle(y_train, xgb_train_pred))



model_lgb.fit(train.values, y_train)
lgb_train_pred = model_lgb.predict(train.values)
lgb_pred = model_lgb.predict(test.values)
print(rmsle(y_train, lgb_train_pred))



'''RMSE on the entire Train data when averaging'''

print('RMSLE score on train data:')
print(rmsle(y_train,stacked_train_pred*0.70 +xgb_train_pred*0.15 + lgb_train_pred*0.15 ))

train_new=pd.DataFrame(index=train.index)
for i in range(3):
    if i ==0:
        train_new.loc[:,i]=stacked_train_pred
    elif i==1:
        train_new.loc[:,i]=xgb_train_pred
    else:
        train_new.loc[:,i]=lgb_train_pred
test_new=pd.DataFrame(index=test.index)
for i in range(3):
    if i ==0:
        test_new.loc[:,i]=stacked_pred
    elif i==1:
        test_new.loc[:,i]=xgb_pred
    else:
        test_new.loc[:,i]=lgb_pred
from sklearn.linear_model import Lasso
Lasso = Lasso()
Lasso.fit(train_new, y_train.astype('int'))
y_pred=Lasso.predict(test_new)
rmsle(y_train,Lasso.predict(train_new))
ensemble = stacked_pred
submit.loc[:, 'y'] = ensemble
submit.to_csv('stacking.csv', index=False)

请留意7-25行，是罗列的各个模型，第89行是最后组合模型的损失函数。最后结果MSE能够到17.9x，排名进入前25，单用xgboost优化MSE能够在18.x，排名30左右。到了最后排名越靠前提升难度越大。