Predicting voters turnout for US election in 2016 using AutoML and spatial feature engineering - Part II¶
Introduction ¶
The objective of this notebook is to demonstrate the application of AutoML on tabular data and show the improvements that can be achieved using this method, rather than conventional workflows. In part 1 of this notebook series, a considerable increase was obtained when implementing AutoML, and in this notebook, the result will be further enhanced using spatial feature engineering. These new features will be estimated by considering, and subsequently extracting, the inherent spatial patterns present in the data.
The percentage of voter turnout by county for the general election for US in 2016 will be predicted using the demographic characteristics of US counties and their socioeconomic parameters.
Imports ¶
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
from IPython.display import Image, HTML
from sklearn.preprocessing import MinMaxScaler,RobustScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
import sklearn.metrics as metrics
from fastai.imports import *
from datetime import datetime as dt
import arcgis
from arcgis.gis import GIS
from arcgis.learn import prepare_tabulardata, AutoML, MLModel
import arcpy
Connecting to ArcGIS ¶
gis = GIS("home")
Accessing & Visualizing datasets ¶
The 2016 election data is downloaded from the portal as a zipped shapefile, which is then unzipped and processed in the following.
voter_zip = gis.content.get('650e7d6aa8fb4601a75d632a2c114425')
voter_zip
Shapefile by api_data_owner
import os, zipfile
filepath_new = voter_zip.download(file_name=voter_zip.name)
with zipfile.ZipFile(filepath_new, 'r') as zip_ref:
zip_ref.extractall(Path(filepath_new).parent)
output_path = Path(os.path.join(os.path.splitext(filepath_new)[0]))
output_path = os.path.join(output_path,"VotersTurnoutCountyEelction2016.shp")
The attribute table contains voter turnout data per county for the entire US, which is extracted here as a pandas dataframe. The voter_turn field in the dataframe contains voter turnout percentages for each county for the 2016 election. This will be used as the dependent variable and will be predicted using the various demographic and socioeconomic variables of each county.
# getting the attribute table from the shapefile which will be used for building the model
sdf_main = pd.DataFrame.spatial.from_featureclass(output_path)
sdf_main.head()
sdf_main.shape
Here, the data is visualized by mapping the voter turnout field into five classes. It can be observed that there are belts running along the eastern and southern parts of the country that represent comparatively lower voter turnout of less than 55%.
The AutoMl process significantly improves the fit, compared to the standalone random forest model, and the validation R-squared jumps to a new high. Now, the previous visualization of the data reveals the presence of a spatial pattern in the data. Next, this spatial pattern will be estimated and included as spatial features to further improve the model.
Estimating Spatial Autocorrelation ¶
This characteristic is also known as spatial autocorrelation and is measured by the index known as Moran's I, which is estimated using the ClustersOutliers tool available in Arcpy.
# First the Arcpy env is specified which will be used saving the result of the Arcpy tool
arcpy.env.workspace = output_path.replace(output_path.split('\\')[-1], "arcpy_test_env")
if os.path.exists(arcpy.env.workspace):
shutil.rmtree(arcpy.env.workspace)
os.makedirs(arcpy.env.workspace)
Calculating Local Moran's I¶
The ClustersOutliers tool will calculate the local Moran's I index for each county and identify statistically significant hot spots, cold spots, and spatial outliers. As input, the tool takes the shapefile containing the data, the field name for which the clustering is to be estimated, and the output name of the shapefile, and outputs a Moran's I value, a z-score, a pseudo p-value, and a code representing the cluster type for each statistically significant feature. The z-scores and pseudo p-values represent the statistical significance of the computed index values.
arcpy.env.workspace = arcpy.env.workspace
output_path = output_path
result = arcpy.stats.ClustersOutliers(output_path,
"voter_turn", "voters_turnout_ClusterOutlier.shp",
'INVERSE_DISTANCE',
'EUCLIDEAN_DISTANCE','ROW', "#", "#","NO_FDR", 499)
# accessing the attribute table from the output shapefile
sdf_main_LMi = pd.DataFrame.spatial.from_featureclass(result[0])
sdf_main_LMi.head()
Here, the Moran's I value is stored in the LMiIndex, field, with its z-score and pseudo p-value in the fields LMiZScore and LMiPValuerespectively, and the code in COType.
Visualizing the spatial autocorrelation¶
The COType field in the Output Feature Class will be HH for a statistically significant cluster of high values and LL for a statistically significant cluster of low values. The COType field in the Output Feature Class will also indicate if the feature has a high value and is surrounded by features with low values (HL) or if the feature has a low value and is surrounded by features with high values (LH). This is visualized in the map below:
# visualizing spatial autocorrelation in voters turnout
m2= GIS().map('United States', zoomlevel=4)
sdf_main_LMi.spatial.plot(map_widget = m2, renderer_type='u', col='COType', line_width=0.5)
m2.legend=True
m2
The black pixels in the map above show that there is spatial clustering of low voter turnout along the eastern coast, while the white pixels in the northeastern, central, and northwestern portions of the country indicate areas of spatial clustering of high voter turnout.
To include this data as spatial features, the counties with the most significant (lowest) p-values will be identified, and the distance and the angle or direction of each county will be measured from those lowest-p value counties. These two variables, the distance and the angle, are included as the new spatial features in the model.
# checking the field names having the p values
sdf_main_LMi.columns
Selecting highly significant spatial clustering county ¶
The most significant (lowest) p-value here is 0.002. All counties with this p value will be selected, and a field will be created that will be used to generate a shapefile containing these highly significant counties. Another field will also be created for counties with p-values less than or equal to 0.05, representing the remaining significantly clustering counties, that will be used as the third spatial feature in the final model.
# creating new fields with highly significant clustering counties
sdf_main_LMi['LMi_hi_sig<.002'] = np.where(sdf_main_LMi['LMiPValue']<=.002, 1,0)
sdf_main_LMi['LMi_sig_<0.05'] = np.where(sdf_main_LMi['LMiPValue']<=.05, 1,0)
sdf_main_LMi.head()
# create new dataframe for LMi_hi_sig<.002
LMi_hi_sig_county_main = sdf_main_LMi[sdf_main_LMi['LMi_hi_sig<.002']==1].copy()
LMi_hi_sig_county_main.columns
# creating a new shapefile for the most significant clustering counties from spatial dataframe
near_dist_from_main_county = sdf_main_LMi.spatial.to_featureclass('voters_turnout_train_LMi'+str(dt.now().microsecond))
near_dist_to_hi_sig_county = LMi_hi_sig_county_main.spatial.to_featureclass('LMi_hi_sig_county_train'+str(dt.now().microsecond))
Estimating distances and angle of counties from highly clustered counties¶
The Near(Analysis) tool from Arcpy is used to calculate the distance and the angle of all the counties from the highly significant clustering counties. As input, it takes the counties of high significance from which the distance is to be estimated, followed by the shapefile containing all of the counties to which the distance and the angle is to be calculated.
# Using the Near tool to calculate distance and angle
dist_to_nearest_hi_sig = arcpy.analysis.Near(near_dist_from_main_county,near_dist_to_hi_sig_county,'#','#','ANGLE','GEODESIC')
# Accessing the attribute table from the resulting shapefile
sdf_nearest_hi_sig = pd.DataFrame.spatial.from_featureclass(dist_to_nearest_hi_sig[0])
sdf_nearest_hi_sig.head()
sdf_nearest_hi_sig.columns
#LMi_hi_sig_county_main.columns
In the resulting dataframe above, the fields NEAR_DIST and NEAR_ANGLE ( third and the second field from the last) represent the distance and angle of the counties from the highly significant clustering counties, while the field named LMi_sig_<0, represents all of the significant counties. All three will be used as the spatial predictors in the final model.
sdf_main.head(2)
# dropping the existing p-values estimated columns from the main table to be replaced by the newly calculated values
sdf_main_final = sdf_main.drop(['SOURCE_ID', 'voter_tu_1',
'Shape_Leng', 'Shape_Area', 'LMiIndex', 'LMiZScore', 'LMiPValue',
'COType', 'NNeighbors', 'ZTransform', 'SpatialLag', 'LMi_hi_sig',
'LMi_normal', 'NEAR_FID', 'Shape_Le_1', 'Shape_Ar_1', 'LMiHiDist', 'SHAPE'], axis=1)
sdf_main_final.head(2)
Final dataset with spatial cluster variables ¶
# joining the newly calculated spatial features with the main dataset
sdf_main_final_merged = sdf_main_final.merge(sdf_nearest_hi_sig, on='FID', how='inner')
# checking the final merged data
sdf_main_final_merged.head(2)
Model Building ¶
Next, the dataset containing the new spatial variables will be used to fit the AutoML model for further model improvements.
Train-Test split¶
Here, the dataset with 3112 samples is split into training and test datasets with a 90 to 10 ratio.
# Splitting data with test size of 10% data for validation
test_size = 0.10
sdf_train, sdf_test = train_test_split(sdf_main_final_merged, test_size = test_size, random_state=32)
# checking train-test split
print(sdf_train.shape)
print(sdf_test.shape)
sdf_train.head(2)
sdf_train.columns
Data Preprocessing ¶
Here, X is the list of explanatory variables chosen from the new feature data that will be used for predicting voter turnout. The new spatial cluster features used here are NEAR_DIST, NEAR_ANGLE,LMi_sig_<0 as explained in the previous section. Some additional spatial features (City10Ang, City9Ang,City8Ang etc.) were also included to account for the direction of the counties in terms of the angle of the counties from various grades of cities that were pre-calculated.
Also, the categorical variables are marked with a True value inside of a tuple. The scaler is defined in the preprocessors.
#listing explanatory variables
X =[('county',True), ('state',True),'gender_med', 'householdi', 'electronic', 'raceandhis',
('voter_laws',True), 'educationa', 'educatio_1', 'educatio_2', 'educatio_3',
'maritalsta', 'F5yearincr', 'F5yearin_1', 'F5yearin_2', 'F5yearin_3',
'F5yearin_4', 'F5yearin_5', 'F5yearin_6', 'language_a', 'hispanicor',
'hispanic_1', 'raceandh_1', 'atrisk_avg', 'disposable', 'disposab_1',
'disposab_2', 'disposab_3', 'disposab_4', 'disposab_5', 'disposab_6',
'disposab_7', 'disposab_8', 'disposab_9', 'disposa_10', 'househol_1',
'househol_2', 'househol_3', 'househol_4', 'househol_5', 'househol_6',
'househol_7', 'househol_8', 'househol_9', 'language_1', 'language_2',
'households', 'househo_10', 'educatio_4', 'educatio_5', 'educatio_6',
'educatio_7', 'psychograp', 'psychogr_1', 'financial_', 'financial1',
'financia_1', 'miscellane', 'state_vote', 'state_vo_1',
'City10Ang', 'City9Dist', 'City9Ang',
'City8Dist', 'City8Ang', 'City7Dist', 'City7Ang', 'City6Dist',
'City6Ang', 'City5Dist', 'City5Ang', 'LMi_sig_<0', 'NEAR_DIST', 'NEAR_ANGLE']
# defining the preprocessors for scaling data
preprocessors = [('county', 'state','gender_med', 'householdi', 'electronic', 'raceandhis',
'voter_laws', 'educationa', 'educatio_1', 'educatio_2', 'educatio_3',
'maritalsta', 'F5yearincr', 'F5yearin_1', 'F5yearin_2', 'F5yearin_3',
'F5yearin_4', 'F5yearin_5', 'F5yearin_6', 'language_a', 'hispanicor',
'hispanic_1', 'raceandh_1', 'atrisk_avg', 'disposable', 'disposab_1',
'disposab_2', 'disposab_3', 'disposab_4', 'disposab_5', 'disposab_6',
'disposab_7', 'disposab_8', 'disposab_9', 'disposa_10', 'househol_1',
'househol_2', 'househol_3', 'househol_4', 'househol_5', 'househol_6',
'househol_7', 'househol_8', 'househol_9', 'language_1', 'language_2',
'households', 'househo_10', 'educatio_4', 'educatio_5', 'educatio_6',
'educatio_7', 'psychograp', 'psychogr_1', 'financial_', 'financial1',
'financia_1', 'miscellane', 'state_vote', 'state_vo_1',
'City10Ang', 'City9Dist', 'City9Ang',
'City8Dist', 'City8Ang', 'City7Dist', 'City7Ang', 'City6Dist',
'City6Ang', 'City5Dist', 'City5Ang', 'LMi_sig_<0', 'NEAR_DIST', 'NEAR_ANGLE', MinMaxScaler())]
# preparing data for the model
data = prepare_tabulardata(sdf_train,
variable_predict='voter_turn_x',
explanatory_variables=X,
preprocessors=preprocessors)
data.show_batch()
Fitting a random forest model ¶
First, a random forest model is fitted to the new spatial data.
Model Initialization ¶
The MLModel is initialized with the Random Forest model from Scikit-learn (Sklearn), along with its model parameters
# defining the model along with the parameters
model = MLModel(data, 'sklearn.ensemble.RandomForestRegressor', n_estimators=500, random_state=43)
model.fit()
model.score()
# validating trained model on test dataset
voter_county_mlmodel_predicted = model.predict(sdf_test, prediction_type='dataframe')
voter_county_mlmodel_predicted.head(2)
# calculating validation model score
r_square_voter_county_mlmodel_Test = metrics.r2_score(voter_county_mlmodel_predicted['voter_turn_x'], voter_county_mlmodel_predicted['prediction_results'])
print('r_square_voter_county_mlmodel_Test: ', round(r_square_voter_county_mlmodel_Test,2))
The validation r square for the random forest model is satisfactory, and now AutoML will be used to improve it.
Fitting Using AutoML ¶
The same data obtained using the prepare_taulardata function is used as input for the AutoML model. Here, the model is initialized using the Compete mode, which is the best performing option of the available modes.
# initializing AutoML model with the Compete mode
AutoML_voters_county_obj_compete = AutoML(data, eval_metric='r2', mode='Compete', n_jobs=1)
# training the AutoML model
AutoML_voters_county_obj_compete.fit()
Here, the ensemble model is the best model, and its R-squared validation score shows the final improvements achieved after including the new, spatially engineered variables. The best model diagnostics and related reports, like feature importance, model performance, etc., are saved in the folder mentioned in the output message for further reference.
# train score of the model
AutoML_voters_county_obj_compete.score()
Model output ¶
# The output diagnostics can also be printed in a report form
AutoML_voters_county_obj_compete.report()
AutoML Leaderboard
| Best model | name | model_type | metric_type | metric_value | train_time |
|---|---|---|---|---|---|
| 1_DecisionTree | Decision Tree | r2 | 0.399429 | 14.33 | |
| 2_DecisionTree | Decision Tree | r2 | 0.448983 | 13.5 | |
| 3_DecisionTree | Decision Tree | r2 | 0.445253 | 13.6 | |
| 4_Linear | Linear | r2 | 0.66238 | 13.57 | |
| 5_Default_LightGBM | LightGBM | r2 | 0.770059 | 59.24 | |
| 6_Default_Xgboost | Xgboost | r2 | 0.766568 | 168.39 | |
| 7_Default_RandomForest | Random Forest | r2 | 0.587453 | 65.2 | |
| 8_Default_ExtraTrees | Extra Trees | r2 | 0.526491 | 24.35 | |
| 18_LightGBM | LightGBM | r2 | 0.777045 | 28.97 | |
| 9_Xgboost | Xgboost | r2 | 0.751944 | 160.86 | |
| 27_RandomForest | Random Forest | r2 | 0.585405 | 48.18 | |
| 36_ExtraTrees | Extra Trees | r2 | 0.505466 | 23.13 | |
| 19_LightGBM | LightGBM | r2 | 0.754122 | 22.17 | |
| 10_Xgboost | Xgboost | r2 | 0.744709 | 244.06 | |
| 28_RandomForest | Random Forest | r2 | 0.522452 | 41.72 | |
| 37_ExtraTrees | Extra Trees | r2 | 0.46222 | 22.43 | |
| 20_LightGBM | LightGBM | r2 | 0.773785 | 45.16 | |
| 11_Xgboost | Xgboost | r2 | 0.774746 | 43.31 | |
| 29_RandomForest | Random Forest | r2 | 0.654542 | 87.83 | |
| 38_ExtraTrees | Extra Trees | r2 | 0.611703 | 30.18 | |
| 21_LightGBM | LightGBM | r2 | 0.760481 | 79.71 | |
| 12_Xgboost | Xgboost | r2 | 0.760265 | 26.88 | |
| 30_RandomForest | Random Forest | r2 | 0.653129 | 64.25 | |
| 39_ExtraTrees | Extra Trees | r2 | 0.591046 | 25.79 | |
| 22_LightGBM | LightGBM | r2 | 0.748574 | 22.81 | |
| 13_Xgboost | Xgboost | r2 | 0.769551 | 35.16 | |
| 31_RandomForest | Random Forest | r2 | 0.682243 | 74.91 | |
| 40_ExtraTrees | Extra Trees | r2 | 0.629549 | 28.58 | |
| 23_LightGBM | LightGBM | r2 | 0.771215 | 40.85 | |
| 14_Xgboost | Xgboost | r2 | 0.780234 | 40.45 | |
| 32_RandomForest | Random Forest | r2 | 0.676783 | 86.82 | |
| 41_ExtraTrees | Extra Trees | r2 | 0.638892 | 33.17 | |
| 24_LightGBM | LightGBM | r2 | 0.774532 | 46.36 | |
| 15_Xgboost | Xgboost | r2 | 0.770555 | 117 | |
| 14_Xgboost_KMeansFeatures | Xgboost | r2 | 0.767433 | 44.16 | |
| 18_LightGBM_KMeansFeatures | LightGBM | r2 | 0.772894 | 38.96 | |
| 11_Xgboost_KMeansFeatures | Xgboost | r2 | 0.767903 | 59.52 | |
| 14_Xgboost_RandomFeature | Xgboost | r2 | 0.773451 | 45.62 | |
| 14_Xgboost_SelectedFeatures | Xgboost | r2 | 0.774603 | 32.83 | |
| 18_LightGBM_SelectedFeatures | LightGBM | r2 | 0.781371 | 30.84 | |
| 31_RandomForest_SelectedFeatures | Random Forest | r2 | 0.683045 | 64.85 | |
| 41_ExtraTrees_SelectedFeatures | Extra Trees | r2 | 0.640023 | 33.41 | |
| 42_LightGBM_SelectedFeatures | LightGBM | r2 | 0.78418 | 34.36 | |
| 43_Xgboost | Xgboost | r2 | 0.775862 | 41.33 | |
| 44_Xgboost | Xgboost | r2 | 0.766297 | 31.5 | |
| 45_LightGBM | LightGBM | r2 | 0.777641 | 36.61 | |
| 46_Xgboost | Xgboost | r2 | 0.776048 | 55.15 | |
| 47_Xgboost | Xgboost | r2 | 0.770999 | 41.84 | |
| 48_Xgboost_SelectedFeatures | Xgboost | r2 | 0.77757 | 39.02 | |
| 49_Xgboost_SelectedFeatures | Xgboost | r2 | 0.765088 | 30.36 | |
| 50_LightGBM | LightGBM | r2 | 0.779526 | 39.18 | |
| 51_LightGBM_SelectedFeatures | LightGBM | r2 | 0.782666 | 32.57 | |
| 52_LightGBM_SelectedFeatures | LightGBM | r2 | 0.783625 | 39.45 | |
| 53_LightGBM_SelectedFeatures | LightGBM | r2 | 0.784112 | 31.89 | |
| 54_LightGBM_SelectedFeatures | LightGBM | r2 | 0.783465 | 33.66 | |
| 55_Xgboost | Xgboost | r2 | 0.773724 | 45.43 | |
| 56_Xgboost | Xgboost | r2 | 0.768679 | 38.96 | |
| 57_LightGBM | LightGBM | r2 | 0.780059 | 44.44 | |
| 58_LightGBM | LightGBM | r2 | 0.780877 | 43.93 | |
| 42_LightGBM_SelectedFeatures_BoostOnErrors | LightGBM | r2 | 0.783428 | 35.28 | |
| Ensemble | Ensemble | r2 | 0.792354 | 8.42 | |
| 42_LightGBM_SelectedFeatures_Stacked | LightGBM | r2 | 0.780498 | 27.23 | |
| 14_Xgboost_Stacked | Xgboost | r2 | 0.783106 | 28.67 | |
| 31_RandomForest_SelectedFeatures_Stacked | Random Forest | r2 | 0.786935 | 115.47 | |
| 41_ExtraTrees_SelectedFeatures_Stacked | Extra Trees | r2 | 0.791264 | 39.52 | |
| 53_LightGBM_SelectedFeatures_Stacked | LightGBM | r2 | 0.783177 | 25.04 | |
| 48_Xgboost_SelectedFeatures_Stacked | Xgboost | r2 | 0.783833 | 29.56 | |
| 31_RandomForest_Stacked | Random Forest | r2 | 0.790533 | 110.11 | |
| 41_ExtraTrees_Stacked | Extra Trees | r2 | 0.791692 | 41.59 | |
| 52_LightGBM_SelectedFeatures_Stacked | LightGBM | r2 | 0.779776 | 28.53 | |
| 46_Xgboost_Stacked | Xgboost | r2 | 0.776083 | 38.55 | |
| 32_RandomForest_Stacked | Random Forest | r2 | 0.786649 | 145.44 | |
| the best | Ensemble_Stacked | Ensemble | r2 | 0.795538 | 12.06 |
AutoML Performance
AutoML Performance Boxplot
