Predicting voters turnout for US election in 2016 using AutoML and spatial feature engineering - Part II


The objective of this notebook is to demonstrate the application of AutoML on tabular data and show the improvements that can be achieved using this method, rather than conventional workflows. In part 1 of this notebook series, a considerable increase was obtained when implementing AutoML, and in this notebook, the result will be further enhanced using spatial feature engineering. These new features will be estimated by considering, and subsequently extracting, the inherent spatial patterns present in the data.

The percentage of voter turnout by county for the general election for US in 2016 will be predicted using the demographic characteristics of US counties and their socioeconomic parameters.


%matplotlib inline

import matplotlib.pyplot as plt
import pandas as pd

from IPython.display import Image, HTML
from sklearn.preprocessing import MinMaxScaler,RobustScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
import sklearn.metrics as metrics
from fastai.imports import *
from datetime import datetime as dt

import arcgis
from arcgis.gis import GIS
from arcgis.learn import prepare_tabulardata, AutoML, MLModel
import arcpy

Connecting to ArcGIS

gis = GIS("home")

Accessing & Visualizing datasets

The 2016 election data is downloaded from the portal as a zipped shapefile, which is then unzipped and processed in the following.

voter_zip = gis.content.get('650e7d6aa8fb4601a75d632a2c114425') 
voters turnout 2016Shapefile by api_data_owner
Last Modified: August 23, 2021
0 comments, 78 views
import os, zipfile
filepath_new =
with zipfile.ZipFile(filepath_new, 'r') as zip_ref:
output_path = Path(os.path.join(os.path.splitext(filepath_new)[0]))
output_path = os.path.join(output_path,"VotersTurnoutCountyEelction2016.shp")  

The attribute table contains voter turnout data per county for the entire US, which is extracted here as a pandas dataframe. The voter_turn field in the dataframe contains voter turnout percentages for each county for the 2016 election. This will be used as the dependent variable and will be predicted using the various demographic and socioeconomic variables of each county.

# getting the attribute table from the shapefile which will be used for building the model
sdf_main = pd.DataFrame.spatial.from_featureclass(output_path)
001101001AutaugaAlabama0.61373838.625553.04.96...440.2115800.154568002.496745e+052.208598e+09133735.2925020{"rings": [[[-9619465, 3856529.0001000017], [-...
111201003BaldwinAlabama0.62736442.931429.04.64...220.3588940.057952001.642763e+065.671096e+09241925.1964263{"rings": [[[-9746859, 3539643.0001000017], [-...
221301005BarbourAlabama0.51381640.216876.03.49...62-0.868722-0.498354113.202971e+053.257816e+090.0000000{"rings": [[[-9468394, 3771591.0001000017], [-...
331401007BibbAlabama0.50136439.319360.03.64...43-1.0033410.286440002.279101e+052.311955e+09170214.4857597{"rings": [[[-9692114, 3928124.0001000017], [-...
441501009BlountAlabama0.60306440.921785.03.86...510.096177-0.336198012.918753e+052.456919e+0921128.5687847{"rings": [[[-9623907, 4063676.0001000017], [-...

5 rows × 97 columns

(3112, 97)

Here, the data is visualized by mapping the voter turnout field into five classes. It can be observed that there are belts running along the eastern and southern parts of the country that represent comparatively lower voter turnout of less than 55%.

The AutoMl process significantly improves the fit, compared to the standalone random forest model, and the validation R-squared jumps to a new high. Now, the previous visualization of the data reveals the presence of a spatial pattern in the data. Next, this spatial pattern will be estimated and included as spatial features to further improve the model.

Estimating Spatial Autocorrelation

This characteristic is also known as spatial autocorrelation and is measured by the index known as Moran's I, which is estimated using the ClustersOutliers tool available in Arcpy.

# First the Arcpy env is specified which will be used saving the result of the Arcpy tool
arcpy.env.workspace = output_path.replace(output_path.split('\\')[-1], "arcpy_test_env")

if os.path.exists(arcpy.env.workspace):


Calculating Local Moran's I

The ClustersOutliers tool will calculate the local Moran's I index for each county and identify statistically significant hot spots, cold spots, and spatial outliers. As input, the tool takes the shapefile containing the data, the field name for which the clustering is to be estimated, and the output name of the shapefile, and outputs a Moran's I value, a z-score, a pseudo p-value, and a code representing the cluster type for each statistically significant feature. The z-scores and pseudo p-values represent the statistical significance of the computed index values.

arcpy.env.workspace = arcpy.env.workspace 
output_path = output_path 
result = arcpy.stats.ClustersOutliers(output_path,
                                      "voter_turn", "voters_turnout_ClusterOutlier.shp",
                                     'EUCLIDEAN_DISTANCE','ROW', "#", "#","NO_FDR", 499)
# accessing the attribute table from the output shapefile
sdf_main_LMi = pd.DataFrame.spatial.from_featureclass(result[0])
0000.6137380.0326930.9475220.172440.2115800.154568{"rings": [[[-9619465, 3856529.0001000017], [-...
1110.6273640.0207920.2846780.376220.3588940.057952{"rings": [[[-9746859, 3539643.0001000017], [-...
2220.5138160.4327913.6584270.002LL62-0.868722-0.498354{"rings": [[[-9468394, 3771591.0001000017], [-...
3330.501364-0.287305-1.7233580.042LH43-1.0033410.286440{"rings": [[[-9692114, 3928124.0001000017], [-...
4440.603064-0.032324-2.0451580.024HL510.096177-0.336198{"rings": [[[-9623907, 4063676.0001000017], [-...

Here, the Moran's I value is stored in the LMiIndex, field, with its z-score and pseudo p-value in the fields LMiZScore and LMiPValuerespectively, and the code in COType.

Visualizing the spatial autocorrelation

The COType field in the Output Feature Class will be HH for a statistically significant cluster of high values and LL for a statistically significant cluster of low values. The COType field in the Output Feature Class will also indicate if the feature has a high value and is surrounded by features with low values (HL) or if the feature has a low value and is surrounded by features with high values (LH). This is visualized in the map below:

# visualizing spatial autocorrelation in voters turnout
m2= GIS().map('United States', zoomlevel=4)
sdf_main_LMi.spatial.plot(map_widget = m2, renderer_type='u', col='COType', line_width=0.5)

The black pixels in the map above show that there is spatial clustering of low voter turnout along the eastern coast, while the white pixels in the northeastern, central, and northwestern portions of the country indicate areas of spatial clustering of high voter turnout.

To include this data as spatial features, the counties with the most significant (lowest) p-values will be identified, and the distance and the angle or direction of each county will be measured from those lowest-p value counties. These two variables, the distance and the angle, are included as the new spatial features in the model.

# checking the field names having the p values
Index(['FID', 'SOURCE_ID', 'voter_turn', 'LMiIndex', 'LMiZScore', 'LMiPValue',
       'COType', 'NNeighbors', 'ZTransform', 'SpatialLag', 'SHAPE'],

Selecting highly significant spatial clustering county

The most significant (lowest) p-value here is 0.002. All counties with this p value will be selected, and a field will be created that will be used to generate a shapefile containing these highly significant counties. Another field will also be created for counties with p-values less than or equal to 0.05, representing the remaining significantly clustering counties, that will be used as the third spatial feature in the final model.

# creating new fields with highly significant clustering counties
sdf_main_LMi['LMi_hi_sig<.002'] =  np.where(sdf_main_LMi['LMiPValue']<=.002, 1,0)
sdf_main_LMi['LMi_sig_<0.05'] =  np.where(sdf_main_LMi['LMiPValue']<=.05, 1,0)
0000.6137380.0326930.9475220.172440.2115800.154568{"rings": [[[-9619465, 3856529.0001000017], [-...00
1110.6273640.0207920.2846780.376220.3588940.057952{"rings": [[[-9746859, 3539643.0001000017], [-...00
2220.5138160.4327913.6584270.002LL62-0.868722-0.498354{"rings": [[[-9468394, 3771591.0001000017], [-...11
3330.501364-0.287305-1.7233580.042LH43-1.0033410.286440{"rings": [[[-9692114, 3928124.0001000017], [-...01
4440.603064-0.032324-2.0451580.024HL510.096177-0.336198{"rings": [[[-9623907, 4063676.0001000017], [-...01
# create new dataframe for LMi_hi_sig<.002 
LMi_hi_sig_county_main = sdf_main_LMi[sdf_main_LMi['LMi_hi_sig<.002']==1].copy()
Index(['FID', 'SOURCE_ID', 'voter_turn', 'LMiIndex', 'LMiZScore', 'LMiPValue',
       'COType', 'NNeighbors', 'ZTransform', 'SpatialLag', 'SHAPE',
       'LMi_hi_sig<.002', 'LMi_sig_<0.05'],
# creating a new shapefile for the most significant clustering counties from spatial dataframe 
near_dist_from_main_county = sdf_main_LMi.spatial.to_featureclass('voters_turnout_train_LMi'+str(
near_dist_to_hi_sig_county = LMi_hi_sig_county_main.spatial.to_featureclass('LMi_hi_sig_county_train'+str(

Estimating distances and angle of counties from highly clustered counties

The Near(Analysis) tool from Arcpy is used to calculate the distance and the angle of all the counties from the highly significant clustering counties. As input, it takes the counties of high significance from which the distance is to be estimated, followed by the shapefile containing all of the counties to which the distance and the angle is to be calculated.

# Using the Near tool to calculate distance and angle
dist_to_nearest_hi_sig = arcpy.analysis.Near(near_dist_from_main_county,near_dist_to_hi_sig_county,'#','#','ANGLE','GEODESIC')
# Accessing the attribute table from the resulting shapefile
sdf_nearest_hi_sig = pd.DataFrame.spatial.from_featureclass(dist_to_nearest_hi_sig[0])
00000.6137380.0326930.9475220.172440.2115800.154568001236505.87166089.398064{"rings": [[[-9619465, 3856529.0001000017], [-...
11010.6273640.0207920.2846780.376220.3588940.057952007171810.77538269.106286{"rings": [[[-9746859, 3539643.0001000017], [-...
22020.5138160.4327913.6584270.002LL62-0.868722-0.4983540000.0000000.000000{"rings": [[[-9468394, 3771591.0001000017], [-...
33030.501364-0.287305-1.7233580.042LH43-1.0033410.286440001292415.309691119.739256{"rings": [[[-9692114, 3928124.0001000017], [-...
44040.603064-0.032324-2.0451580.024HL510.096177-0.3361980040.0000000.000000{"rings": [[[-9623907, 4063676.0001000017], [-...
Index(['FID', 'Id', 'SOURCE_ID', 'voter_turn', 'LMiIndex', 'LMiZScore',
       'LMiPValue', 'COType', 'NNeighbors', 'ZTransform', 'SpatialLag',
       'LMi_hi_sig', 'LMi_sig_<0', 'NEAR_FID', 'NEAR_DIST', 'NEAR_ANGLE',
Index(['FID', 'SOURCE_ID', 'voter_turn', 'LMiIndex', 'LMiZScore', 'LMiPValue',
       'COType', 'NNeighbors', 'ZTransform', 'SpatialLag', 'SHAPE',
       'LMi_hi_sig<.002', 'LMi_sig_<0.05'],

In the resulting dataframe above, the fields NEAR_DIST and NEAR_ANGLE ( third and the second field from the last) represent the distance and angle of the counties from the highly significant clustering counties, while the field named LMi_sig_<0, represents all of the significant counties. All three will be used as the spatial predictors in the final model.

001101001AutaugaAlabama0.61373838.625553.04.96...440.2115800.154568002.496745e+052.208598e+09133735.2925020{"rings": [[[-9619465, 3856529.0001000017], [-...
111201003BaldwinAlabama0.62736442.931429.04.64...220.3588940.057952001.642763e+065.671096e+09241925.1964263{"rings": [[[-9746859, 3539643.0001000017], [-...

2 rows × 97 columns

# dropping the existing p-values estimated columns from the main table to be replaced by the newly calculated values
sdf_main_final = sdf_main.drop(['SOURCE_ID', 'voter_tu_1',
       'Shape_Leng', 'Shape_Area', 'LMiIndex', 'LMiZScore', 'LMiPValue',
       'COType', 'NNeighbors', 'ZTransform', 'SpatialLag', 'LMi_hi_sig',
       'LMi_normal', 'NEAR_FID', 'Shape_Le_1', 'Shape_Ar_1', 'LMiHiDist',  'SHAPE'], axis=1)

2 rows × 79 columns

Final dataset with spatial cluster variables

# joining the newly calculated spatial features with the main dataset 
sdf_main_final_merged = sdf_main_final.merge(sdf_nearest_hi_sig, on='FID', how='inner')
# checking the final merged data
001101001AutaugaAlabama0.61373838.625553.04.96...440.2115800.154568001236505.87166089.398064{'rings': [[[-9619465, 3856529.0001000017], [-...
111201003BaldwinAlabama0.62736442.931429.04.64...220.3588940.057952007171810.77538269.106286{'rings': [[[-9746859, 3539643.0001000017], [-...

2 rows × 95 columns

Model Building

Next, the dataset containing the new spatial variables will be used to fit the AutoML model for further model improvements.

Train-Test split

Here, the dataset with 3112 samples is split into training and test datasets with a 90 to 10 ratio.

# Splitting data with test size of 10% data for validation 
test_size = 0.10
sdf_train, sdf_test = train_test_split(sdf_main_final_merged, test_size = test_size, random_state=32)
# checking train-test split
(2800, 95)
(312, 95)
224422441224542061HuntingdonPennsylvania0.53447243.023471.03.73...41-0.645395-0.1986090068335718.647361114.935085{'rings': [[[-8647447, 4972564.000100002], [-8...
271027101271148435SuttonTexas0.54777639.331334.03.77...22-0.501568-0.2597700087641735.520773-1.056234{'rings': [[[-11144914, 3540920.0001000017], [...

2 rows × 95 columns

Index(['FID', 'Join_Count', 'TARGET_FID', 'FIPS', 'county', 'state',
       'voter_turn_x', 'gender_med', 'householdi', 'electronic', 'raceandhis',
       'voter_laws', 'educationa', 'educatio_1', 'educatio_2', 'educatio_3',
       'maritalsta', 'F5yearincr', 'F5yearin_1', 'F5yearin_2', 'F5yearin_3',
       'F5yearin_4', 'F5yearin_5', 'F5yearin_6', 'language_a', 'hispanicor',
       'hispanic_1', 'raceandh_1', 'atrisk_avg', 'disposable', 'disposab_1',
       'disposab_2', 'disposab_3', 'disposab_4', 'disposab_5', 'disposab_6',
       'disposab_7', 'disposab_8', 'disposab_9', 'disposa_10', 'househol_1',
       'househol_2', 'househol_3', 'househol_4', 'househol_5', 'househol_6',
       'househol_7', 'househol_8', 'househol_9', 'language_1', 'language_2',
       'households', 'househo_10', 'educatio_4', 'educatio_5', 'educatio_6',
       'educatio_7', 'psychograp', 'psychogr_1', 'financial_', 'financial1',
       'financia_1', 'miscellane', 'state_vote', 'state_vo_1', 'randomized',
       'random_num', 'City10Dist', 'City10Ang', 'City9Dist', 'City9Ang',
       'City8Dist', 'City8Ang', 'City7Dist', 'City7Ang', 'City6Dist',
       'City6Ang', 'City5Dist', 'City5Ang', 'Id', 'SOURCE_ID', 'voter_turn_y',
       'LMiIndex', 'LMiZScore', 'LMiPValue', 'COType', 'NNeighbors',
       'ZTransform', 'SpatialLag', 'LMi_hi_sig', 'LMi_sig_<0', 'NEAR_FID',

Data Preprocessing

Here, X is the list of explanatory variables chosen from the new feature data that will be used for predicting voter turnout. The new spatial cluster features used here are NEAR_DIST, NEAR_ANGLE,LMi_sig_<0 as explained in the previous section. Some additional spatial features (City10Ang, City9Ang,City8Ang etc.) were also included to account for the direction of the counties in terms of the angle of the counties from various grades of cities that were pre-calculated.

Also, the categorical variables are marked with a True value inside of a tuple. The scaler is defined in the preprocessors.

#listing explanatory variables
X =[('county',True), ('state',True),'gender_med', 'householdi', 'electronic', 'raceandhis',
       ('voter_laws',True), 'educationa', 'educatio_1', 'educatio_2', 'educatio_3',
       'maritalsta', 'F5yearincr', 'F5yearin_1', 'F5yearin_2', 'F5yearin_3',
       'F5yearin_4', 'F5yearin_5', 'F5yearin_6', 'language_a', 'hispanicor',
       'hispanic_1', 'raceandh_1', 'atrisk_avg', 'disposable', 'disposab_1',
       'disposab_2', 'disposab_3', 'disposab_4', 'disposab_5', 'disposab_6',
       'disposab_7', 'disposab_8', 'disposab_9', 'disposa_10', 'househol_1',
       'househol_2', 'househol_3', 'househol_4', 'househol_5', 'househol_6',
       'househol_7', 'househol_8', 'househol_9', 'language_1', 'language_2',
       'households', 'househo_10', 'educatio_4', 'educatio_5', 'educatio_6',
       'educatio_7', 'psychograp', 'psychogr_1', 'financial_', 'financial1',
       'financia_1', 'miscellane', 'state_vote', 'state_vo_1',
        'City10Ang', 'City9Dist', 'City9Ang',
       'City8Dist', 'City8Ang', 'City7Dist', 'City7Ang', 'City6Dist',
       'City6Ang', 'City5Dist', 'City5Ang', 'LMi_sig_<0', 'NEAR_DIST', 'NEAR_ANGLE']
# defining the preprocessors for scaling data
preprocessors = [('county', 'state','gender_med', 'householdi', 'electronic', 'raceandhis',
       'voter_laws', 'educationa', 'educatio_1', 'educatio_2', 'educatio_3',
       'maritalsta', 'F5yearincr', 'F5yearin_1', 'F5yearin_2', 'F5yearin_3',
       'F5yearin_4', 'F5yearin_5', 'F5yearin_6', 'language_a', 'hispanicor',
       'hispanic_1', 'raceandh_1', 'atrisk_avg', 'disposable', 'disposab_1',
       'disposab_2', 'disposab_3', 'disposab_4', 'disposab_5', 'disposab_6',
       'disposab_7', 'disposab_8', 'disposab_9', 'disposa_10', 'househol_1',
       'househol_2', 'househol_3', 'househol_4', 'househol_5', 'househol_6',
       'househol_7', 'househol_8', 'househol_9', 'language_1', 'language_2',
       'households', 'househo_10', 'educatio_4', 'educatio_5', 'educatio_6',
       'educatio_7', 'psychograp', 'psychogr_1', 'financial_', 'financial1',
       'financia_1', 'miscellane', 'state_vote', 'state_vo_1',
        'City10Ang', 'City9Dist', 'City9Ang',
       'City8Dist', 'City8Ang', 'City7Dist', 'City7Ang', 'City6Dist',
       'City6Ang', 'City5Dist', 'City5Ang', 'LMi_sig_<0', 'NEAR_DIST', 'NEAR_ANGLE', MinMaxScaler())]
# preparing data for the model
data = prepare_tabulardata(sdf_train,
C:\Users\sup10432\AppData\Local\ESRI\conda\envs\pro_automl_26Octb\lib\site-packages\arcgis\learn\_utils\ UserWarning:

Column county has more than 20 unique value. Sure this is categorical?

C:\Users\sup10432\AppData\Local\ESRI\conda\envs\pro_automl_26Octb\lib\site-packages\arcgis\learn\_utils\ UserWarning:

Column state has more than 20 unique value. Sure this is categorical?

176181.04786073.56328796856.4338630.0000000.000000123.794591863.02494389.12513217583.489697-126.132351...37.033.988.4567.446.10New Jersey0.141027546345.0no_doc0.701084

5 rows × 75 columns

Fitting a random forest model

First, a random forest model is fitted to the new spatial data.

Model Initialization

The MLModel is initialized with the Random Forest model from Scikit-learn (Sklearn), along with its model parameters

# defining the model along with the parameters 
model = MLModel(data, 'sklearn.ensemble.RandomForestRegressor', n_estimators=500, random_state=43)
# validating trained model on test dataset
voter_county_mlmodel_predicted = model.predict(sdf_test, prediction_type='dataframe')
115115111605067JacksonArkansas0.37133641.817956.03.33...44-2.409135-1.08920200270.0000000.000000{'rings': [[[-10133692, 4284820.000100002], [-...0.471453
152915291153029153OzarkMissouri0.59120751.418827.03.92...42-0.032009-0.297026004326336.364318179.584017{'rings': [[[-10253900, 4410465.000100002], [-...0.584314

2 rows × 96 columns

# calculating validation model score
r_square_voter_county_mlmodel_Test = metrics.r2_score(voter_county_mlmodel_predicted['voter_turn_x'], voter_county_mlmodel_predicted['prediction_results']) 
print('r_square_voter_county_mlmodel_Test: ', round(r_square_voter_county_mlmodel_Test,2))
r_square_voter_county_mlmodel_Test:  0.79

The validation r square for the random forest model is satisfactory, and now AutoML will be used to improve it.

Fitting Using AutoML

The same data obtained using the prepare_taulardata function is used as input for the AutoML model. Here, the model is initialized using the Compete mode, which is the best performing option of the available modes.

# initializing AutoML model with the Compete mode 
AutoML_voters_county_obj_compete = AutoML(data, eval_metric='r2', mode='Compete', n_jobs=1)
# training the AutoML model
Neural Network algorithm was disabled because it doesn't support n_jobs parameter.
AutoML directory: AutoML_8
The task is regression with evaluation metric r2
AutoML will use algorithms: ['Linear', 'Decision Tree', 'Random Forest', 'Extra Trees', 'LightGBM', 'Xgboost']
AutoML will stack models
AutoML will ensemble availabe models
AutoML steps: ['adjust_validation', 'simple_algorithms', 'default_algorithms', 'not_so_random', 'kmeans_features', 'insert_random_feature', 'features_selection', 'hill_climbing_1', 'hill_climbing_2', 'boost_on_errors', 'ensemble', 'stack', 'ensemble_stacked']
* Step adjust_validation will try to check up to 1 model
1_DecisionTree r2 0.469355 trained in 1.49 seconds
Adjust validation. Remove: 1_DecisionTree
Validation strategy: 10-fold CV Shuffle
* Step simple_algorithms will try to check up to 4 models
1_DecisionTree r2 0.399429 trained in 13.92 seconds
2_DecisionTree r2 0.448983 trained in 13.09 seconds
3_DecisionTree r2 0.445253 trained in 13.18 seconds
4_Linear r2 0.66238 trained in 13.15 seconds
* Step default_algorithms will try to check up to 4 models
5_Default_LightGBM r2 0.770059 trained in 58.69 seconds
6_Default_Xgboost r2 0.766568 trained in 167.82 seconds
7_Default_RandomForest r2 0.587453 trained in 64.63 seconds
8_Default_ExtraTrees r2 0.526491 trained in 23.81 seconds
* Step not_so_random will try to check up to 36 models
18_LightGBM r2 0.777045 trained in 28.4 seconds
9_Xgboost r2 0.751944 trained in 160.33 seconds
27_RandomForest r2 0.585405 trained in 47.67 seconds
36_ExtraTrees r2 0.505466 trained in 22.52 seconds
19_LightGBM r2 0.754122 trained in 21.66 seconds
10_Xgboost r2 0.744709 trained in 243.49 seconds
28_RandomForest r2 0.522452 trained in 41.12 seconds
37_ExtraTrees r2 0.46222 trained in 21.93 seconds
20_LightGBM r2 0.773785 trained in 44.6 seconds
11_Xgboost r2 0.774746 trained in 42.8 seconds
29_RandomForest r2 0.654542 trained in 87.34 seconds
38_ExtraTrees r2 0.611703 trained in 29.63 seconds
21_LightGBM r2 0.760481 trained in 79.12 seconds
12_Xgboost r2 0.760265 trained in 26.37 seconds
30_RandomForest r2 0.653129 trained in 63.72 seconds
39_ExtraTrees r2 0.591046 trained in 25.27 seconds
22_LightGBM r2 0.748574 trained in 22.28 seconds
13_Xgboost r2 0.769551 trained in 34.65 seconds
31_RandomForest r2 0.682243 trained in 74.34 seconds
40_ExtraTrees r2 0.629549 trained in 28.03 seconds
23_LightGBM r2 0.771215 trained in 40.33 seconds
14_Xgboost r2 0.780234 trained in 39.87 seconds
32_RandomForest r2 0.676783 trained in 86.29 seconds
41_ExtraTrees r2 0.638892 trained in 32.57 seconds
24_LightGBM r2 0.774532 trained in 45.83 seconds
15_Xgboost r2 0.770555 trained in 116.43 seconds
* Step kmeans_features will try to check up to 3 models
14_Xgboost_KMeansFeatures r2 0.767433 trained in 43.52 seconds
18_LightGBM_KMeansFeatures r2 0.772894 trained in 38.32 seconds
11_Xgboost_KMeansFeatures r2 0.767903 trained in 58.95 seconds
* Step insert_random_feature will try to check up to 1 model
14_Xgboost_RandomFeature r2 0.773451 trained in 44.52 seconds
Drop features ['City5Dist', 'raceandh_1', 'hispanic_1', 'househol_3', 'households', 'language_1', 'househo_10', 'state_vo_1', 'househol_2', 'City6Dist', 'gender_med', 'househol_9', 'disposab_3', 'househol_5', 'random_feature', 'hispanicor', 'language_2', 'disposab_2', 'disposab_4', 'househol_6', 'househol_4']
* Step features_selection will try to check up to 4 models
14_Xgboost_SelectedFeatures r2 0.774603 trained in 32.27 seconds
18_LightGBM_SelectedFeatures r2 0.781371 trained in 30.29 seconds
31_RandomForest_SelectedFeatures r2 0.683045 trained in 64.27 seconds
41_ExtraTrees_SelectedFeatures r2 0.640023 trained in 32.86 seconds
* Step hill_climbing_1 will try to check up to 20 models
42_LightGBM_SelectedFeatures r2 0.78418 trained in 33.8 seconds
43_Xgboost r2 0.775862 trained in 40.78 seconds
44_Xgboost r2 0.766297 trained in 30.97 seconds
45_LightGBM r2 0.777641 trained in 36.05 seconds
46_Xgboost r2 0.776048 trained in 54.6 seconds
47_Xgboost r2 0.770999 trained in 41.35 seconds
48_Xgboost_SelectedFeatures r2 0.77757 trained in 38.38 seconds
49_Xgboost_SelectedFeatures r2 0.765088 trained in 29.81 seconds
50_LightGBM r2 0.779526 trained in 38.57 seconds
* Step hill_climbing_2 will try to check up to 17 models
51_LightGBM_SelectedFeatures r2 0.782666 trained in 32.02 seconds
52_LightGBM_SelectedFeatures r2 0.783625 trained in 38.86 seconds
53_LightGBM_SelectedFeatures r2 0.784112 trained in 31.33 seconds
54_LightGBM_SelectedFeatures r2 0.783465 trained in 33.09 seconds
55_Xgboost r2 0.773724 trained in 44.82 seconds
56_Xgboost r2 0.768679 trained in 38.39 seconds
57_LightGBM r2 0.780059 trained in 43.84 seconds
58_LightGBM r2 0.780877 trained in 43.36 seconds
* Step boost_on_errors will try to check up to 1 model
42_LightGBM_SelectedFeatures_BoostOnErrors r2 0.783428 trained in 34.72 seconds
* Step ensemble will try to check up to 1 model
Ensemble r2 0.792354 trained in 8.42 seconds
* Step stack will try to check up to 35 models
42_LightGBM_SelectedFeatures_Stacked r2 0.780498 trained in 26.64 seconds
14_Xgboost_Stacked r2 0.783106 trained in 28.13 seconds
31_RandomForest_SelectedFeatures_Stacked r2 0.786935 trained in 114.92 seconds
41_ExtraTrees_SelectedFeatures_Stacked r2 0.791264 trained in 39.03 seconds
53_LightGBM_SelectedFeatures_Stacked r2 0.783177 trained in 24.41 seconds
48_Xgboost_SelectedFeatures_Stacked r2 0.783833 trained in 29.0 seconds
31_RandomForest_Stacked r2 0.790533 trained in 109.56 seconds
41_ExtraTrees_Stacked r2 0.791692 trained in 41.03 seconds
52_LightGBM_SelectedFeatures_Stacked r2 0.779776 trained in 27.97 seconds
46_Xgboost_Stacked r2 0.776083 trained in 38.06 seconds
32_RandomForest_Stacked r2 0.786649 trained in 144.92 seconds
* Step ensemble_stacked will try to check up to 1 model
Ensemble_Stacked r2 0.795538 trained in 12.06 seconds
AutoML fit time: 3621.83 seconds
AutoML best model: Ensemble_Stacked
All the evaluated models are saved in the path  C:\Users\sup10432\review_notebooks\voters_turnout\part II\2\AutoML_8

Here, the ensemble model is the best model, and its R-squared validation score shows the final improvements achieved after including the new, spatially engineered variables. The best model diagnostics and related reports, like feature importance, model performance, etc., are saved in the folder mentioned in the output message for further reference.

# train score of the model

Model output

# The output diagnostics can also be printed in a report form
C:\Users\sup10432\AppData\Local\ESRI\conda\envs\pro_automl_26Octb\lib\site-packages\arcgis\learn\models\ UserWarning:

In case the report html is not rendered appropriately in the notebook, the same can be found in the path AutoML_8\README.html

AutoML Leaderboard

Best modelnamemodel_typemetric_typemetric_valuetrain_time
1_DecisionTreeDecision Treer20.39942914.33
2_DecisionTreeDecision Treer20.44898313.5
3_DecisionTreeDecision Treer20.44525313.6
7_Default_RandomForestRandom Forestr20.58745365.2
8_Default_ExtraTreesExtra Treesr20.52649124.35
27_RandomForestRandom Forestr20.58540548.18
36_ExtraTreesExtra Treesr20.50546623.13
28_RandomForestRandom Forestr20.52245241.72
37_ExtraTreesExtra Treesr20.4622222.43
29_RandomForestRandom Forestr20.65454287.83
38_ExtraTreesExtra Treesr20.61170330.18
30_RandomForestRandom Forestr20.65312964.25
39_ExtraTreesExtra Treesr20.59104625.79
31_RandomForestRandom Forestr20.68224374.91
40_ExtraTreesExtra Treesr20.62954928.58
32_RandomForestRandom Forestr20.67678386.82
41_ExtraTreesExtra Treesr20.63889233.17
31_RandomForest_SelectedFeaturesRandom Forestr20.68304564.85
41_ExtraTrees_SelectedFeaturesExtra Treesr20.64002333.41
31_RandomForest_SelectedFeatures_StackedRandom Forestr20.786935115.47
41_ExtraTrees_SelectedFeatures_StackedExtra Treesr20.79126439.52
31_RandomForest_StackedRandom Forestr20.790533110.11
41_ExtraTrees_StackedExtra Treesr20.79169241.59
32_RandomForest_StackedRandom Forestr20.786649145.44
the bestEnsemble_StackedEnsembler20.79553812.06

AutoML Performance

AutoML Performance

AutoML Performance Boxplot

AutoML Performance Boxplot

Spearman Correlation of Models

models spearman correlation

Voter turnout prediction & Validation

# validating trained model on test dataset
voter_county_automl_predicted = AutoML_voters_county_obj_compete.predict(sdf_test, prediction_type='dataframe')
115115111605067JacksonArkansas0.37133641.817956.03.33...44-2.409135-1.08920200270.0000000.000000{'rings': [[[-10133692, 4284820.000100002], [-...0.423994
152915291153029153OzarkMissouri0.59120751.418827.03.92...42-0.032009-0.297026004326336.364318179.584017{'rings': [[[-10253900, 4410465.000100002], [-...0.584422

2 rows × 96 columns

Estimate model metrics for validation

import sklearn.metrics as metrics
r_square_voter_county_automl_Test = metrics.r2_score(voter_county_automl_predicted['voter_turn_x'], voter_county_automl_predicted['prediction_results']) 
print('r_square_voter_county_automl_Test: ', round(r_square_voter_county_automl_Test,2))
r_square_voter_county_automl_Test:  0.84


In the first part of this notebook series, AutoML was applied to a regression dataset, where it was able to achieve significant improvements over traditional methods of modeling. In this notebook, the model's fit was further improved by extracting the spatial patterns in the voter turnout dataset and including them as additional spatial features.

The spatial feature engineering employed consisted of calculating the spatial autocorrelation in the data using the cluster outlier analysis tool from Arcpy, followed by measuring the distances of each county and their respective angles from the highly significant clustering counties. Including these new spatial variables enhanced the model further. Similarly, this process could be applied to other spatial dataframes.

Data resources & References

Voters turnout by county for 2016 US general electionEsri

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.