ArcGIS Developers
Dashboard

ArcGIS API for Python

Download the samples Try it live

Finding a New Home

Buying or selling of a house can be a very stressful event in one's life. The process could be frustrating as it is lengthy, uncertain and needs a lot of examination. Through this workflow we will guide a couple (Mark and Lisa) who is interested in selling their home and relocating to a place nearest to both of their work places. In this case study, we will explore the current housing market, estimate average house prices in their area and hunt for a new one. We will download the Zillow data for their current home for our analysis. You can use your own data and follow along with this workflow which aims to help Mark and Lisa in finding their new home.

The notebook is divided into two parts. In the first part, we will calculate the following:

  • Percentage of decrease/increase in house price since Mark and Lisa bought their home.
  • Suggested selling price for their home.
  • Whether their zip code is a buyer’s market or seller’s market.
  • Average number of days it takes for homes to sell in their neighbourhood.

In the second part of the notebook, we will explore the investment potential of homes close to their work places. Based on how much a person is willing to spend commuting to work, we will create a drive-time buffer. This will narrow down the search areas. Zillow also provides data for market health and projected home value appreciation. Visualizing the zip codes by their market health will help them focus only on areas with good market health. Hence they will get a list of areas to choose from, for buying their new home.

Selling your home

Determine an appropriate selling price

1) Download home sales time series data from Zillow at www.zillow.com/research/data.

Mark and Lisa have a 3-bedroom home, so we will select the ZHVI 3-Bedroom time-series ($) data set at the ZIP Code level.

2) Prepare the Excel data as follows:

a) Using Excel, open the .csv file.

Notice that the RegionName field has ZIP Codes as numbers (if we sort the RegionName field we will notice the ZIP Codes for Massachusetts, for example, don't have leading zeros; 01001 is 1001). Also, notice the median home value columns are named using the year and month. The first data available is for April 1996 (1996-04). b) Copy all the column headings and the one record with data for their ZIP Code to a new Excel sheet.

Apply a filter to the RegionName field. Mark and Lisa live in Crestline, California, so we will apply a filter for the 92325 ZIP Code.

c) Select (highlight) fields starting with the month and year when they bought their home and continuing to the last month and year column in the Excel table. So, for example, since Mark and Lisa bought their home in December 2007, they highlight the the two rows from column 2007-01 to column 2018-08.

d) Copy (press Ctrl+C) the selected data and paste it, along with the column headings, to a new Excel sheet using Paste Transposed (right-click in the first cell of the new sheet to see the paste options; select Paste Transposed). This gives two columns of data.

e) The first column has date values but only includes the year and month. In column C, create a proper date field.

  • Right-click column C and format the cells to be category date.
  • In the first cell of column C, enter the following formula: = DATEVALUE(CONCATENATE(A1, "-01"))
  • Drag the Autofill handle down to the last data cell in the column.

f) Insert a top row and type the column headings:

YYYYMM, Value, and date.

g) Rename the Excel sheet (probably called Sheet2 at present) something like AveSellingPrice and delete the other sheets (the first sheet contains a large amount of data that we won't be using further in the workflow).

Mark and Lisa named their price Excel sheet CrestlineAveSellingPrice.

h) Save this new sheet as an Excel workbook.

Mark and Lisa named their Excel file Crestline3BdrmAveSellingPrice.xlsx.

3) Connect to your ArcGIS Online organization.

In [1]:
from arcgis.gis import GIS
In [2]:
import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime as dt
In [3]:
gis = GIS('home')

4) Load the excel file for analysis.

In [4]:
data = gis.content.search('finding_a_new_home owner:api_data_owner type: csv collection')[0]
data
Out[4]:
finding_a_new_home
CSV Collection by api_data_owner
Last Modified: March 17, 2021
0 comments, 0 views
In [5]:
filepath = data.download(file_name=data.name)
In [6]:
import os
import zipfile
from pathlib import Path
with zipfile.ZipFile(filepath, 'r') as zip_ref:
    zip_ref.extractall(Path(filepath).parent)
In [7]:
data_path = Path(os.path.join(os.path.splitext(filepath)[0]))
data_path
In [8]:
datapath = [os.path.abspath(os.path.join(data_path, p)) for p in os.listdir(data_path)]
datapath
Out[8]:
['C:\\Users\\shi10484\\AppData\\Local\\Temp\\finding_a_new_home\\BuyerSellerIndex.xlsx',
 'C:\\Users\\shi10484\\AppData\\Local\\Temp\\finding_a_new_home\\Crestline3BdrmAveSellingPrice.xlsx',
 'C:\\Users\\shi10484\\AppData\\Local\\Temp\\finding_a_new_home\\ImportantPlaces.xlsx',
 'C:\\Users\\shi10484\\AppData\\Local\\Temp\\finding_a_new_home\\MarketHealthIndex.xlsx']
In [9]:
file_name1 = datapath[1]
data1 = pd.pandas.read_excel(file_name1)
In [10]:
data1.head()
Out[10]:
YYYYMM value date
0 2007-01 291000 2007-01-01
1 2007-02 289000 2007-02-01
2 2007-03 287400 2007-03-01
3 2007-04 286100 2007-04-01
4 2007-05 284000 2007-05-01
In [11]:
data1.tail()
Out[11]:
YYYYMM value date
135 2018-04 252200 2018-04-01
136 2018-05 254000 2018-05-01
137 2018-06 254800 2018-06-01
138 2018-07 254900 2018-07-01
139 2018-08 254900 2018-08-01
In [12]:
data1.shape
Out[12]:
(140, 3)
In [13]:
data1[['year','month','day']] = data1.date.apply(lambda x: pd.Series(
    x.strftime("%Y,%m,%d").split(","))) # split date into year, month, day
In [14]:
data1.head()
Out[14]:
YYYYMM value date year month day
0 2007-01 291000 2007-01-01 2007 01 01
1 2007-02 289000 2007-02-01 2007 02 01
2 2007-03 287400 2007-03-01 2007 03 01
3 2007-04 286100 2007-04-01 2007 04 01
4 2007-05 284000 2007-05-01 2007 05 01
In [15]:
grpby_data1 = data1.groupby(['year']).mean()
In [16]:
type(grpby_data1)
Out[16]:
pandas.core.frame.DataFrame

5) We will Create a graph showing how average home prices have changed since they bought their home.

In [17]:
grpby_data1.reset_index(inplace=True)
In [18]:
grpby_data1.head()
Out[18]:
year value
0 2007 276616.666667
1 2008 221875.000000
2 2009 188391.666667
3 2010 176216.666667
4 2011 154766.666667
In [19]:
grpby_data1.value
Out[19]:
0     276616.666667
1     221875.000000
2     188391.666667
3     176216.666667
4     154766.666667
5     145158.333333
6     163741.666667
7     183975.000000
8     197133.333333
9     229566.666667
10    235966.666667
11    252025.000000
Name: value, dtype: float64
In [20]:
grpby_data1.year
Out[20]:
0     2007
1     2008
2     2009
3     2010
4     2011
5     2012
6     2013
7     2014
8     2015
9     2016
10    2017
11    2018
Name: year, dtype: object
In [21]:
plt.plot(grpby_data1.year, grpby_data1.value)
plt.title("average home prices (2007-2018)")
plt.xlabel("date")
plt.ylabel("average house price")
Out[21]:
Text(0, 0.5, 'average house price')

7) Determine an appropriate selling price based on home sales trends as follows:

a) Determine the current average selling price and the average selling price when they bought their home. Divide the current average price by the beginning average price to see how much homes in their ZIP Code have appreciated or depreciated. When Mark and Lisa bought their home in December of 2007, 3-bedroom homes were selling for \$276,617.

In [22]:
price_initial = grpby_data1.iloc[0]
In [23]:
price_initial
Out[23]:
year       2007
value    276617
Name: 0, dtype: object
In [24]:
price_current = grpby_data1.iloc[-1]
In [25]:
price_current
Out[25]:
year       2018
value    252025
Name: 11, dtype: object
In [26]:
house_worth = price_current.value / price_initial.value
In [27]:
house_worth
Out[27]:
0.9110983912755316

This indicates that homes in Crestline are only worth 91 percent of what they were at the end of 2007.

b) We can get a rough estimate of what their home is worth by summing what they paid for their home plus what they invested in it, and multiplying that sum by the ratio computed above. Mark and Lisa, for example, paid \$291,000 in 2007 and invested \\$100,000 in solid improvements (new kitchen, major landscaping, hardwood flooring, and so on). Multiplying (\$291,000 + \\$100,000) by 0.91 gives a rough suggested selling price of \$343,134.

In [28]:
(price_initial.value + 100000)*house_worth
Out[28]:
343134.83912755316

Get additional information about the local real estate market

If their home is part of a seller's market, they are more likely to get their asking price.

1) Download the Buyer-Seller Index data at the ZIP Code level from www.zillow.com/research/data. In Home Listings and Sales select data type as seller-buyer index and geography as zip codes.

2) Open the .csv file using Excel. Zillow reports ZIP Codes as numbers. We will need to pad the ZIP Code numbers with leading zeros so the Zillow data will link to the ArcGIS ZIP Code geometry.

Follow these steps:

a) Sort the RegionName column from smallest to largest so we will be able to see how the formula below works.

b) Name a new column in the Excel table zipstring.

c) In the first cell of the new column, enter the formula to pad each RegionName value with leading zeros, keeping the rightmost five characters: =RIGHT(CONCAT("00000",B2),5)

d) Drag the Autofill handle down to the last data cell in the column.

3) Load the excel file for analysis

In [29]:
file_name2 = datapath[0]
data2 = pd.read_excel(file_name2)
In [30]:
data2.head()
Out[30]:
RegionType RegionName State CBSA Title SizeRankCity SizeRankMetro PctPriceCut DaysOnMarket BuyerSellerIndex BuyerSellerIndexMetro zipstring
0 Zip 1001 MA Springfield, MA 8273 83 22.916667 81.0 6.190476 9.185083 1001
1 Zip 1002 MA Springfield, MA 6684 83 29.787234 94.0 10.000000 9.185083 1002
2 Zip 1007 MA Springfield, MA 9088 83 16.091954 83.0 2.857143 9.185083 1007
3 Zip 1013 MA Springfield, MA 7061 83 31.147541 76.5 8.333333 9.185083 1013
4 Zip 1020 MA Springfield, MA 5172 83 25.000000 73.0 3.809524 9.185083 1020
In [31]:
data2.dtypes
Out[31]:
RegionType                object
RegionName                 int64
State                     object
CBSA Title                object
SizeRankCity               int64
SizeRankMetro              int64
PctPriceCut              float64
DaysOnMarket             float64
BuyerSellerIndex         float64
BuyerSellerIndexMetro    float64
zipstring                  int64
dtype: object
In [32]:
data2.columns
Out[32]:
Index(['RegionType', 'RegionName', 'State', 'CBSA Title', 'SizeRankCity',
       'SizeRankMetro', 'PctPriceCut', 'DaysOnMarket', 'BuyerSellerIndex',
       'BuyerSellerIndexMetro', 'zipstring'],
      dtype='object')

4) Select BuyerSellerIndex, DaysOnMarket, zipstring fields

In [33]:
cols = ['BuyerSellerIndex', 'DaysOnMarket', 'zipstring']
In [34]:
selected_data2 = data2[cols]

5) Explore the data values as follows:

a) Sort on the DaysOnMarket field and notice the range. For the data Mark and Lisa downloaded, the range is 35 to 294.5 days.

In [35]:
days_on_mrkt_df = selected_data2.sort_values(by='DaysOnMarket', axis=0)
In [36]:
days_on_mrkt_df.DaysOnMarket.min()
Out[36]:
35.0
In [37]:
days_on_mrkt_df.DaysOnMarket.max()
Out[37]:
294.5
In [38]:
days_on_mrkt_df.head()
Out[38]:
BuyerSellerIndex DaysOnMarket zipstring
7420 1.120000 35.0 98107
7434 4.960000 35.0 98136
7097 0.526316 35.0 94022
7426 0.640000 35.0 98117
7399 0.080000 35.0 98043
In [39]:
days_on_mrkt_df.tail()
Out[39]:
BuyerSellerIndex DaysOnMarket zipstring
800 9.006849 252.0 8752
1177 10.000000 254.0 12428
1141 8.938356 261.5 11963
1189 9.160959 288.0 12545
753 6.000000 294.5 8403

b) Sort on the BuyerSellerIndex field and notice the range of values. ZIP Codes with index values near 0 are part of a strong seller's market; ZIP Codes with index values near 10 are part of a strong buyer's market.

In [40]:
buyer_seller_df = data2.sort_values(by='BuyerSellerIndex', axis=0)
In [41]:
buyer_seller_df.head()
Out[41]:
RegionType RegionName State CBSA Title SizeRankCity SizeRankMetro PctPriceCut DaysOnMarket BuyerSellerIndex BuyerSellerIndexMetro zipstring
517 Zip 7063 NJ New York-Newark-Jersey City, NY-NJ-PA 10423 1 4.687500 90.0 0.017123 9.585635 7063
4546 Zip 53168 WI Chicago-Naperville-Elgin, IL-IN-WI 10505 3 9.756098 59.5 0.033223 9.488950 53168
485 Zip 7004 NJ New York-Newark-Jersey City, NY-NJ-PA 10846 1 7.894737 92.0 0.034247 9.585635 7004
6572 Zip 90025 CA Los Angeles-Long Beach-Anaheim, CA 2122 2 6.451613 46.0 0.036364 4.185083 90025
1609 Zip 19152 PA Philadelphia-Camden-Wilmington, PA-NJ-DE-MD 4927 5 6.818182 67.0 0.041667 9.150552 19152
In [42]:
buyer_seller_df.tail()
Out[42]:
RegionType RegionName State CBSA Title SizeRankCity SizeRankMetro PctPriceCut DaysOnMarket BuyerSellerIndex BuyerSellerIndexMetro zipstring
6091 Zip 80016 CO Denver-Aurora-Lakewood, CO 2658 21 27.226891 62.0 10.0 4.502762 80016
4598 Zip 53954 WI Madison, WI 10773 87 21.875000 75.5 10.0 3.522099 53954
262 Zip 3049 NH Manchester-Nashua, NH 10693 131 28.787879 74.0 10.0 3.577348 3049
7321 Zip 97209 OR Portland-Vancouver-Hillsboro, OR-WA 4742 23 26.258993 78.5 10.0 5.096685 97209
1224 Zip 13104 NY Syracuse, NY 8916 79 28.323699 89.0 10.0 9.033149 13104
In [43]:
buyer_seller_df.BuyerSellerIndex.min()
Out[43]:
0.017123288
In [44]:
buyer_seller_df.BuyerSellerIndex.max()
Out[44]:
10.0

c) Filter the data to only show their home's ZIP code

In [45]:
selected_data2[selected_data2['zipstring'] == 92325]
Out[45]:
BuyerSellerIndex DaysOnMarket zipstring
6888 9.622642 79.0 92325

Notice the average number of days. Determine if the home is part of a buyer's or seller's market. Mark and Lisa learn that their home is part of a buyer's market (9.6), and they can expect their home to be on the market approximately 79 days before it sells.

In [46]:
selected_data2.rename(columns={"zipstring": "ZIP_CODE"}, inplace=True)
C:\Users\shi10484\AppData\Local\ESRI\conda\envs\dl_testing2\lib\site-packages\pandas\core\frame.py:4301: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,
In [47]:
selected_data2.shape
Out[47]:
(7548, 3)
In [48]:
selected_data2.dtypes
Out[48]:
BuyerSellerIndex    float64
DaysOnMarket        float64
ZIP_CODE              int64
dtype: object
In [49]:
selected_data2 = selected_data2.astype({"ZIP_CODE": int})
In [50]:
selected_data2.dtypes
Out[50]:
BuyerSellerIndex    float64
DaysOnMarket        float64
ZIP_CODE              int32
dtype: object

6) Search for the United States ZIP Code Boundaries 2017 layer. We can specify the owner's name to get more specific results. To search for content from the Living Atlas, or content shared by other users on ArcGIS Online, set outside_org=True

In [51]:
items = gis.content.search('United States ZIP Code Boundaries 2017 owner: esri_dm',
                           outside_org=True)

Display the list of results.

In [52]:
from IPython.display import display

for item in items:
    display(item)
United States ZIP Code Boundaries 2017
This layer shows the ZIP Code level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 82,049 views
USA ZIP Code Areas
This group layer presents five-digit ZIP Code areas used by the U.S. Postal Service to deliver mail more effectively.Layer Package by esri_dm
Last Modified: February 12, 2020
12 comments, 1,58,207 views
United States Tract Boundaries 2017
This layer shows the Tract level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 969 views
United States Block Group Boundaries 2017
This layer shows the Block Group level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 1,713 views
United States Country Boundary 2017
This layer shows the Country boundary of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 3,664 views
United States County Boundaries 2017
This layer shows the County level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 8,027 views
United States State Boundaries 2017
This layer shows the State level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 41,609 views

Select the desired item from the list.

In [53]:
us_zip = items[0]
In [54]:
us_zip
Out[54]:
United States ZIP Code Boundaries 2017
This layer shows the ZIP Code level boundaries of United States in 2017. The boundaries are optimized to improve Data Enrichment analysis performance.Feature Layer Collection by esri_dm
Last Modified: June 21, 2019
0 comments, 82,049 views

Get the layer names from the item

In [55]:
for lyr in us_zip.layers:
    print(lyr.properties.name)
USA_Country
USA_State
USA_County
USA_ZipCode
USA_Tract
USA_BlockGroup

7) We want to merge the zip_code layer with data2 to visualize the result on the map.

In [56]:
us_zip_lyr = us_zip.layers[3]
In [57]:
zip_df = pd.DataFrame.spatial.from_layer(us_zip_lyr)
In [58]:
zip_df.head()
Out[58]:
OBJECTID POPULATION PO_NAME SHAPE SQMI STATE Shape__Area Shape__Length ZIP_CODE
0 1 -99 N Dillingham Census Area {"rings": [[[-160.186183929443, 58.82004642486... 16279.47 AK 6.765048 24.602921 00001
1 2 -99 Yukon Flats Nat Wildlife {"rings": [[[-159.971336364746, 64.42843627929... 95704.72 AK 48.867324 130.944574 00002
2 3 -99 Alaska Peninsula NWR {"rings": [[[-159.347519999648, 55.77196200034... 14491.70 AK 5.622721 41.443107 00003
3 4 -99 W Kenai Peninsula Boroug {"rings": [[[-153.309794000393, 58.85487400023... 6568.13 AK 2.751546 20.460970 00004
4 5 -99 N Lake and Peninsula Bor {"rings": [[[-153.436194999999, 60.90853799962... 3713.14 AK 1.573790 9.474710 00005
In [59]:
zip_df.shape
Out[59]:
(30924, 9)
In [60]:
zip_df.dtypes
Out[60]:
OBJECTID            int64
POPULATION          int64
PO_NAME            object
SHAPE            geometry
SQMI              float64
STATE              object
Shape__Area       float64
Shape__Length     float64
ZIP_CODE           object
dtype: object
In [61]:
zip_df = zip_df.astype({"ZIP_CODE": int})
In [62]:
zip_df.dtypes
Out[62]:
OBJECTID            int64
POPULATION          int64
PO_NAME            object
SHAPE            geometry
SQMI              float64
STATE              object
Shape__Area       float64
Shape__Length     float64
ZIP_CODE            int32
dtype: object
In [63]:
merged_df = pd.merge(zip_df, selected_data2, on='ZIP_CODE')
In [64]:
merged_df.shape
Out[64]:
(7547, 11)
In [65]:
merged_df.spatial.set_geometry('SHAPE')
In [66]:
mergd_lyr = gis.content.import_data(merged_df,
                                    title='MergedLayer',
                                    tags='datascience')

8) Create a map of the BuyerSellerIndex field using the following steps:

Visualize Results

In [67]:
m1 = gis.map('United States', 8)
m1
Out[67]: