Data Preparation - Hurricane analysis, part 1/3¶

Table of Contents¶

• Introduction
• Download hurricane data from NCEI FTP portal
  • Download each file into the hurricanes_raw directory
• Process CSV files by removing header rows
  • Automate across all files
• Cleaning hurricane observations with Dask
  • Read input CSV data
  • Merge all location columns
  • Merge similar columns
    • Merge wind columns
      • Merge pressure columns
      • Merge grade columns
      • Merge eye diameter columns
    • Identify remaining redundant columns
    • Drop all redundant columns
  • Perform delayed computation
    • Preview results
• Creating hurricane tracks using Geoanalytics
  • Create a data store
  • Perform data aggregation using reconstruct tracks tool
    • Execute reconstruct tracks tool
    • Analyze the result of aggregation
• Conclusion

Introduction¶

Hurricanes are large swirling storms that produce winds of speeds 74 miles per hour (119 kmph) or higher. When hurricanes make a landfall, they produce heavy rainfall, cause storm surges and intense flooding. Often hurricanes strike places that are dense in population, causing devastating amounts of death and destruction throughout the world.

Since the recent past, agencies such as the National Hurricane Center have been collecting quantitative data about hurricanes. In this study we use meteorological data of hurricanes recorded in the past 169 years to analyze their location, intensity and investigate if there are any statistically significant trends. We also analyze the places most affected by hurricanes and what their demographic make up is. We conclude by citing relevant articles that draw similar conclusions.

This notebook covers part 1 of this study. In this notebook, we:

• download data from NCEI portal
• do extensive pre-processing in the form of clearing headers, merging redundant columns
• aggregate the observations into hurricane tracks.

Note: To run this sample, you need a few extra libraries in your conda environment. If you don't have the libraries, install them by running the following commands from cmd.exe or your shell.

Download hurricane data from NCEI FTP portal¶

The National Centers for Environmental Information, formerly National Climatic Data Center shares the historic hurricane track datasets at ftp://eclipse.ncdc.noaa.gov/pub/ibtracs/v03r09/all/csv/. We use the ftplib Python library to login in and download these datasets.

Establish an anonymous connection to FTP site.

Change directory to folder containing the hurricane files. List the files.

Print the top 10 items.

Download each file into the hurricanes_raw directory¶

Now we are going to download data from 1842-2017, whih might take around 15 mins.

Process CSV files by removing header rows¶

The CSV files have multiple header rows. Let us start by processing one of the files as an example.

The input looks mangled. This is because the file's row 1 has a header that pandas fails to read. So let us skip that row.

A little better. But the file's 3rd row is also a header. Let us drop that row.

Automate across all files¶

Now we need to repeat the above cleaning steps across all CSV files. In the steps below, we will read all CSV files, drop the headers, and write to disk. This step is necessary as it will ease subsequent processing using the DASK library.

Cleaning hurricane observations with Dask¶

The data collected from NOAA NCDC source is just too large to clean with Pandas or Excel. With 350,000 x 200 in dense matrix, this data is larger than memory for a normal computer. Hence traditional packages such as Pandas cannot be used as they expect data to fit fully in memory.

Thus, in this part of the study, we use Dask, a distributed data analysis library. Functionally, Dask provides a DataFrame object that behaves similar to a traditional pandas DataFrame object. You can perform slicing, dicing, exploration on them. However transformative operations on the DataFrame get queued and are operated only when necessary. When executed, Dask will read data in chunks, distribute it to workers (be it cores on a single machine or multiple machines in a cluster set up) and collect the data back for you. Thus, DASK allows you to work with any larger than memory dataset as it performs operations on chunks of it, in a distributed manner.

Read input CSV data¶

As mentioned earlier, DASK allows you to work with larger than memory datasets. These datasets can reside as one large file or as multiple files in a folder. For the latter, DASK allows you to just specify the folder containing the datasets as input. In turn, it provides you a single DataFrame object that represents all your datasets combined together. The operations you perform on this DataFrame get queued and executed only when necessary.

Preemptively, specify the assortment of values that should be treated as null values.

You can query the top few (or bottom few) records as you do on a regular Pandas DataFrame object.

Drop the first duplicate index column.

This dataset has 200 columns. Not all are unique, as you can see from the print out below:

Reading the metadata from NOAA NCDC site, we find sensor measurements get unique columns if they are collected by a different agency. Thus we find multiple pressure, wind speed, latitude, longitude, etc. columns with different suffixes and prefixes. Data is sparse as it gets distributed between these columns. For our geospatial analysis, it suffices if we can merge these columns together and get location information from the coordinates.

Merge all location columns¶

Below we prototype merging location columns. If this succeeds, we will proceed to merge all remaining columns.

In this dataset, if data is collected by 1 agency, the corresponding duplicate columns from other agencies are empty. However there may be exceptions. Hence we define a custom function that will pick median value for a row, from a given list of columns. This way, we can consolidate latitude / longitude information from all the agencies.

With dask, the above operation was delayed and stored in a queue. It has not been evaluated yet. Next, let us evaluate for 5 records and print output. If results look good, we will merge all remaining related columns together.

The results look good. Two additional columns (latitude_merged, longitude_merged) have been added. By merging related columns, the redundant sparse columns can be removed, thereby simplifying the dimension of the input dataset.

Now that this prototype looks good, we will proceed by identifying the lists of remaining columns that are redundant and can be merged.

Merge similar columns¶

To keep track of which columns have been accounted for, we will duplicate the all_columns list and remove ones that we have identified.

From the columns_tracker list, let us remove the redundant columns we already identified for location columns.

Thus, we have reduced the number of columns from 200 to 142. We will progressively reduce this while retaining key information.

Merge wind columns¶

Wind, pressure, grade are some of the meteorological observations this dataset contains. To start off, let us identify the wind columns:

Merge pressure columns¶

We proceed to identify all pressure columns. But before that, we update the columns_tracker list by removing those we identified for wind:

Merge grade columns¶

Notice the length of columns_tracker is reducing progressively as we identify redundant columns.

Merge eye diameter columns¶

Identify remaining redundant columns¶

We are down to 49 columns, let us visualize what those look like.

Based on metadata shared by data provider, we choose to retain only the first 11 columns. We add the rest to the list columns_to_eliminate.

Drop all redundant columns¶

So far, we have merged similar columns together and collected the lists of redundant columns to drop. Below we compile them into a single list.

Perform delayed computation¶

In Dask, all computations are delayed and queued. The apply() functions called earlier are not executed yet, however respective columns have been created as you can see from the DataFrame display above. In the cells below, we will call save() to make Dask compute on this larger than memory dataset.

Calling visualize() on the delayed compute operation or the DataFrame object will plot the dask task queue as shown below. The graphic below provides a glimpse on how Dask distributes its tasks and how it reads this 'larger than memory dataset' in chunks and operates on them.

Drawing dask graphs requires the graphviz python library and the graphviz system library to be installed.