Table of Contents¶
The ability of SAR data to let us see through clouds make it more valuable specially in cloudy areas and bad weather. This is the time when earth observation can reap maximum benefits, but optical sensors prevent us doing that. Now a days a lot of organizations are investing in SAR data making it more available to users than before. The only disadvantage of SAR data is the unavailability of labelled data as it is more difficult for users to understand and label SAR data than optical imagery.
In this sample notebook, we will see how we can make use of benefits of SAR and optical imagery to perform all season earth observation. We will train a deep learning model to translate SAR imagery to RGB imagery, thereby making optical data (translated) available even in extreme weather days and cloudy areas.
import os, zipfile from pathlib import Path from arcgis.gis import GIS from arcgis.learn import prepare_data, CycleGAN
# Connect to GIS gis = GIS('home')
For this usecase, we have SAR imagery from Capella Space and world imagery in the form of RGB tiles near Rotterdam city in the Netherlands. We have exported that data in a new “Export_Tiles” metadata format available in the
Export Training Data For Deep Learning tool. This
Export Training Data For Deep Learning tool available in ArcGIS Pro as well as ArcGIS Image Server.
Input Raster: SAR imagery tile
Tile Size X & Tile Size Y: 256
Stride X & Stride Y: 128
Meta Data Format: 'Export_Tiles' as we are training a
Environments: Set optimum
Now, in case of exporting RGB imagery into chips, let us first create a tile cache of world imagery for the area of extent, on which we are working. We can do this using Manage Tile Cache tool available in Geoprocessing tools.
Cache Location: provide a path to save tile cache.
Cache Name: choose a name to the tile cache.
Manage Mode: Recreate all tiles.
Input Data Source: select the map with basemap set as imagery. Uncheck all the layers on the map except the imagery basemap.
Scale: select the appropriate scales at which you want imagery cache.
The cache once exported, will be automatically added to the ArcGIS Pro project. Use
Export training Data for Deep Learning tool as we did in the case of SAR imagery to export image chips from the cache.
Once, both the imagery are exported in the required format, we need to follow the directory structure shown in figure 3 that
CycleGAN model follows.
Here, 'train_a' and 'train_b' folders contain all the image tiles (excluding other files like 'esri_accumulated_stats.json', 'esri_model_definition.emd', 'map.txt', 'stats.txt') exported from SAR imagery and world imagery map respectively. Now we are ready to train the
Alternatively, we have provided a subset of training data containing a few samples that follows the same directory structure mentioned above. You can use the data directly to run the experiments.
training_data = gis.content.get('25ed4a30219e4ba7acb3633e1a75bae1') training_data
filepath = training_data.download(file_name=training_data.name)
with zipfile.ZipFile(filepath, 'r') as zip_ref: zip_ref.extractall(Path(filepath).parent)
output_path = Path(os.path.join(os.path.splitext(filepath)))
We will train CycleGAN model  that performs the task of Image-to-Image translation where it learns mapping between input and output images using unpaired dataset. This model is an extension of GAN architecture which involves simultaneous training of two generator models and two discriminator models. In GAN, we can generate images of domain Y from domain X, but in CycleGAN, we can also generate images of domain X from domain Y using the same model architecture.
It has two mapping functions: G : X → Y and F : Y → X, and associated adversarial discriminators Dy and Dx. G tries to generate images that look similar to images from domain Y, while Dy aims to distinguish between translated samples G(x) and real samples y. G aims to minimize this objective against an adversary D that tries to maximize it. The same process happens in generation of the images of domain X from domain Y using F as a generator and Dx as a discriminator.
We will specify the path to our training data and a few hyperparameters.
path: path of the folder containing training data.
batch_size: Number of images your model will train on each step inside an epoch, it directly depends on the memory of your graphic card. 4 worked for us on a 11GB GPU.
# output_path = r"D:\CycleGAN\Data\data_for_cyclegan_le_3Bands" data = prepare_data(output_path, dataset_type="CycleGAN", batch_size=4)
ImageDataBunch; Train: LabelList (2773 items) x: ImageTupleList ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])) y: EmptyLabelList ,,,, Path: C:\Users\gun10635\AppData\Local\Temp\sar_to_rgb_image_translation_using_cyclegan\Images; Valid: LabelList (308 items) x: ImageTupleList ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])),ImageTuple(torch.Size([3, 256, 256]), torch.Size([3, 256, 256])) y: EmptyLabelList ,,,, Path: C:\Users\gun10635\AppData\Local\Temp\sar_to_rgb_image_translation_using_cyclegan\Images; Test: None
To get a sense of what the training data looks like,
arcgis.learn.show_batch() method randomly picks a few training chips and visualizes them.
rows: Number of rows to visualize