SAR to RGB image translation using CycleGAN¶
Table of Contents¶
Introduction¶
The ability of SAR data to let us see through clouds make it more valuable specially in cloudy areas and bad weather. This is the time when earth observation can reap maximum benefits, but optical sensors prevent us doing that. Now a days a lot of organizations are investing in SAR data making it more available to users than before. The only disadvantage of SAR data is the unavailability of labelled data as it is more difficult for users to understand and label SAR data than optical imagery.
In this sample notebook, we will see how we can make use of benefits of SAR and optical imagery to perform all season earth observation. We will train a deep learning model to translate SAR imagery to RGB imagery, thereby making optical data (translated) available even in extreme weather days and cloudy areas.
Necessary imports¶
import os, zipfile
from pathlib import Path
from arcgis.gis import GIS
from arcgis.learn import prepare_data, CycleGAN
Connect to your GIS¶
# Connect to GIS
gis = GIS('home')
Export training data¶
For this usecase, we have SAR imagery from Capella Space and world imagery in the form of RGB tiles near Rotterdam city in the Netherlands. We have exported that data in a new “Export_Tiles” metadata format available in the Export Training Data For Deep Learning
tool. This Export Training Data For Deep Learning
tool available in ArcGIS Pro as well as ArcGIS Image Server.
Input Raster
: SAR imagery tileTile Size X & Tile Size Y
: 256Stride X & Stride Y
: 128Meta Data Format
: 'Export_Tiles' as we are training aCycleGAN
model.Environments
: Set optimumCell Size
,Processing Extent
.
Now, in case of exporting RGB imagery into chips, let us first create a tile cache of world imagery for the area of extent, on which we are working. We can do this using Manage Tile Cache tool available in Geoprocessing tools.
Cache Location
: provide a path to save tile cache.Cache Name
: choose a name to the tile cache.Manage Mode
: Recreate all tiles.Input Data Source
: select the map with basemap set as imagery. Uncheck all the layers on the map except the imagery basemap.Scale
: select the appropriate scales at which you want imagery cache.
The cache once exported, will be automatically added to the ArcGIS Pro project. Use Export training Data for Deep Learning
tool as we did in the case of SAR imagery to export image chips from the cache.
Once, both the imagery are exported in the required format, we need to follow the directory structure shown in figure 3 that CycleGAN
model follows.
Here, 'train_a' and 'train_b' folders contain all the image tiles (excluding other files like 'esri_accumulated_stats.json', 'esri_model_definition.emd', 'map.txt', 'stats.txt') exported from SAR imagery and world imagery map respectively. Now we are ready to train the CycleGAN
model.
Alternatively, we have provided a subset of training data containing a few samples that follows the same directory structure mentioned above. You can use the data directly to run the experiments.
training_data = gis.content.get('25ed4a30219e4ba7acb3633e1a75bae1')
training_data
filepath = training_data.download(file_name=training_data.name)
with zipfile.ZipFile(filepath, 'r') as zip_ref:
zip_ref.extractall(Path(filepath).parent)
output_path = Path(os.path.join(os.path.splitext(filepath)[0]))
Train the model¶
We will train CycleGAN model [1] that performs the task of Image-to-Image translation where it learns mapping between input and output images using unpaired dataset. This model is an extension of GAN architecture which involves simultaneous training of two generator models and two discriminator models. In GAN, we can generate images of domain Y from domain X, but in CycleGAN, we can also generate images of domain X from domain Y using the same model architecture.
It has two mapping functions: G : X → Y and F : Y → X, and associated adversarial discriminators Dy and Dx. G tries to generate images that look similar to images from domain Y, while Dy aims to distinguish between translated samples G(x) and real samples y. G aims to minimize this objective against an adversary D that tries to maximize it. The same process happens in generation of the images of domain X from domain Y using F as a generator and Dx as a discriminator.
Prepare data¶
We will specify the path to our training data and a few hyperparameters.
path
: path of the folder containing training data.dataset_type
: CycleGANbatch_size
: Number of images your model will train on each step inside an epoch, it directly depends on the memory of your graphic card. 4 worked for us on a 11GB GPU.
# output_path = r"D:\CycleGAN\Data\data_for_cyclegan_le_3Bands"
data = prepare_data(output_path, dataset_type="CycleGAN", batch_size=4)
data
Visualize training data¶
To get a sense of what the training data looks like, arcgis.learn.show_batch()
method randomly picks a few training chips and visualizes them.
rows
: Number of rows to visualize
data.show_batch(rows=5)