SAR to RGB image translation using CycleGAN¶

Table of Contents¶

• Introduction
• Necessary imports
• Connect to your GIS
• Export training data
• Train the model
  • Prepare data
  • Visualize training data
  • Load model architecture
  • Find an optimal learning rate
  • Fit the model
  • Visualize results in validation set
  • Save the model
• Model inference
• Conclusion
• References

Introduction¶

The ability of SAR data to let us see through clouds make it more valuable specially in cloudy areas and bad weather. This is the time when earth observation can reap maximum benefits, but optical sensors prevent us doing that. Now a days a lot of organizations are investing in SAR data making it more available to users than before. The only disadvantage of SAR data is the unavailability of labelled data as it is more difficult for users to understand and label SAR data than optical imagery.

In this sample notebook, we will see how we can make use of benefits of SAR and optical imagery to perform all season earth observation. We will train a deep learning model to translate SAR imagery to RGB imagery, thereby making optical data (translated) available even in extreme weather days and cloudy areas.

Necessary imports¶

Connect to your GIS¶

Export training data¶

For this usecase, we have SAR imagery from Capella Space and world imagery in the form of RGB tiles near Rotterdam city in the Netherlands. We have exported that data in a new “Export_Tiles” metadata format available in the Export Training Data For Deep Learning tool. This Export Training Data For Deep Learning tool available in ArcGIS Pro as well as ArcGIS Image Server.

• Input Raster: SAR imagery tile
• Tile Size X & Tile Size Y: 256
• Stride X & Stride Y: 128
• Meta Data Format: 'Export_Tiles' as we are training a CycleGAN model.
• Environments: Set optimum Cell Size, Processing Extent.

Now, in case of exporting RGB imagery into chips, let us first create a tile cache of world imagery for the area of extent, on which we are working. We can do this using Manage Tile Cache tool available in Geoprocessing tools.

• Cache Location: provide a path to save tile cache.
• Cache Name: choose a name to the tile cache.
• Manage Mode: Recreate all tiles.
• Input Data Source: select the map with basemap set as imagery. Uncheck all the layers on the map except the imagery basemap.
• Scale: select the appropriate scales at which you want imagery cache.

The cache once exported, will be automatically added to the ArcGIS Pro project. Use Export training Data for Deep Learning tool as we did in the case of SAR imagery to export image chips from the cache.

Once, both the imagery are exported in the required format, we need to follow the directory structure shown in figure 3 that CycleGAN model follows.

Here, 'train_a' and 'train_b' folders contain all the image tiles (excluding other files like 'esri_accumulated_stats.json', 'esri_model_definition.emd', 'map.txt', 'stats.txt') exported from SAR imagery and world imagery map respectively. Now we are ready to train the CycleGAN model.

Alternatively, we have provided a subset of training data containing a few samples that follows the same directory structure mentioned above. You can use the data directly to run the experiments.

Train the model¶

We will train CycleGAN model [1] that performs the task of Image-to-Image translation where it learns mapping between input and output images using unpaired dataset. This model is an extension of GAN architecture which involves simultaneous training of two generator models and two discriminator models. In GAN, we can generate images of domain Y from domain X, but in CycleGAN, we can also generate images of domain X from domain Y using the same model architecture.

It has two mapping functions: G : X → Y and F : Y → X, and associated adversarial discriminators Dy and Dx. G tries to generate images that look similar to images from domain Y, while Dy aims to distinguish between translated samples G(x) and real samples y. G aims to minimize this objective against an adversary D that tries to maximize it. The same process happens in generation of the images of domain X from domain Y using F as a generator and Dx as a discriminator.

Prepare data¶

We will specify the path to our training data and a few hyperparameters.

• path: path of the folder containing training data.
• dataset_type: CycleGAN
• batch_size: Number of images your model will train on each step inside an epoch, it directly depends on the memory of your graphic card. 4 worked for us on a 11GB GPU.

Visualize training data¶

To get a sense of what the training data looks like, arcgis.learn.show_batch() method randomly picks a few training chips and visualizes them.

• rows: Number of rows to visualize