Increase Image Resolution using SuperResolution

Prerequisites

  • Please refer to the prerequisites section in our guide for more information. This sample demonstrates how to export training data and model inference using ArcGIS Pro. Alternatively, they can be done using ArcGIS Image Server as well.
  • If you have already exported training samples using ArcGIS Pro, you can jump straight to the training section. The saved model can also be imported into ArcGIS Pro directly.

Introduction

High resolution imagery is desirable for both visualization and image interpretation. However, high resolution imagery is expensive to procure. This sample notebook demonstrates how the SuperResolution model in arcgis.learn module can be used to increase image resolution. This model uses deep learning to add texture and detail to low resolution satellite imagery and turn it into higher resolution imagery.

We first start with high resolution aerial imagery to train the model. The data preparation step first downsamples the higher resolution imagery to create lower resolution, blurred imagery. The SuperResolution model uses this training data and learns how to upsample the lower resolution imagery and produce realistic high resolution images that closely resemble the higher quality images that we started with. We then use the trained SuperResolution model to produce simulated high resolution aerial imagery from relatively lower resolution satellite imagery.

Export Training Data

We will be using ArcGIS Pro to find the area where high resolution imagery. To simplify our job, we have already created polygon representing the extent of high resolution imagery. We can add the polygon from here.

Training data can be exported by using the Export Training Data For Deep Learning tool available in ArcGIS Pro as well as ArcGIS Image Server.

  • Input Raster: Imagery
  • Image Format: JPEG Format
  • Tile Size X & Tile Size Y: 512
  • Meta Data Format: Export Tiles
  • In 'Environments' tab set an optimum 'Cell Size' (0.1 in our case).
  • Set the extent same as the polygon layer which we have added.
arcpy.ia.ExportTrainingDataForDeepLearning("Imagery", r"C:\sample\Data\Hi_res_superres_data", "JPEG", 512, 512, 0, 0, "Export Tiles", 0, "ecode", 75, None, 0)

After filling all details and running the Export Training Data For Deep Learning tool, a code like above will be generated and executed. That will create all the necessary files needed for the next step in the 'Output Folder', and we will now call it our training data.

Model Training

We will train our model using arcgis.learn module within ArcGIS API for Python. arcgis.learn contains tools and deep learning capabilities required for this study. A detailed documentation to install and setup the environment is available here.

Necessary Imports

Input
import os
from pathlib import Path

from arcgis.gis import GIS
from arcgis.learn import SuperResolution, prepare_data

We will now use the prepare_data() function to apply various types of transformations and augmentations on the training data. These augmentations enable us to train a better model with limited data and also prevent the model from overfitting.
prepare_data(): Takes 4 parameters:

  • path: Path of folder containing training data.
  • batch_size: No. of images your model will train on each step inside an epoch, it directly depends on the memory of your graphic card, size of the images you are training on and the type of model which you are working with. For this sample a batch size of 8 worked for us on a GPU with 8GB memory.
  • dataset_type: To infer the supported dataset type, in our case 'superres'.
  • downsample_factor: Factor to degrade the quality of image by resizing and adding compression artifacts in order to create labels.
    Note:The quality of degraded image should be similar to the image on which we are going to do inferencing.

This function returns a data object which can be fed into a model for training.

Input
gis = GIS('home')
Input
training_data = gis.content.get('abc0812aa82c4fe681662e5ba495b6b8')
training_data
Output
increase-image-resolution-using-superresolution
Image Collection by api_data_owner
Last Modified: August 28, 2020
0 comments, 2 views
Input
filepath = training_data.download(file_name=training_data.name)
Input
import zipfile
with zipfile.ZipFile(filepath, 'r') as zip_ref:
    zip_ref.extractall(Path(filepath).parent)
Input
data_path = Path(os.path.join(os.path.splitext(filepath)[0]))
Input
data = prepare_data(data_path, 
                    batch_size=8, 
                    dataset_type="superres", 
                    downsample_factor=8)

Visualize training data

To make sense of training data we will use the show_batch() method in arcgis.learn. show_batch() randomly picks few samples from the training data and visualizes them.

Input
data.show_batch()

The imagery chips above show images which we have been downsampled in prepare_data and corresponding high resolution images with them. data.show_batch() shows a batch of images from our training data. We can visualize the the low resolution training data generated using prepare_data function on left along with the original data on the right. You can degrade the image quality more by increasing downsample_factor in prepare_data.

arcgis.learn provides the SuperResolution model for increasing image resolution, which is based on a pretrained convnet, like ResNet that acts as the 'backbone'.

Input
superres_model = SuperResolution(data)

We will use the lr_find() method to find an optimum learning rate. It is important to set a learning rate at which we can train a model with good accuracy and speed.

Input
superres_model.lr_find()
Output
0.0001584893192461114

Train a model

We will now train the SuperResolution model using the suggested learning rate from the previous step. We can specify how many epochs we want to train for. Let's train the model for 10 epochs.

Input
superres_model.fit(10, lr=0.0001584893192461114)
epoch train_loss valid_loss pixel time
0 1.764578 1.667279 0.262905 12:04
1 1.460097 1.422421 0.224943 11:17
2 1.359061 1.346903 0.212538 11:17
3 1.313709 1.309428 0.208903 11:16
4 1.283329 1.291093 0.208715 11:16
5 1.280456 1.274125 0.205538 11:17
6 1.245067 1.257469 0.202738 11:17
7 1.239518 1.248023 0.202647 11:17
8 1.230550 1.243982 0.203220 11:17
9 1.232157 1.245707 0.204075 11:17

After the training is complete, we can view the plot with training and validation losses.

Input
superres_model.learn.recorder.plot_losses()

Visualize results on validation set

This method displays the chips from the validation dataset with downsampled chips (left), predicted chips (middle) and ground truth (right). This visual analysis helps in assessing the qualitative results of the trained model.

Input
superres_model.show_results()

Save the model

Let's save the model by giving it a name and calling the save() method, so that we can load it later whenever required. By default, the model gets saved in training/data/models directory, however a custom path can optionally be provided to save model at desired location.

Input
superres_model.save("superres_model")

Deploy the Model

Prediction and Upscaling

We can upscale and improve the details of a single image chip with the help of predict() method.

Using predict function, we can apply the trained model on image to increase its resolution while also upscaling. It can be done by additionally providing height and width parameters.

Input
superres_model.predict(img_path=r"arcgis/home/data/super-res-test-img.jpeg", height=256, width=256)
Output

In the above step, we are upscaling an image with dimensions 32x32 to 256x256.

The model is not only generating a high resolution image, but is also removing the compression artifacts from a low resolution image chip.

Note: The model is not reconstructing your photo exactly as it would have been in upscaled version. It is hallucinating the perceptual details based on its training from example images.

Model inference on ArcGIS Pro

We will use saved model to improve the image quality using Classify Pixels Using Deep Learning tool available in both ArcGIS Pro and ArcGIS Image Server. For this sample we will deploy the model in Redlands,CA region where low resolution imagery is available.

  • Input Raster: Imagery
  • Output Classified Raster: Superres_Inferencing_arcgis
  • Model Definition: model.emd
  • padding: The 'Input Raster' is tiled and the deep learning model runs on each individual tile separately before producing the final result. This may lead to unwanted artifacts along the edges of each tile as the model has little context to detect objects accurately. Padding as the name suggests allows us to supply some extra information along the tile edges, this helps the model to predict better.
  • Cell size: The model is very susceptible to the cell size. The cell size should be same as high resolution imagery we used to train the model which is 0.1 in our case.

Classify Pixels Using Deep Learning returns a raster layer. We have published the output from this sample here as a hosted raster layer.

Conclusion

In this notebook, we demonstrated how to use SuperResolution model using ArcGIS API for Python in order to obtain high-resolution image from a low-resolution satellite imagery.

References

[1] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for realtime style transfer and super-resolution”, 2016; arXiv:1603.08155.
[2] Fast.ai lesson 7.

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.