Landsat 8 to Sentinel-2 using Pix2Pix

Introduction

The Landsat program is the longest running mission for the acquisition of satellite imagery of the earth. The images acquired have opened multiple doors for studies observing the earth. In 2015, Sentinel-2, with 10 m resolution imagery, further enhanced our ability to study and observe the earth. Now, with the advancements in deep learning techniques, there is a need to revisit historic images from the Landsat program and increase their resolution to perform studies at a higher resolution.

In this sample notebook, we will see how we can make use of the Pix2Pix model to convert 30 meter resolution Landsat 8 imagery to 10 meter resolution Sentinel-2 imagery, thus allowing us to use Landsat 8 imagery for processes like precision agriculture.

Necessary imports

import os, zipfile
from pathlib import Path

from arcgis.gis import GIS
from arcgis.learn import prepare_data, Pix2Pix
gis = GIS(profile = "your_online_profile")

Export training data

For this example, we will be using both Landsat 8 and Sentinel-2 imagery. We have exported this data in a “Export_Tiles” metadata format, available in the Export Training Data For Deep Learning tool. This Export Training Data For Deep Learning tool is available in ArcGIS Pro and ArcGIS Image Server.

  • Input Raster: Landsat 8 imagery
  • Additional Raster: Sentinel-2 imagery
  • Tile Size X & Tile Size Y: 256
  • Stride X & Stride Y: 128
  • Meta Data Format: 'Export_Tiles', as we are training a Pix2Pix model.
  • Environments: Set Cell Size as 10 to convert Landsat 8 imagery to Sentinel-2 imagery with a 10m resolution.

Here is the link to the sample data exported by following the steps mentioned above.

output_l1 = r"./Landsat8 to Sentinel2"
if not os.path.exists(output_l1):
    os.mkdir(output_l1)
output_l2 = r"./Landsat8 to Sentinel2/L_to_S_data_export_larger_extent"
if not os.path.exists(output_l2):
    os.mkdir(output_l2)
sample_data = gis.content.get("40b1bae898c441c9832d9dd15a73ab70")
sample_data.download(save_path=output_l2)
'/arcgis/home/Landsat8 to Sentinel2/L_to_S_data_export_larger_extent/Exported_Data_Landsat_to_Sentinel_10.zip'
zf = zipfile.ZipFile(os.path.join(output_l2, 'Exported_Data_Landsat_to_Sentinel_10.zip'))
zf.extractall(path = os.path.join(output_l2, ''))

Train the model

Prepare the data

Here, we will specify the path to our training data and a few hyperparameters.

  • path: path of the folder containing the training data.
  • batch_size: The number of images your model will train on each step inside an epoch. This directly depends on the memory of your graphic card. 64 worked for us on a 32GB GPU.
output_path = r"./Landsat8 to Sentinel2/L_to_S_data_export_larger_extent/Exported_Data_Landsat_to_Sentinel_10"
data = prepare_data(output_path, batch_size=64)

Visualize the training data

To get a sense of what the training data looks like, the arcgis.learn.show_batch() method will randomly select a few training chips and visualize them.

  • rows: The number of rows to visualize
data.show_batch(rows=2)
<Figure size 1440x720 with 4 Axes>

Load the model architecture

model = Pix2Pix(data)

Find an optimal learning rate

Learning rate is one of the most important hyperparameters in model training. The ArcGIS API for Python provides a learning rate finder that will automatically select the optimal learning rate for you.

lr = model.lr_find()
<Figure size 432x288 with 1 Axes>

Fit the model

Next, we will train the model for a few epochs with the learning rate found above.

model.fit(100, lr=lr)
epochtrain_lossvalid_lossgen_lossl1_lossD_losstime
067.922554101.0588230.8066390.6711590.88879404:38
152.82703078.9361950.7889840.5203800.88206304:25
238.65738363.0509450.7803960.3787700.87767004:24
328.06267458.5134960.7795230.2728310.87460504:24
421.15562856.8536640.7748550.2038080.87037604:35
516.30853156.1668400.7652490.1554330.86649804:37
614.42946255.7816390.7534770.1367600.86339804:40
712.36810755.4964560.7408460.1162730.86048104:42
811.97243755.2708630.7274830.1124500.85837304:42
911.42036655.1597480.7159360.1070440.85663904:43
1010.36647655.0614470.7052560.0966120.85537604:44
1110.21039955.0014190.6967260.0951370.85365304:29
129.47575154.8802950.6940760.0878170.85188504:41
139.76370854.6927760.6893420.0907440.85028304:43
149.22602854.7396160.6863190.0853970.84829904:39
159.12413754.5909040.6819290.0844220.84599704:41
169.70065754.6051790.6785980.0902210.84381104:44
179.80335254.4668500.6747270.0912860.84170204:38
189.25024754.4437520.6717120.0857850.83861504:39
198.33596454.4381220.6681050.0766790.83643404:41
207.96855254.4150280.6633300.0730520.83391904:39
217.48005354.3284800.6608250.0681920.83094004:39
226.97778354.4227030.6587390.0631900.82783904:38
238.00251854.2751310.6560070.0734650.82516804:23
248.12046954.3237110.6538570.0746660.82148604:36
258.97137854.2001650.6531200.0831830.81877704:37
268.52423454.2233050.6506780.0787360.81551304:39
277.69357254.1759000.6500670.0704350.81144704:38
287.58577454.1693500.6489580.0693680.80820404:42
297.29456254.1933630.6481750.0664640.80472604:39
307.42363754.1420860.6473780.0677630.80112504:41
317.69110354.1329770.6462670.0704480.79825004:40
328.03200154.1864320.6451210.0738690.79522304:40
337.71693354.0232320.6452450.0707170.79204104:37
347.80797854.0976830.6462860.0716170.78892604:38
357.05491854.0514140.6456210.0640930.78524304:40
367.36442854.0435980.6464390.0671800.78268904:41
377.99230954.1060180.6461570.0734620.78012304:42
387.75632654.0195920.6464350.0710990.77807704:39
397.37725754.0132030.6467470.0673050.77572105:04
407.50161953.9444850.6477650.0685390.77324204:41
417.90817053.9969670.6473100.0726090.77127404:38
427.07662353.9708370.6484040.0642820.76885204:41
436.94464453.9279290.6494750.0629520.76698404:40
446.45181454.0159450.6519090.0579990.76474204:39
456.64843954.0318490.6538230.0599460.76284204:35
466.86248353.8947600.6554730.0620700.76126704:26
476.54075753.8506620.6572630.0588350.75947104:38
486.77233453.8899770.6583140.0611400.75768004:36
496.98965253.8315160.6604640.0632920.75569804:37
507.04495053.8784180.6623660.0638260.75408804:39
516.79907353.8208540.6623510.0613670.75257504:41
526.42871953.8300320.6631300.0576560.75140704:40
536.39883953.7705610.6634660.0573540.75030204:39
546.19631353.7895240.6654390.0553090.74868804:39
557.44275353.3826520.6653680.0677740.74750404:37
567.59524853.3540310.6653220.0692990.74625804:39
576.78431353.2428930.6658200.0611850.74462204:38
586.32776653.2298620.6673710.0566040.74334004:39
597.04592953.2231100.6695500.0637640.74171904:44
606.68728953.2093890.6699570.0601730.74058604:40
616.50063353.2321240.6710550.0582960.73956804:40
626.58333253.2851100.6714050.0591190.73842704:41
636.97229353.2518540.6715640.0630070.73732204:41
646.80759753.1929210.6720220.0613560.73582504:39
656.94970153.2340320.6719130.0627780.73492504:42
666.99217053.3533210.6718700.0632030.73401704:40
676.41720953.1875920.6713730.0574580.73335104:39
686.81502353.1609190.6708660.0614420.73257704:46
697.49540953.1325490.6723040.0682310.73176604:32
706.96463053.1396450.6727810.0629190.73072204:33
717.15996653.1263620.6740600.0648590.73015904:34
727.18124553.1683390.6752540.0650600.72945504:37
736.73363453.1726530.6758140.0605780.72877404:36
747.28991153.1513630.6760470.0661390.72812004:37
757.49831253.0917470.6765420.0682180.72764904:37
767.42754353.0792160.6766910.0675090.72715604:38
776.53676853.1068080.6775480.0585920.72640704:38
786.33742753.0739400.6785340.0565890.72582204:40
796.97796753.0952910.6784630.0629950.72542704:40
807.06523853.0604970.6786380.0638660.72482904:42
817.16399053.0706330.6791820.0648480.72470104:43
827.80314653.0935170.6799200.0712320.72451804:47
836.78148753.0542370.6797950.0610170.72399204:52
846.67401253.0540310.6804610.0599360.72387304:49
855.71309553.0542950.6809500.0503210.72374104:53
865.72389453.0484430.6810450.0504280.72339504:41
875.97815453.0272940.6811040.0529700.72312204:35
885.87611953.0328940.6810880.0519500.72276604:38
896.55366253.0387150.6810610.0587260.72294004:35
907.40732753.0241390.6817870.0672550.72305204:39
917.02169253.0201110.6823310.0633940.72283004:38
926.47968253.0175320.6829170.0579680.72267704:37
935.85384653.0178260.6822700.0517160.72215404:27
946.39323053.0151020.6821490.0571110.72244604:34
957.15264853.0183140.6822150.0647040.72255304:39
966.64611253.0199510.6820400.0596410.72246504:35
976.61026953.0193670.6822150.0592810.72238904:37
986.44965953.0175130.6824480.0576720.72229504:36
996.10753053.0171390.6824200.0542510.72216504:35

Here, with 100 epochs, we can see reasonable results, as both the training and validation losses have gone down considerably. This indicates that the model is learning to translate Landsat 8 imagery to Sentinel-2.

Visualize the results in validation set

It is a good practice to see results of the model viz-a-viz the ground truth. The code below picks random samples and displays the ground truth and model predictions side by side. This enables us to preview the results of the model within the notebook.

model.show_results(2)
<Figure size 1440x720 with 6 Axes>

Generative Adversarial Network (GAN) models require a significant amount time to train, and even with the initial 100 epochs trained so far, there is still room for more training. As such, we trained this model for an additional 400 epochs (500 in total), to achieve good results. With the additional training, the training and validation losses dropped to 5.45 and 52.6 respectively and the D_loss stabilized at 0.54.

Below is the loss curve, which represents the training and validation losses during the training process.

model.plot_losses()
<Figure size 432x288 with 1 Axes>

Save the model

Next, we will save the trained model as a 'Deep Learning Package' ('.dlpk' format). Deep Learning package is the standard format used to deploy deep learning models on the ArcGIS platform.

We will use the save() method to save the trained model. By default, it will be saved to the 'models' sub-folder within our training data folder.

model.save("Landsat_to_Sentinel_LE_500e")
Computing model metrics...
WindowsPath('D:/Landsat8 to Sentinel2/L_to_S_data_export_larger_extent/models/Landsat_to_Sentinel_LE_500e')

Model inference

In this step, we will generate a classified raster using the 'Classify Pixels Using Deep Learning' tool available in both ArcGIS Pro and ArcGIS Enterprise.

  • Input Raster: The raster layer you want to classify.
  • Model Definition: Located inside the saved model in the 'models' folder in '.emd' format.
  • Padding: The 'Input Raster' is tiled, and the deep learning model classifies each individual tile separately before producing the final 'Output Classified Raster'. This may lead to unwanted artifacts along the edges of each tile, as the model has little context to predict accurately. Padding allows us to supply extra information along the tile edges, thus helping the model to make better predictions.
  • Cell Size: Should be close to the size used to train the model.
  • Processor Type: Allows you to control whether the system's 'GPU' or 'CPU' will be used to classify pixels. By default, 'GPU' will be used if available.

It is advised to zoom in to the right extent of the area of interest in order to avoid/reduce noise from the results as the model is not trained to be generalized to work across the globe.

Results

The gif below was achieved with the model trained in this notebook. The model converted a Landsat 8 image with a 30m resolution to an image with a 10m resolution and 15 bands, similar to Sentinel-2 images.

Below is the same generated image viewed through the agriculture band combination of SWIR-1 (B11), near-infrared (B8), and blue (B2). This band combination can be used to monitor the health of crops.

Here, the dark green patches highlight dense vegetation and healthy crops.

Conclusion

In this notebook, we have demonstrated how to use the Pix2Pix model, available in the ArcGIS API for Python, to translate Landsat 8 imagery to Sentinel-2 imagery.

References

[1] How Pix2Pix works ?, https://developers.arcgis.com/python/guide/how-pix2pix-works/.

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.