Landsat 8 to Sentinel-2 using Pix2Pix

Introduction

The Landsat program is the longest running mission for the acquisition of satellite imagery of the earth. The images acquired have opened multiple doors for studies observing the earth. In 2015, Sentinel-2, with 10 m resolution imagery, further enhanced our ability to study and observe the earth. Now, with the advancements in deep learning techniques, there is a need to revisit historic images from the Landsat program and increase their resolution to perform studies at a higher resolution.

In this sample notebook, we will see how we can make use of the Pix2Pix model to convert 30 meter resolution Landsat 8 imagery to 10 meter resolution Sentinel-2 imagery, thus allowing us to use Landsat 8 imagery for processes like precision agriculture.

Necessary imports

Input
import os, zipfile
from pathlib import Path

from arcgis.gis import GIS
from arcgis.learn import prepare_data, Pix2Pix

Export training data

For this example, we will be using both Landsat 8 and Sentinel-2 imagery. We have exported this data in a “Export_Tiles” metadata format, available in the Export Training Data For Deep Learning tool. This Export Training Data For Deep Learning tool is available in ArcGIS Pro and ArcGIS Image Server.

  • Input Raster: Landsat 8 imagery
  • Additional Raster: Sentinel-2 imagery
  • Tile Size X & Tile Size Y: 256
  • Stride X & Stride Y: 128
  • Meta Data Format: 'Export_Tiles', as we are training a Pix2Pix model.
  • Environments: Set Cell Size as 10 to convert Landsat 8 imagery to Sentinel-2 imagery with a 10m resolution.

Here is the link to the sample data exported by following the steps mentioned above.

Train the model

Prepare the data

Here, we will specify the path to our training data and a few hyperparameters.

  • path: path of the folder containing the training data.
  • batch_size: The number of images your model will train on each step inside an epoch. This directly depends on the memory of your graphic card. 64 worked for us on a 32GB GPU.
Input
output_path = r"D:\Landsat8 to Sentinel2\L_to_S_data_export_larger_extent"
data = prepare_data(output_path, batch_size=64)

Visualize the training data

To get a sense of what the training data looks like, the arcgis.learn.show_batch() method will randomly select a few training chips and visualize them.

  • rows: The number of rows to visualize
Input
data.show_batch(rows=2)

Load the model architecture

Input
model = Pix2Pix(data)

Find an optimal learning rate

Learning rate is one of the most important hyperparameters in model training. The ArcGIS API for Python provides a learning rate finder that will automatically select the optimal learning rate for you.

Input
lr = model.lr_find()

Fit the model

Next, we will train the model for a few epochs with the learning rate found above.

Input
model.fit(100, lr=lr)
epoch train_loss valid_loss gen_loss l1_loss D_loss time
0 67.922554 101.058823 0.806639 0.671159 0.888794 04:38
1 52.827030 78.936195 0.788984 0.520380 0.882063 04:25
2 38.657383 63.050945 0.780396 0.378770 0.877670 04:24
3 28.062674 58.513496 0.779523 0.272831 0.874605 04:24
4 21.155628 56.853664 0.774855 0.203808 0.870376 04:35
5 16.308531 56.166840 0.765249 0.155433 0.866498 04:37
6 14.429462 55.781639 0.753477 0.136760 0.863398 04:40
7 12.368107 55.496456 0.740846 0.116273 0.860481 04:42
8 11.972437 55.270863 0.727483 0.112450 0.858373 04:42
9 11.420366 55.159748 0.715936 0.107044 0.856639 04:43
10 10.366476 55.061447 0.705256 0.096612 0.855376 04:44
11 10.210399 55.001419 0.696726 0.095137 0.853653 04:29
12 9.475751 54.880295 0.694076 0.087817 0.851885 04:41
13 9.763708 54.692776 0.689342 0.090744 0.850283 04:43
14 9.226028 54.739616 0.686319 0.085397 0.848299 04:39
15 9.124137 54.590904 0.681929 0.084422 0.845997 04:41
16 9.700657 54.605179 0.678598 0.090221 0.843811 04:44
17 9.803352 54.466850 0.674727 0.091286 0.841702 04:38
18 9.250247 54.443752 0.671712 0.085785 0.838615 04:39
19 8.335964 54.438122 0.668105 0.076679 0.836434 04:41
20 7.968552 54.415028 0.663330 0.073052 0.833919 04:39
21 7.480053 54.328480 0.660825 0.068192 0.830940 04:39
22 6.977783 54.422703 0.658739 0.063190 0.827839 04:38
23 8.002518 54.275131 0.656007 0.073465 0.825168 04:23
24 8.120469 54.323711 0.653857 0.074666 0.821486 04:36
25 8.971378 54.200165 0.653120 0.083183 0.818777 04:37
26 8.524234 54.223305 0.650678 0.078736 0.815513 04:39
27 7.693572 54.175900 0.650067 0.070435 0.811447 04:38
28 7.585774 54.169350 0.648958 0.069368 0.808204 04:42
29 7.294562 54.193363 0.648175 0.066464 0.804726 04:39
30 7.423637 54.142086 0.647378 0.067763 0.801125 04:41
31 7.691103 54.132977 0.646267 0.070448 0.798250 04:40
32 8.032001 54.186432 0.645121 0.073869 0.795223 04:40
33 7.716933 54.023232 0.645245 0.070717 0.792041 04:37
34 7.807978 54.097683 0.646286 0.071617 0.788926 04:38
35 7.054918 54.051414 0.645621 0.064093 0.785243 04:40
36 7.364428 54.043598 0.646439 0.067180 0.782689 04:41
37 7.992309 54.106018 0.646157 0.073462 0.780123 04:42
38 7.756326 54.019592 0.646435 0.071099 0.778077 04:39
39 7.377257 54.013203 0.646747 0.067305 0.775721 05:04
40 7.501619 53.944485 0.647765 0.068539 0.773242 04:41
41 7.908170 53.996967 0.647310 0.072609 0.771274 04:38
42 7.076623 53.970837 0.648404 0.064282 0.768852 04:41
43 6.944644 53.927929 0.649475 0.062952 0.766984 04:40
44 6.451814 54.015945 0.651909 0.057999 0.764742 04:39
45 6.648439 54.031849 0.653823 0.059946 0.762842 04:35
46 6.862483 53.894760 0.655473 0.062070 0.761267 04:26
47 6.540757 53.850662 0.657263 0.058835 0.759471 04:38
48 6.772334 53.889977 0.658314 0.061140 0.757680 04:36
49 6.989652 53.831516 0.660464 0.063292 0.755698 04:37
50 7.044950 53.878418 0.662366 0.063826 0.754088 04:39
51 6.799073 53.820854 0.662351 0.061367 0.752575 04:41
52 6.428719 53.830032 0.663130 0.057656 0.751407 04:40
53 6.398839 53.770561 0.663466 0.057354 0.750302 04:39
54 6.196313 53.789524 0.665439 0.055309 0.748688 04:39
55 7.442753 53.382652 0.665368 0.067774 0.747504 04:37
56 7.595248 53.354031 0.665322 0.069299 0.746258 04:39
57 6.784313 53.242893 0.665820 0.061185 0.744622 04:38
58 6.327766 53.229862 0.667371 0.056604 0.743340 04:39
59 7.045929 53.223110 0.669550 0.063764 0.741719 04:44
60 6.687289 53.209389 0.669957 0.060173 0.740586 04:40
61 6.500633 53.232124 0.671055 0.058296 0.739568 04:40
62 6.583332 53.285110 0.671405 0.059119 0.738427 04:41
63 6.972293 53.251854 0.671564 0.063007 0.737322 04:41
64 6.807597 53.192921 0.672022 0.061356 0.735825 04:39
65 6.949701 53.234032 0.671913 0.062778 0.734925 04:42
66 6.992170 53.353321 0.671870 0.063203 0.734017 04:40
67 6.417209 53.187592 0.671373 0.057458 0.733351 04:39
68 6.815023 53.160919 0.670866 0.061442 0.732577 04:46
69 7.495409 53.132549 0.672304 0.068231 0.731766 04:32
70 6.964630 53.139645 0.672781 0.062919 0.730722 04:33
71 7.159966 53.126362 0.674060 0.064859 0.730159 04:34
72 7.181245 53.168339 0.675254 0.065060 0.729455 04:37
73 6.733634 53.172653 0.675814 0.060578 0.728774 04:36
74 7.289911 53.151363 0.676047 0.066139 0.728120 04:37
75 7.498312 53.091747 0.676542 0.068218 0.727649 04:37
76 7.427543 53.079216 0.676691 0.067509 0.727156 04:38
77 6.536768 53.106808 0.677548 0.058592 0.726407 04:38
78 6.337427 53.073940 0.678534 0.056589 0.725822 04:40
79 6.977967 53.095291 0.678463 0.062995 0.725427 04:40
80 7.065238 53.060497 0.678638 0.063866 0.724829 04:42
81 7.163990 53.070633 0.679182 0.064848 0.724701 04:43
82 7.803146 53.093517 0.679920 0.071232 0.724518 04:47
83 6.781487 53.054237 0.679795 0.061017 0.723992 04:52
84 6.674012 53.054031 0.680461 0.059936 0.723873 04:49
85 5.713095 53.054295 0.680950 0.050321 0.723741 04:53
86 5.723894 53.048443 0.681045 0.050428 0.723395 04:41
87 5.978154 53.027294 0.681104 0.052970 0.723122 04:35
88 5.876119 53.032894 0.681088 0.051950 0.722766 04:38
89 6.553662 53.038715 0.681061 0.058726 0.722940 04:35
90 7.407327 53.024139 0.681787 0.067255 0.723052 04:39
91 7.021692 53.020111 0.682331 0.063394 0.722830 04:38
92 6.479682 53.017532 0.682917 0.057968 0.722677 04:37
93 5.853846 53.017826 0.682270 0.051716 0.722154 04:27
94 6.393230 53.015102 0.682149 0.057111 0.722446 04:34
95 7.152648 53.018314 0.682215 0.064704 0.722553 04:39
96 6.646112 53.019951 0.682040 0.059641 0.722465 04:35
97 6.610269 53.019367 0.682215 0.059281 0.722389 04:37
98 6.449659 53.017513 0.682448 0.057672 0.722295 04:36
99 6.107530 53.017139 0.682420 0.054251 0.722165 04:35

Here, with 100 epochs, we can see reasonable results, as both the training and validation losses have gone down considerably. This indicates that the model is learning to translate Landsat 8 imagery to Sentinel-2.

Visualize the results in validation set

It is a good practice to see results of the model viz-a-viz the ground truth. The code below picks random samples and displays the ground truth and model predictions side by side. This enables us to preview the results of the model within the notebook.

Input
model.show_results(2)

Generative Adversarial Network (GAN) models require a significant amount time to train, and even with the initial 100 epochs trained so far, there is still room for more training. As such, we trained this model for an additional 400 epochs (500 in total), to achieve good results. With the additional training, the training and validation losses dropped to 5.45 and 52.6 respectively and the D_loss stabilized at 0.54.

Below is the loss curve, which represents the training and validation losses during the training process.

Input
model.plot_losses()

Save the model

Next, we will save the trained model as a 'Deep Learning Package' ('.dlpk' format). Deep Learning package is the standard format used to deploy deep learning models on the ArcGIS platform.

We will use the save() method to save the trained model. By default, it will be saved to the 'models' sub-folder within our training data folder.

Input
model.save("Landsat_to_Sentinel_LE_500e")
Computing model metrics...
Output
WindowsPath('D:/Landsat8 to Sentinel2/L_to_S_data_export_larger_extent/models/Landsat_to_Sentinel_LE_500e')

Model inference

In this step, we will generate a classified raster using the 'Classify Pixels Using Deep Learning' tool available in both ArcGIS Pro and ArcGIS Enterprise.

  • Input Raster: The raster layer you want to classify.
  • Model Definition: Located inside the saved model in the 'models' folder in '.emd' format.
  • Padding: The 'Input Raster' is tiled, and the deep learning model classifies each individual tile separately before producing the final 'Output Classified Raster'. This may lead to unwanted artifacts along the edges of each tile, as the model has little context to predict accurately. Padding allows us to supply extra information along the tile edges, thus helping the model to make better predictions.
  • Cell Size: Should be close to the size used to train the model.
  • Processor Type: Allows you to control whether the system's 'GPU' or 'CPU' will be used to classify pixels. By default, 'GPU' will be used if available.

It is advised to zoom in to the right extent of the area of interest in order to avoid/reduce noise from the results as the model is not trained to be generalized to work across the globe.

Results

The gif below was achieved with the model trained in this notebook. The model converted a Landsat 8 image with a 30m resolution to an image with a 10m resolution and 15 bands, similar to Sentinel-2 images.

Below is the same generated image viewed through the agriculture band combination of SWIR-1 (B11), near-infrared (B8), and blue (B2). This band combination can be used to monitor the health of crops.

Here, the dark green patches highlight dense vegetation and healthy crops.

Conclusion

In this notebook, we have demonstrated how to use the Pix2Pix model, available in the ArcGIS API for Python, to translate Landsat 8 imagery to Sentinel-2 imagery.

References

Your browser is no longer supported. Please upgrade your browser for the best experience. See our browser deprecation post for more details.