Geospatial Deep Learning with Python and PyTorch

Preface

This web text can be viewed in a light mode or a dark mode. A toggle is provided in the sidebar.

License

Creative Commons BY-NC-SA 4.0 license badge

Copyright © 2026 Aaron Maxwell.

Unless otherwise noted, Geospatial Deep Learning with Python and PyTorch by Aaron Maxwell is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). You are free to share and adapt this material for non-commercial purposes, provided that you give appropriate credit to the original source, provide a link to the license, indicate if changes were made, and distribute any adapted material under the same license.

Goals of Text

Geospatial Deep Learning with Python and PyTorch is an introduction to the Python language and PyTorch library for deep learning with a specific focus on geospatial scene classification and semantic segmentation (i.e., pixel-level labeling) using convolutional neural network (CNN) architectures. We do not cover the basics of the Python language in this text; instead, that topic is covered in a companion text titled Open Python for Geospatial Professionals. Readers may also be interested in the text GeoAI with Python by Qiusheng Wu.

This text does not provide a full conceptualization of key deep learning concepts. Instead, our focus is on implementing these methods, and applying them to spatial data, using PyTorch. For a more conceptual text, we suggest Dive into Deep Learning by Zhang et al., which is freely available. We also have a conceptual text available through Elsevier titled Supervised Learning in Remote Sensing and Geospatial Science. If you are interested in exploring geospatial deep learning in the R language, we recommend our text Geospatial Supervised Learning using R, which is freely available. Specifically, the third part of that text is focused on deep learning.

Use of AI Tools

This text was developed with substantial assistance from Claude, an artificial intelligence (AI) model developed by Anthropic. Claude was used in 2026 through Claude Code, a tool that allows the model to read and edit files and to run code on the author’s computer under the author’s direction. Since Claude contributed to the text, code, data, and trained models, we describe its role in detail here.

Claude was used to:

• Convert and reorganize material: convert the author’s original lessons and exercises into the book format and assist with restructuring and reorganizing the text and its exercises.
• Draft text and code: draft initial versions of several chapters and sections, including the chapters on simplifying the training loop with PyTorch Lightning, UNet modules, creating image chips, instance segmentation with UNet-ID and Mask R-CNN, ConvNeXt, fully convolutional masked autoencoders (FCMAEs), and foundation models (SAM 2); new sections on regression losses, data augmentation, activation functions, label smoothing, MobileNetV2, and SegFormer; and portions of this Preface.
• Write questions and exercises: draft the review questions at the end of each chapter and several of the exercises.
• Prepare data, run code, and train models: write and execute code to prepare some of the datasets provided with the text (e.g., vfillInst), execute the chapter code to test it, and train some of the models provided with the text.
• Review and maintain the text: review the chapters for typographical, grammatical, and coding errors; update the code for current package versions; and organize the data, file paths, and downloads used by the text.

The author designed the scope and structure of the text; selected the topics, datasets, and methods; directed Claude’s work; and reviewed, tested, revised, and approved all AI-generated content, including the code, results, and trained models. The author takes full responsibility for the content of this text, including any errors. Claude is not an author of this work.

Error Statement

We have tried our best to provide accurate and complete information. Any errors are unintentional. The code has been tested using the package versions listed in the provided environment.yml file. Python packages, and PyTorch specifically, change over time, so you may encounter warnings or errors if you use different package versions than those used to build this text. If you would like to comment on any material in this text, please feel free to contact the author, Aaron Maxwell, at Aaron.Maxwell@mail.wvu.edu.

Audience

This text was written for several key audiences.

1. Geospatial professionals interested in learning to implement deep learning for mapping and modeling tasks
2. Researchers needing to apply geospatial deep learning to problems that interest them
3. General data scientists and analysts needing to learn more about applying deep learning to geospatial data

The material covered in this text may be appropriate for an advanced undergraduate course or a graduate-level course. Since the focus is code implementation and practical considerations, a more theoretical companion text may be necessary. If you have no prior experience with Python, we suggest working through our Open Python for Geospatial Professionals text, or a comparable text or resource, prior to attempting this text.

Structure of Text

This text consists of 28 chapters organized into four parts, Introduction to Deep Learning with PyTorch, CNNs for Scene Labeling, CNNs for Geospatial Semantic Segmentation, and Additional Topics, followed by a final chapter that describes two final projects. Each chapter consists of examples with associated code and explanations. At the end of each chapter, we provide review questions. Several chapters also include an exercise, as described below. The data needed to execute the examples and complete the exercises have been provided or linked, as described in the Data and Source Files section.

The first part, Introduction to Deep Learning with PyTorch, begins with a conceptualization and implementation of fully connected artificial neural networks (ANNs) (1  Introduction to PyTorch: Fully Connected Neural Networks), while 2  Working with Tensors explores the tensor data model. 3  Linear Regression and Training Loops implements linear regression using PyTorch and an ANN architecture and compares the result to ordinary least squares regression. 4  Losses and Assessment Metrics explores loss and assessment metrics for regression and classification, including custom losses, while 5  DataSets and DataLoaders explores building and using DataSet and DataLoader subclasses and applying data augmentations using Albumentations. 6  Train a Fully Connected Neural Network steps through the process of training an ANN using a training loop built from scratch, and 7  Improving Models demonstrates some means to potentially improve model performance, including alternative activation functions, class weighting, label smoothing, alternative losses, learning rate finders and schedulers, gradient accumulation, and reproducibility. 8  Simplifying the Training Loop introduces PyTorch Lightning, which simplifies the training process and is used throughout the remainder of the text, and demonstrates training loop features such as mixed precision, multi-GPU training, checkpointing, early stopping, experiment tracking, and weight averaging.

The second part, CNNs for Scene Labeling, focuses on CNN architectures for scene labeling tasks (i.e., predicting a label for an entire image as opposed to each individual pixel). 9  Convolutional Neural Network Components explores the components of a CNN, 10  Build a Convolutional Neural Network steps through building a CNN architecture, and 11  Train a Convolutional Neural Network demonstrates training a CNN from scratch. We explore common CNN architectures, including VGGNet, ResNet, and the efficient MobileNetV2 architecture, in 12  Convolutional Neural Network Architectures and transfer learning, including the use of different learning rates for different components of a model, in 13  Transfer Learning and CNNs. The part concludes with a midterm project (14  Midterm Project) in which you design and report on a scene classification experiment.

CNNs for Geospatial Semantic Segmentation, the third part of the text, focuses on pixel-level labeling and explores the UNet architecture (15  UNet for Semantic Segmentation), training this architecture (16  Train a UNet for Semantic Segmentation), and incorporating pre-trained scene labeling encoders (17  UNet Encoders). 18  UNet Modules builds a customizable UNet that can incorporate residual connections, squeeze-and-excitation modules, attention gates, convolutional block attention modules, atrous spatial pyramid pooling, alternative activation functions, and deep supervision. 19  Spatial Predictions demonstrates using a trained model to make a spatial prediction to an entire image extent, and 20  Creating Image Chips demonstrates generating static and dynamic image chips and masks from geospatial data using our terrainseg package. Lastly, we explore the Segmentation Models library, which implements a variety of semantic segmentation architectures and encoder backbones, in 21  Segmentation Models Package.

The last part, Additional Topics, begins with instance segmentation, in which individual objects are differentiated. 22  Instance Segmentation with UNet-ID builds the segmentation-based UNet-ID architecture from scratch, and 23  Instance Segmentation with Mask R-CNN explores the detection-based Mask R-CNN architecture. 24  ConvNeXt explores the ConvNeXt architecture and its integration into UNet, while 25  SegFormer discusses transformers and the SegFormer architecture. 26  Anomaly Detection with VAEs covers building and training a variational autoencoder (VAE) for anomaly detection. 27  Pre-Training with Fully Convolutional Masked Autoencoders explores self-supervised pre-training using fully convolutional masked autoencoders, and 28  Foundation Models and Segment Anything introduces foundation models using the Segment Anything Model (SAM 2). Lastly, 29  Final Projects describes two final projects that ask you to design, execute, and report on a deep learning experiment.

Exercises

The exercises listed in Table 1 provide practice applying the methods covered in the associated chapters. The midterm project (14  Midterm Project) and the two final projects (29  Final Projects) ask you to design, execute, and report on your own experiment. Point values are not assigned, so instructors using this text can weight the exercises as they see fit.

Table 1. Exercises included in the text.

Exercise	Topic	Data
4.1	Classification losses	data/files/lossMetrics
4.2	Regression losses	data/files/lossMetrics
4.3	Classification accuracy assessment	data/files/lossMetrics
5.1	DataSet and DataLoader for a CNN-based scene labeling task	PatternNet
6.1	Fully connected ANN for tabulated data	Covertype (data/files/tables)
7.1	ANN model comparisons	Covertype (data/files/tables)
8.1	Training with PyTorch Lightning	Covertype (data/files/tables)
10.1	Define a CNN architecture	PatternNet
11.1	Train a CNN for a scene labeling task	PatternNet
Midterm Project	CNN scene classification experiment	PatternNet or another approved dataset
15.1	Build a configurable UNet	None
16.1	Train a UNet to map surface mines	topoDLMini
21.1	Build a custom training function for semantic segmentation	A semantic segmentation dataset of your choice
26.1	Detect surface mining using a VAE	topoDLMini
Final Projects 1 and 2	Semantic segmentation model comparison or deep learning computational experiment	Your choice, with instructor approval

For each exercise, you will submit a Quarto (.qmd) file that contains your code and your answers to the questions, along with the HTML webpage rendered from it. Exercises 4.1, 4.2, and 4.3 also ask you to perform calculations in Microsoft Excel or similar spreadsheet software. A simple header for an exercise file is shown below. Setting embed-resources to true stores the figures and styling within the HTML file, so that it can be submitted as a single file.

---
title: "Exercise 6.1"
author: "Your Name"
format:
  html:
    embed-resources: true
---

To render the file, open a terminal, activate the torchbook environment, navigate to the folder containing the file, and run quarto render followed by the file name (e.g., quarto render exercise6_1.qmd). Quarto uses Jupyter, which is installed in the torchbook environment, to execute the Python code chunks. The file can also be rendered from RStudio, VS Code, or Positron. Note that rendering executes all of the code in the file, including any training loops, so you may want to render the file once your code is complete or set #| eval: false for training chunks that you have already executed and load the saved models instead. Save any models, logs, or other files that your code generates to the data/output folder.

Data and Source Files

This book was written using Quarto. To learn more about Quarto, we recommend Quarto for Scientists by Nicholas Tierney, which was also written using Quarto and has been made available free-of-charge.

The book files and data can be downloaded using the download menu in the sidebar or using the links below. You will need to uncompress each archive before use.

• gsdlBookFiles.zip: the Quarto files, exercises, smaller datasets, and trained models (described below)
• topoDLMini.zip: image chips of topographic maps used in 21  Segmentation Models Package and Exercises 16.1 and 26.1
• vfillDLMini.zip: image chips of land surface parameters used in 27  Pre-Training with Fully Convolutional Masked Autoencoders
• vfillInst.zip: image chips and instance masks used in 22  Instance Segmentation with UNet-ID and 23  Instance Segmentation with Mask R-CNN

The uncompressed gsdlBook folder contains:

• The Quarto files for each chapter (e.g., T2_Tensors.qmd) and the project files needed to render the book (_quarto.yml, _setup.qmd, and the style files)
• The exercises folder, which contains the exercise files that are included at the end of the chapters
• The dlGraphics and figs folders, which contain the conceptual diagrams and other images displayed in the text
• The data/files folder, which contains the smaller datasets used in the chapters and exercises, organized into subfolders (e.g., tables, lossMetrics, EuroSat_Tabulated, chipGen, forSAM, and other)
• The data/models folder, which contains the trained models, training logs, and model predictions provided with the text
• The data/output folder, which is initially empty; all files generated by the code, including trained models, checkpoints, training logs, image chips, and predictions, are written to this folder so that the provided files are not overwritten
• The environment.yml file, which is used to create the torchbook Python environment, as described below

If you open a Quarto file from the uncompressed gsdlBook folder, the relative file paths used in the code (e.g., data/files/tables/cars.csv) should work so that the paths do not need to be updated. If you copy the code into your own scripts or notebooks, make sure the working directory is set to the gsdlBook folder or update the paths accordingly.

We have provided trained models in the data/models folder so that you can execute the example code without executing the associated training loops, which can take hours in some cases. Generally, inference using a trained model is much less computationally demanding than training.

Larger Datasets

The larger image datasets used in this text are not included in gsdlBookFiles.zip. Three of them are provided as separate downloads, as described above, and the rest must be downloaded from the sources listed in Table 2. By default, the code expects each dataset to be uncompressed into the data/files folder using the folder name listed in the table (e.g., data/files/vfillInst). If you would prefer to store a dataset elsewhere, set the folder path defined near the beginning of each chapter that uses it, which is highlighted in a Data Location callout, to its location on your machine.

Table 2. Larger datasets used in this text that must be downloaded separately.

Dataset	Default Folder	Used In	Source
EuroSATallBands (EuroSAT_MS.zip)	data/files/EuroSAT_MS	5  DataSets and DataLoaders, 11  Train a Convolutional Neural Network, 13  Transfer Learning and CNNs, 26  Anomaly Detection with VAEs	Zenodo
Landcover.ai	data/files/landcover.ai.v1	16  Train a UNet for Semantic Segmentation, 25  SegFormer	landcover.ai
topoDLMini	data/files/topoDLMini	21  Segmentation Models Package and Exercises 16.1 and 26.1	topoDLMini.zip
vfillDL	data/files/vfillDL	20  Creating Image Chips, 22  Instance Segmentation with UNet-ID, and 23  Instance Segmentation with Mask R-CNN	FigShare
vfillDLMini	data/files/vfillDLMini	27  Pre-Training with Fully Convolutional Masked Autoencoders	vfillDLMini.zip
vfillInst	data/files/vfillInst	22  Instance Segmentation with UNet-ID and 23  Instance Segmentation with Mask R-CNN	vfillInst.zip
PatternNet	data/files/PatternNet	Exercises 5.1, 10.1, and 11.1 and the midterm project	PatternNet website

The image chips for the Landcover.ai dataset are generated in 16  Train a UNet for Semantic Segmentation using a script provided with the dataset and are saved to the data/output/landcoverai_chips folder by default.

Setting Up Python

Why Use Conda?

Almost all deep learning and geospatial work in Python relies on additional packages, such as PyTorch, NumPy, and Rasterio, and many of these packages depend on specific versions of each other and on compiled, non-Python libraries (for example, the CUDA libraries used by PyTorch or the GDAL library used by Rasterio). Installing everything into a single, system-wide copy of Python quickly becomes difficult to manage.

For this reason, we recommend using conda to install Python and manage packages. Conda allows you to create environments, which are isolated installations of Python and a specific set of packages. All of the code in this book runs in a conda environment called torchbook. If you worked through Open Python for Geospatial Professionals, note that this is a separate environment from the one used in that text.

If you do not already have conda, download and install Miniconda, which provides conda, Python, and a small number of supporting packages. On Windows, use the Anaconda Prompt that the installer adds to the Start Menu to run the commands below.

GPU Requirements

PyTorch allows computation to be performed on central processing units (CPUs) or graphics processing units (GPUs). The early chapters can be executed on a CPU. However, training CNNs and semantic segmentation models on a CPU is extremely slow or intractable. A CUDA-enabled NVIDIA GPU is required to train the models presented in this text and to complete the exercises. You will need a recent NVIDIA graphics driver installed. The PyTorch packages installed by the environment file include the required CUDA libraries, so you do not need to install the CUDA Toolkit separately. Consult the PyTorch installation page for the options available for your operating system and hardware.

If you do not have access to a local GPU, you can use a GPU in Google Colab by going to Edit → Notebook Settings and setting the hardware accelerator to a GPU. Note that there are limits to the amount of computational time available in Colab.

Create the torchbook Environment

The environment.yml file, included in the book download, lists the packages used in this text.

1. Open a terminal (on Windows, the Anaconda Prompt) and navigate to the uncompressed gsdlBook folder, which contains environment.yml, using the cd command. For example:

   cd C:/Users/yourname/gsdlBook

2. Create the environment from the file:

   conda env create -f environment.yml

   Conda will resolve, download, and install all of the listed packages and their dependencies. This may take several minutes.

3. Activate the environment whenever you work through the book:

   conda activate torchbook

4. Confirm that PyTorch can access your GPU. The following command should print True:

   python -c "import torch; print(torch.cuda.is_available())"

You can then run the examples in JupyterLab (jupyter lab), VS Code, Positron, or any other editor that uses this environment. If the environment file changes, update the environment with conda env update -f environment.yml --prune.

Troubleshooting on Windows

The packages used in this text rely on compiled libraries (DLLs), and on Windows, conflicts between different copies of the same library are a common source of errors. Here are the issues we encountered and how to resolve them.

rasterio fails to import with “DLL load failed while importing _base”. This generally means that another installation of GDAL, the library Rasterio uses to read and write geospatial data, is listed on your system PATH and is being loaded instead of the copy installed in the torchbook environment. Stand-alone GDAL installations (e.g., C:\GDAL or C:\Program Files\GDAL) and some GIS software can add such entries. You can check by running the following in the Anaconda Prompt and looking for folders containing GDAL:

echo %PATH%

One option is to remove these folders from your system PATH. However, other software may depend on them. A less disruptive option is to remove them from PATH only while the torchbook environment is active. Conda runs any scripts stored in the environment’s etc/conda/activate.d folder when the environment is activated and those in etc/conda/deactivate.d when it is deactivated. You can find the environment’s folder by running conda env list. Create the following two files for the Anaconda Prompt (cmd). The first removes any PATH entry whose final folder is named GDAL and saves the original PATH, and the second restores the original PATH when the environment is deactivated.

activate.d/strip-system-gdal.bat:

@echo off
set "_TORCHBOOK_OLD_PATH=%PATH%"
for /f "usebackq delims=" %%i in (`powershell -NoProfile -Command "($env:PATH -split ';' | Where-Object { $_ -and ((Split-Path $_.TrimEnd([char]92) -Leaf) -ne 'GDAL') }) -join ';'"`) do set "PATH=%%i"

deactivate.d/strip-system-gdal.bat:

@echo off
if defined _TORCHBOOK_OLD_PATH (
    set "PATH=%_TORCHBOOK_OLD_PATH%"
    set "_TORCHBOOK_OLD_PATH="
)

If you activate the environment from PowerShell, create equivalent .ps1 files:

activate.d/strip-system-gdal.ps1:

$env:_TORCHBOOK_OLD_PATH = $env:PATH
$env:PATH = ($env:PATH -split ';' | Where-Object { $_ -and ((Split-Path $_.TrimEnd([char]92) -Leaf) -ne 'GDAL') }) -join ';'

deactivate.d/strip-system-gdal.ps1:

if ($env:_TORCHBOOK_OLD_PATH) {
    $env:PATH = $env:_TORCHBOOK_OLD_PATH
    Remove-Item Env:_TORCHBOOK_OLD_PATH
}

Deactivate and reactivate the environment, then confirm that Rasterio imports with python -c "import rasterio; print(rasterio.__version__)". Note that these scripts only run when the environment is activated. Editors that launch the environment’s python.exe directly, without activating it, will not run them. When rendering the book with RStudio and reticulate, the hidden setup code in _setup.qmd removes these folders from PATH for the R session, so the scripts are not needed for rendering.

importing torch fails with “[WinError 127] The specified procedure could not be found … shm.dll”. This occurs when torch is imported after NumPy or SciPy has loaded the Intel Math Kernel Library (MKL) from conda-forge. MKL loads its own copy of an OpenMP library, libiomp5md.dll, and the PyTorch packages include a different copy with the same name; Windows only loads one, and torch cannot use the copy loaded by MKL. The provided environment.yml avoids this by configuring NumPy and SciPy to use OpenBLAS as opposed to MKL. If you encounter this error in an environment you created yourself, you can switch to OpenBLAS with conda install "libblas=*=*openblas" or, as a temporary workaround, import torch before NumPy, Pandas, or SciPy.

Installing SAM 2 fails while compiling a CUDA extension. SAM 2, which is used in 28  Foundation Models and Segment Anything, attempts to compile an optional CUDA extension during installation, which requires Microsoft’s C++ build tools. The extension is only used to clean up small holes and specks in the predicted masks. To skip it, set the SAM2_BUILD_CUDA environment variable to 0 before creating or updating the environment (set SAM2_BUILD_CUDA=0 in the Anaconda Prompt or $env:SAM2_BUILD_CUDA=0 in PowerShell).

Using Python in RStudio with reticulate

This book was written in RStudio, and the Python code in each chapter was executed using the reticulate R package. If you would like to work in RStudio, install reticulate from CRAN and connect it to the torchbook environment before running any Python code:

reticulate::use_condaenv("torchbook", required = TRUE)

Every chapter begins by connecting reticulate to the torchbook environment in a hidden code chunk stored in _setup.qmd. If you render the book yourself and installed the environment into a different location, edit _setup.qmd accordingly. For a more detailed discussion of using Python with reticulate, see Open Python for Geospatial Professionals.

Reproducibility and Fair Model Comparisons

Training a deep learning model involves many random processes, so training the same model twice will generally not produce identical results. This matters for two reasons. First, you may want to be able to reproduce a result, for example, to confirm that a reported model can be re-created. Second, and more importantly for the exercises and projects in this text, when you compare two models or training configurations, you want the differences in their performance to result from the change you made, not from differences in how the data were partitioned, which augmentations were applied, or how the parameters were initialized. This section summarizes how to control these sources of randomness. The specific settings are demonstrated in 7  Improving Models and 8  Simplifying the Training Loop.

Sources of Randomness

Randomness enters a typical workflow in the following places:

1. Data partitioning: randomly splitting the samples into training, validation, and testing sets
2. Shuffling: the order in which a DataLoader delivers the training samples in each epoch
3. Data augmentation: the random flips, rotations, and other transformations applied to each sample
4. Parameter initialization: the random initial values of the trainable parameters, including any new layers added to a pre-trained model
5. Regularization: random operations applied during training, such as dropout
6. Non-deterministic GPU operations: some GPU operations, such as certain convolution algorithms and operations that accumulate values in parallel, can produce slightly different results each time they are executed

Use the Same Data Partitions

The most important step is to define the training, validation, and testing partitions once, save them to disk, and use the saved partitions for every model that you compare. For example, in 5  DataSets and DataLoaders, the EuroSAT partitions are saved as tables, and these tables are used in all later chapters. If partitions must be generated in code, set the random state explicitly (e.g., the random_state argument of scikit-learn’s train_test_split() or of the Pandas sample() method). For geospatial data, also consider how the data are partitioned, not just whether the partitions are reproducible. Since nearby samples tend to be similar, partitioning by image, tile, or geographic area, as done for the chips in 20  Creating Image Chips and 22  Instance Segmentation with UNet-ID, generally provides a more realistic assessment than randomly partitioning individual chips.

Seed the Random Number Generators

Python, NumPy, and PyTorch each have their own random number generators, and all three may be used during training. They can be seeded individually using random.seed(), np.random.seed(), and torch.manual_seed(), which also seeds the generators for all GPUs. PyTorch Lightning provides L.seed_everything(), which seeds all three in a single call. Setting workers=True also derives a different, but reproducible, seed for each DataLoader worker process, so that random augmentations applied in parallel workers are reproducible:

import lightning as L

L.seed_everything(42, workers=True)

The seed should be set at the beginning of each experiment, before the data, model, and DataLoaders are created.

Use the Same Augmentations and Sample Order

Seeding the global generators makes a single training run reproducible, but it does not guarantee that two different models receive the same sequence of augmented samples. The model and the data pipeline draw from the same generators, so if one model draws more random numbers than another (e.g., it has more parameters to initialize or uses dropout), all of the later random draws, including those used to shuffle and augment the data, will differ. To keep the data pipeline independent of the model, give it its own generators:

import torch
import albumentations as A
from torch.utils.data import DataLoader

trainTransform = A.Compose([A.HorizontalFlip(p=0.5), A.VerticalFlip(p=0.5),
                            A.RandomRotate90(p=0.5)], seed=42)
g = torch.Generator().manual_seed(42)
trainDL = DataLoader(trainDS, batch_size=32, shuffle=True, generator=g)

The generator argument controls the order in which the samples are shuffled, and the seed argument of Compose(), available in recent versions of Albumentations, controls the augmentations. Note that the augmentations applied to each sample also depend on the number of DataLoader workers, so the num_workers and batch_size arguments should also be kept the same.

Use the Same Initial Parameters

When comparing training settings, such as losses, optimizers, or learning rate schedules, for the same architecture, the most reliable way to ensure that every model starts from the same parameters is to initialize the model once, save its state dictionary, and load it at the beginning of each experiment:

torch.manual_seed(42)
model = myUNet(encoderChn=[16, 32, 64, 128], decoderChn=[128, 64, 32, 16], inChn=3, botChn=512, nCls=5)
torch.save(model.state_dict(), "data/output/initialParameters.pt")

# At the start of each experiment
model = myUNet(encoderChn=[16, 32, 64, 128], decoderChn=[128, 64, 32, 16], inChn=3, botChn=512, nCls=5)
model.load_state_dict(torch.load("data/output/initialParameters.pt"))

When comparing different architectures, or the same architecture with and without a module, the parameters cannot be identical, since the models have different parameters. In such cases, set the same seed before instantiating each model, and use the same pre-trained weights where applicable.

Use Deterministic Algorithms

To remove the variability caused by non-deterministic GPU operations, set deterministic=True when defining a Lightning Trainer, which calls torch.use_deterministic_algorithms(True). Setting deterministic="warn" issues a warning, as opposed to an error, when an operation without a deterministic implementation is used. Lightning also sets the CUBLAS_WORKSPACE_CONFIG environment variable to ":4096:8", which some GPU matrix operations require in order to be deterministic, and disables cuDNN benchmarking, which selects the fastest convolution algorithms at run time and can select different algorithms in different runs. Do not set benchmark=True when using deterministic=True. If you are not using Lightning, these settings must be made manually, as demonstrated in 7  Improving Models.

trainer = L.Trainer(max_epochs=50, accelerator="gpu", devices=1, deterministic=True,
                    default_root_dir="data/output/")

It is also good practice to record the settings used for each experiment. Calling self.save_hyperparameters() within a LightningModule stores its arguments in the checkpoint and in the hparams.yaml file written by the CSVLogger.

Limitations

These steps make experiments more consistent, but they have several important limitations:

1. Reproducibility is limited to the same environment: even with all of the settings above, results are generally only reproducible on the same hardware with the same drivers and package versions. Different GPUs, CUDA versions, or PyTorch versions can produce different results, and results computed on a CPU will differ from those computed on a GPU.
2. Some operations are not deterministic: some PyTorch operations do not have deterministic implementations, and libraries built on PyTorch may use them internally. Requiring deterministic algorithms will raise an error when they are used, so you may need to use the warning setting, avoid the operation, or accept some variability.
3. Deterministic algorithms can be slower: requiring deterministic algorithms and disabling benchmarking can increase training time.
4. Other settings change the results: changing the mini-batch size, number of DataLoader workers, number of GPUs, or numeric precision (e.g., bfloat16 mixed precision) changes the order or precision of the calculations and, therefore, the results, even when the seeds are the same.
5. The trained models provided with this text may not be exactly reproducible: they were trained without all of these settings and on a specific machine, so if you re-train them, expect your results to be similar, but not identical, to those reported.
6. A single run is a single sample: making a run reproducible does not make it representative. A different seed can produce a noticeably different result, especially for small datasets. When the differences between models are small, train each configuration several times using different seeds and report the mean and variability of the assessment metrics, as opposed to drawing conclusions from a single run per model.

Thanks

This online text was partially funded by AmericaView. AmericaView is a nationwide, university-based, and state-implemented network that advances Earth observation education through remote sensing science, applied research, workforce development, technology transfer, and community outreach. AmericaView is supported by the United States Geological Survey (USGS) under Grant/Cooperative Agreement No. G23AP00683. The views and conclusions contained in this text are those of the authors and should not be interpreted as representing the opinions or policies of the U.S. Geological Survey. Mention of trade names or commercial products does not constitute their endorsement by the U.S. Geological Survey.

Partial funding was also provided by the National Science Foundation (NSF) (Federal Award ID No. 2046059: “CAREER: Mapping Anthropocene Geomorphology with Deep Learning, Big Data Spatial Analytics, and LiDAR”). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Nomenclature

We use the following type standards throughout the text:

1. Bold text is used to highlight key or important terms at first use in each chapter (e.g., tensor).
2. Underlined text is used for Python library, package, or module names (e.g., PyTorch).
3. Italic text is used for file names, folder paths/directories, and book or journal titles (e.g., cars.csv).
4. Table or DataFrame column names are written in “quotes” and italicized (e.g., “Cover_Type”).
5. “Quotes” are placed around names of journal articles.
6. In-text code snippets, function and method names, and variable names are written as fixed-width text (e.g., nn.Conv2d()).
7. Callout blocks are used to highlight key information, offer practical advice, or relate concepts across chapters.

This is an example callout block.

Concluding Remarks

We hope you are excited to start learning to implement deep learning for geospatial applications using PyTorch. In the next chapter, we begin by building a simple fully connected neural network.