import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import pandas as pd
from sklearn import preprocessing
from sklearn.metrics import classification_report
import matplotlib.pyplot as plt
import seaborn as sbn4 Losses and Assessment Metrics
4.1 Introduction
Selecting an appropriate loss metric for a specific task is an important consideration when developing a deep learning workflow. The loss metric guides the weight/parameter updates and is the only measure available to the optimization algorithm as it attempts to adjust the weights/parameters and improve the model performance. I like to think of the learning process as stumbling around in the dark, and the loss metric serves as the only means to determine if it is moving in the right direction.
All loss metrics must be (1) differentiable and (2) minimized as the model improves or learning progresses. Since the goal is to minimize the loss metric, for classification problems you need to use a measure of error as opposed to a measure of accuracy.
In the first section of this module, we will focus on loss metrics for regression followed by classification. I will then explain how to define custom loss metrics. In the last section, we will explore the torchmetrics package, which offers a wide range of assessment metrics for a variety of scenarios.
This site offers examples of a wide range of custom loss functions: https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch/notebook.
I begin by importing the required packages including torch, torch.nn, and torch.nn.functional. I also import NumPy and Pandas since I will be working with arrays and data tables. I will also make use of a few functions from scikit-learn.
Next, I read in some regression and classification results so that I don’t have to train models and generate predictions just to experiment with loss functions and assessment metrics. The r1 object will be used to explore regression problems. The “truth” column represents the correct answer while the m1, m2, and m3 columns represent different prediction results. The c1 object represents a multiclass classification and includes predicted probabilities for 9 classes along with the correct class codes. Lastly, the b1 object represents results for a binary classification problem.
r1 = pd.read_csv("data/files/lossMetrics/regression_data.csv")
r1.head() truth m1 m2 m3
0 1.425000 1.276554 1.166438 1.173226
1 1.450000 1.716117 1.354679 1.371919
2 1.200000 1.003642 0.897605 0.903885
3 1.852941 1.401898 1.512243 1.497734
4 2.157895 2.326138 2.435581 2.455052c1 = pd.read_csv("data/files/lossMetrics/multiclass.csv")
c1.head() Unnamed: 0 asphalt building car ... shadow soil tree reference
0 1 0.04 0.20 0.41 ... 0.01 0.03 0.02 concrete
1 2 0.34 0.15 0.01 ... 0.49 0.00 0.00 shadow
2 3 0.39 0.01 0.01 ... 0.57 0.00 0.01 shadow
3 4 0.04 0.00 0.00 ... 0.33 0.00 0.61 tree
4 5 0.40 0.09 0.12 ... 0.14 0.04 0.03 asphalt
[5 rows x 11 columns]b1 = pd.read_csv("data/files/lossMetrics/binary_data.csv")
b1.head() truth predicted prob_not prob_fail
0 not not 0.812483 0.187517
1 not not 0.984213 0.015787
2 not not 0.991930 0.008070
3 not not 0.998299 0.001701
4 not not 0.948095 0.0519054.2 Regression Loss Metrics
We will begin with regression. For a regression problem, the loss is calculated from the residuals, or the differences between the predicted and reference values. The four losses discussed here differ in how they convert residuals to a penalty, which determines how strongly the model is pushed to fix large errors relative to small ones. This matters most when the data include outliers or when the error distribution is skewed. Table 4.1 summarizes the losses discussed in this section.
| Loss | Penalty for a residual \(r\) | Units | Strengths | Weaknesses |
|---|---|---|---|---|
| MSE | \(r^2\) | Squared units | Smooth and convex; strongly penalizes large errors; optimum is the conditional mean | Sensitive to outliers; squared units are hard to interpret |
| MAE | \(\lvert r \rvert\) | Original units | Robust to outliers; easy to interpret; optimum is the conditional median | Gradient has a constant magnitude and is undefined at 0, which can slow convergence near the optimum |
| Huber | \(\frac{1}{2}r^2\) if \(\lvert r \rvert \le \delta\); \(\delta(\lvert r \rvert - \frac{1}{2}\delta)\) otherwise | Mixed | Behaves like MSE for small errors and like MAE for large errors | Requires choosing \(\delta\), which depends on the scale of the target variable |
| Log-cosh | \(\log(\cosh(r))\) | Mixed | Smooth everywhere; approximately \(\frac{1}{2}r^2\) for small errors and \(\lvert r \rvert - \log 2\) for large errors | Not built into PyTorch; transition point is fixed relative to the target’s units |
4.2.1 Mean Square Error (MSE)
The mean square error (MSE) loss is the mean of the squared residuals:
\[\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2\]
As this measure is reduced, it suggests that the differences between the predicted and actual values are decreasing, or the model is doing a better job. Since the residuals are squared, large errors contribute disproportionately to the loss. A residual of 4 contributes 16 times more than a residual of 1. As a result, MSE strongly encourages the model to avoid large errors. The gradient of the squared error is proportional to the residual, so the size of the parameter updates shrinks as the predictions approach the reference values, which generally results in smooth and stable convergence. MSE is also the loss minimized by ordinary least squares regression, as we saw in Chapter 3, and the optimal constant prediction under MSE is the mean of the target variable.
The main weakness of MSE is its sensitivity to outliers. A small number of samples with very large errors, which may result from noisy or incorrect reference data, can dominate the loss and pull the model toward fitting them at the expense of the other samples. The loss is also reported in squared units, which makes it difficult to interpret directly. For this reason, the root mean square error (RMSE), which is in the original units, is commonly reported as an assessment metric even when MSE is used as the loss.
The MSE loss is provided by PyTorch, so it does not need to be built from scratch. Here, I am using the class-based version of the metric, which requires instantiating an instance of the class. I also need to prepare the reference and predicted values. This requires extracting the columns from the DataFrame, converting them to NumPy arrays, then converting the NumPy arrays to tensors. Since these are simple calculations and not computationally intensive, I will not use a GPU in this chapter. Once the data are prepared and the loss is instantiated, I obtain the loss measure using the loss metric and the reference and predicted values. The result is a tensor with a single value.
For the m1 results, the MSE loss is 0.1365. In the final code block in this section, I calculate this metric from scratch and without using the defined loss metric. This is accomplished by (1) differencing the reference and predicted values, (2) taking the square of each difference, and (3) calculating the average. The same result is obtained.
target = torch.tensor(r1['truth'].to_numpy())
pred = torch.tensor(r1['m1'].to_numpy())loss = nn.MSELoss()
loss(pred, target)tensor(0.1365, dtype=torch.float64)mse = torch.mean(torch.square(pred-target))
msetensor(0.1365, dtype=torch.float64)4.2.2 Mean Absolute Error (MAE)
The mean absolute error (MAE) loss is the mean of the absolute values of the residuals:
\[\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert y_i - \hat{y}_i \rvert\]
In contrast to MSE, each residual contributes to the loss in proportion to its magnitude: a residual of 4 contributes 4 times more than a residual of 1, as opposed to 16 times more. As a result, MAE is much more robust to outliers, and a few samples with very large errors will not dominate the training process. MAE is also reported in the same units as the target variable, so it is easy to interpret; for example, an MAE of 2.5 m for a canopy height model means that predictions are off by 2.5 m on average. The optimal constant prediction under MAE is the median as opposed to the mean.
The weakness of MAE relates to its gradient. The derivative of the absolute value is either -1 or 1, depending only on the sign of the residual, and it is undefined when the residual is exactly 0. So, the magnitude of the gradient does not decrease as the predictions approach the reference values. With a fixed learning rate, this can cause the parameter updates to overshoot and oscillate around the optimum. Using a learning rate scheduler that reduces the learning rate over time can help. In PyTorch, MAE is implemented as nn.L1Loss() since it is based on the L1 norm, while MSE is based on the L2 norm.
loss = nn.L1Loss()
loss(pred, target)mae = torch.mean(torch.abs(pred-target))
mae4.2.3 Huber Loss
The Huber loss combines the strengths of MSE and MAE. For residuals with an absolute value less than or equal to a threshold, \(\delta\), a squared penalty is applied. For larger residuals, a linear penalty is applied:
\[L_{\delta}(r) = \begin{cases} \frac{1}{2}r^2 & \text{if } \lvert r \rvert \le \delta \\ \delta\left(\lvert r \rvert - \frac{1}{2}\delta\right) & \text{otherwise} \end{cases}\]
Small errors are therefore treated as they are with MSE, which results in smooth convergence near the optimum, while large errors are treated as they are with MAE, which reduces the influence of outliers. The two pieces are joined such that the loss and its first derivative are continuous at \(\delta\).
The key practical consideration is selecting \(\delta\). This threshold defines what counts as a “large” error, so it must be selected relative to the scale of the target variable. If \(\delta\) is very large relative to the residuals, the Huber loss behaves like one-half of MSE; if it is very small, it behaves like a scaled MAE. If the target variable has been normalized to z-scores, a \(\delta\) of 1.0 (the PyTorch default) is a reasonable starting point. Otherwise, you may want to base \(\delta\) on the expected magnitude of typical errors or treat it as a hyperparameter to tune.
PyTorch provides nn.HuberLoss(), which accepts a delta argument. PyTorch also provides nn.SmoothL1Loss(), which is closely related: the smooth L1 loss with a beta argument is equal to the Huber loss divided by beta when delta equals beta, so the two are identical when both are set to 1.0. Below, I calculate the Huber loss using the PyTorch implementation and from scratch, using torch.where() to apply the appropriate piece of the function to each residual.
loss = nn.HuberLoss(delta=1.0)
loss(pred, target)delta = 1.0
r = torch.abs(pred-target)
huber = torch.where(r <= delta, 0.5*r**2, delta*(r - 0.5*delta))
torch.mean(huber)4.2.4 Log-Cosh Loss
The log-cosh loss is the logarithm of the hyperbolic cosine of the residuals:
\[L = \frac{1}{n}\sum_{i=1}^{n}\log\left(\cosh(\hat{y}_i - y_i)\right)\]
For small residuals, \(\log(\cosh(r))\) is approximately \(\frac{1}{2}r^2\), and for large residuals it is approximately \(\lvert r \rvert - \log 2\). So, it behaves similarly to the Huber loss with \(\delta = 1\). However, unlike the Huber loss, it is smooth everywhere (i.e., it is twice differentiable), and there is no threshold to select. The trade-off is that the transition between the squared and linear behavior is fixed relative to the units of the target variable. As a result, log-cosh is most appropriate when the target variable has been normalized. With unnormalized targets that have large values, almost all residuals fall in the linear region, and the loss behaves like MAE.
Log-cosh is not provided by PyTorch, so I implement it by subclassing nn.Module. Calculating \(\cosh(r)\) directly can overflow for large residuals since it grows exponentially. To avoid this, I use the following equivalent, numerically stable form, where softplus is defined as \(\log(1 + e^x)\):
\[\log(\cosh(r)) = r + \text{softplus}(-2r) - \log 2\]
import math
class LogCoshLoss(nn.Module):
def __init__(self, reduction="mean"):
super().__init__()
self.reduction = reduction
def forward(self, pred, target):
r = pred - target
# Stable form of log(cosh(r)); avoids overflow in cosh() for large residuals
lc = r + F.softplus(-2.0*r) - math.log(2.0)
if self.reduction == "mean":
return lc.mean()
if self.reduction == "sum":
return lc.sum()
return lcloss = LogCoshLoss()
loss(pred, target)# Direct calculation for comparison (only safe for small residuals)
torch.mean(torch.log(torch.cosh(pred-target)))4.2.5 Comparing the Regression Losses
To visualize how these losses treat residuals of different sizes, I calculate each loss for a range of residual values and plot them. Note how the MSE penalty grows quadratically while the MAE penalty grows linearly. The Huber and log-cosh penalties are nearly identical to one-half of MSE near zero and grow linearly for large residuals.
r = torch.linspace(-4, 4, 401)
delta = 1.0
lossDF = pd.DataFrame({
"residual": r.numpy(),
"MSE": (r**2).numpy(),
"MAE": torch.abs(r).numpy(),
"Huber": torch.where(torch.abs(r) <= delta, 0.5*r**2,
delta*(torch.abs(r) - 0.5*delta)).numpy(),
"Log-cosh": (r + F.softplus(-2.0*r) - math.log(2.0)).numpy(),
})
lossDFLong = lossDF.melt(id_vars="residual", var_name="Loss", value_name="Penalty")
fig, ax = plt.subplots(figsize=(7, 5))
sbn.lineplot(data=lossDFLong, x="residual", y="Penalty", hue="Loss", ax=ax)
ax.set_ylim(0, 8)
ax.set_xlabel("Residual")
plt.show()The following code calculates all four losses for each of the three sets of predictions in the example table. Comparing the ranking of the models using the different losses can be informative. If a model has a relatively low MAE but a relatively high MSE, this suggests that it is generally accurate but makes a small number of large errors.
lossFns = {"MSE": nn.MSELoss(), "MAE": nn.L1Loss(),
"Huber": nn.HuberLoss(delta=1.0), "Log-cosh": LogCoshLoss()}
target = torch.tensor(r1['truth'].to_numpy())
results = []
for m in ["m1", "m2", "m3"]:
pred = torch.tensor(r1[m].to_numpy())
results.append({"model": m, **{k: fn(pred, target).item() for k, fn in lossFns.items()}})
pd.DataFrame(results)So, which loss should you use? There is no single best choice, but here are some general guidelines:
- Use MSE as a default when the reference data are reliable and you want large errors to be strongly penalized.
- Use MAE when the data contain outliers or noisy labels that you do not want to dominate the training process, or when you want the model to predict the conditional median as opposed to the mean.
- Use Huber when you want the stable convergence of MSE but with reduced sensitivity to outliers and are willing to select \(\delta\) relative to the scale of your target variable.
- Use log-cosh when you want behavior similar to Huber but without a threshold to tune. Make sure the target variable is normalized.
Regardless of which loss is used to train the model, it is common to report several assessment metrics, such as RMSE, MAE, and R-squared, to describe a model’s performance, as we will see at the end of this chapter.
4.3 Classification Loss Metrics
When performing a binary classification, or differentiating two classes, it is possible to treat the problem the same as a multiclass classification. In such a case the last layer should output logits for both classes. Another option is to output only a single logit, which will be associated with the positive case. Once logits are converted to probabilities, they must sum to 1. So, for a two class problem, the background or negative class probability can be obtained by simply subtracting the positive class probability from 1. In this section, we will explore the binary cross entropy (BCE) loss, which is appropriate when only the positive case logit is returned, and the cross entropy (CE) loss, which is used when a logit is returned for each class.
Table 4.2 summarizes the classification losses discussed in this chapter. In the equations, \(y\) is the reference value (0 or 1 for a binary problem), \(p\) is the predicted probability, \(N\) is the number of samples, and TP, FP, and FN are the true positive, false positive, and false negative counts, which are calculated from probabilities as opposed to hard classifications for the Dice and Tversky losses so that the losses are differentiable. A key consideration when using the PyTorch implementations is whether the loss expects logits or probabilities. Providing the wrong input will not cause an error, but it will result in an incorrect loss and poor training.
| Loss | Equation | PyTorch Implementation | Expected Predictions | Use Case and Notes |
|---|---|---|---|---|
| Binary cross entropy (BCE) | \(-\frac{1}{N}\sum [y\log(p) + (1-y)\log(1-p)]\) | nn.BCELoss() |
Positive class probabilities (apply a sigmoid first) | Binary classification with a single output node |
| BCE with logits | Same as BCE | nn.BCEWithLogitsLoss() |
Positive class logits | Combines a sigmoid and BCE in a single, more numerically stable operation; preferred over nn.BCELoss(); pos_weight can be used to weight the positive class |
| Cross entropy (CE) | \(-\frac{1}{N}\sum \log(p_{\text{correct class}})\) | nn.CrossEntropyLoss() |
Logits for all classes | Multiclass classification or binary classification with two output nodes; applies a softmax internally; supports class weights and label smoothing |
| Negative log likelihood (NLL) | Same as CE | nn.NLLLoss() |
Log probabilities for all classes | Equivalent to CE when the predictions are log probabilities (e.g., from F.log_softmax()) |
| Dice | \(1 - \frac{2TP}{2TP + FP + FN}\) | Custom (Section 4.4.1) | Probabilities | Less sensitive to class imbalance than BCE or CE since true negatives are not considered |
| Tversky | \(1 - \frac{TP}{TP + \alpha FP + \beta FN}\) | Custom (Section 4.4.2) | Probabilities | Generalization of Dice; \(\alpha\) and \(\beta\) control the relative penalties for FPs and FNs |
| Focal Tversky | \((1 - TI)^{1/\gamma}\), where \(TI\) is the Tversky index | Custom (Section 4.4.3) | Probabilities | Places more emphasis on difficult samples or classes |
| Combination (e.g., Dice + BCE) | Weighted sum of the component losses | Custom (Section 4.4.4) | Depends on the components | Combines the stability of BCE or CE with the class imbalance robustness of Dice or Tversky |
First, I prepare the data using the LabelEncoder() function from scikit-learn. Since the correct class is presented as a text field, the data must be converted to numeric codes where 0 represents the negative class and 1 represents the positive class. The LabelEncoder() function converts text into numeric codes alphabetically. The data I am using represent predictions of whether or not a location is a landslide. “not” indicates the negative class while “slopeD” represents the positive class. So, based on alphabetical order, “not” is coded to 0 and “slopeD” is coded to 1, which is correct in this case. Again, the positive case must be coded to 1 and the negative case must be coded to 0.
label_encoder = preprocessing.LabelEncoder()
b1['class_code'] = label_encoder.fit_transform(b1['truth'])
b1.head() truth predicted prob_not prob_fail class_code
0 not not 0.812483 0.187517 0
1 not not 0.984213 0.015787 0
2 not not 0.991930 0.008070 0
3 not not 0.998299 0.001701 0
4 not not 0.948095 0.051905 0I next convert the reference class numeric codes and positive class probabilities to NumPy arrays followed by torch tensors. Note that BCELoss() expects both to be stored using the double data type.
I next instantiate an instance of the BCELoss() class then feed it the references and predictions to obtain the BCE loss. Note that the BCELoss() implementation in PyTorch expects the data to be converted to probabilities before using the loss metric. If the neural network outputs raw logits, you will need to apply a sigmoid function to obtain probabilities prior to using the loss metric. The resulting loss is 0.3026.
target = torch.tensor(b1['class_code'].to_numpy(), dtype=torch.double)
pred = torch.tensor(b1['prob_fail'].to_numpy())loss = nn.BCELoss()
loss(pred, target)tensor(0.3026, dtype=torch.float64)The cell below shows how to implement the BCE loss from scratch. This requires multiplying the correct value (0 or 1) by the log of the positive class probability and then subtracting the class code from 1 and multiplying by the log of 1 minus the positive class probability. The results are averaged then multiplied by -1.
bce =-1*(torch.mean((target * torch.log(pred)) + ((1 - target)*torch.log(1-pred))))
bcetensor(0.3026, dtype=torch.float64)BCELoss() is not appropriate for a multiclass problem or a binary classification where the positive and negative probabilities are both obtained. Instead, cross entropy (CE) loss, implemented with CrossEntropyLoss(), should be used. This loss expects the output to be provided as logits as opposed to probabilities. It internally implements a softmax activation prior to calculating the CE loss. Since I already have class probabilities in the example data, I will use the negative log likelihood loss, implemented with NLLLoss(), which does not apply a softmax activation before calculating probabilities. CrossEntropyLoss() consists of a softmax activation followed by NLLLoss().
Similar to the binary classification problem, I first convert the class labels into numeric codes based on alphabetic order using the scikit-learn LabelEncoder() function. NLLLoss() expects probabilities for all classes; as a result, I extract the columns in the data table that provide the predicted class probabilities. Since these columns are in alphabetical order, the column index will match the calculated class code, which is necessary since probabilities are associated with the correct class such that the index is the same as the class code.
c1.head() Unnamed: 0 asphalt building car ... shadow soil tree reference
0 1 0.04 0.20 0.41 ... 0.01 0.03 0.02 concrete
1 2 0.34 0.15 0.01 ... 0.49 0.00 0.00 shadow
2 3 0.39 0.01 0.01 ... 0.57 0.00 0.01 shadow
3 4 0.04 0.00 0.00 ... 0.33 0.00 0.61 tree
4 5 0.40 0.09 0.12 ... 0.14 0.04 0.03 asphalt
[5 rows x 11 columns]pred = torch.tensor(c1[c1.columns[1:10]].to_numpy())
label_encoder = preprocessing.LabelEncoder()
c1['class_code'] = label_encoder.fit_transform(c1['reference'])
target = torch.tensor(c1['class_code'].to_numpy(), dtype=torch.int64)print(pred.shape)torch.Size([507, 9])print(target.shape)torch.Size([507])Printing the predictions, you can see that the result is a 2D array in which the first dimension, or rows, provide the predicted class probabilities for each sample and the columns correspond to a specific class. All probabilities for each sample or row must sum to 1. The shape of the tensor holding the predictions is (507, 9). So, there are 507 samples and 9 predicted probabilities (one for each class).
In contrast, the reference tensor is 1D and consists of the correct class code for each sample.
loss = nn.NLLLoss()
loss(pred, target)tensor(-0.6677, dtype=torch.float64)4.4 Custom Loss Metrics
It is possible to define a custom loss metric. Although functional forms can be defined, I will generate loss metrics by subclassing nn.Module, similar to how I define neural network architectures. Settings that control how the loss is calculated, such as smoothing terms or weights, are defined in the __init__() constructor method, as is done for the losses provided by PyTorch, while the forward() method accepts only the predictions and targets. This allows a configured loss to be instantiated once and then used repeatedly within a training loop. The following link provides examples of a variety of custom loss function implementations using both Keras/Tensorflow and PyTorch: https://www.kaggle.com/code/bigironsphere/loss-function-library-keras-pytorch/notebook.
4.4.1 Dice Loss
The BCE loss can be negatively impacted by class imbalance. So, it is common to use the Dice loss when there are issues of class imbalance. The Dice loss is calculated as follows:
\[L_{Dice} = 1 - \frac{2\sum \hat{p}_{\text{TP}} + \epsilon}{2\sum \hat{p}_{\text{TP}} + \sum \hat{p}_{\text{FN}} + \sum \hat{p}_{\text{FP}} + \epsilon}\]
True positives (TPs) represent samples that are examples of the positive class and were correctly predicted to the positive class whereas true negatives (TNs) are samples from the negative class that are correctly predicted to the negative class. False positives (FPs) are negative samples that are incorrectly predicted to the positive class while false negatives (FNs) are from the positive class but incorrectly predicted to the negative class. Note that class probabilities are used here as opposed to “hard” counts. Also, the Dice loss can be modified for a multiclass problem, as we will see in Chapter 7. The example below is only appropriate for a binary classification where the logit or probability for the positive case is predicted.
In order to implement the Dice loss, I begin by subclassing nn.Module. A few notes about this implementation:
- To make the loss differentiable, “soft” counts are calculated from the predicted probabilities as opposed to hard counts. For example, the soft TP count is the sum of the predicted positive class probabilities for samples that are actually positive.
- Since the Dice coefficient is a measure of accuracy as opposed to error, Dice loss is 1 minus Dice.
- A smoothing term (\(\epsilon\),
smooth) is added to both the numerator and denominator. This avoids division by zero when a mini-batch contains no positive samples and no positive predictions. Adding it to both terms means that a perfect prediction yields a Dice coefficient of exactly 1 and a loss of 0. It also stabilizes the loss when the counts are small. The value is generally set to 1 or to a small number such as 1e-6; larger values make the loss less sensitive to small mini-batches with few positive samples. - If the user provides raw logits, the fromProbs parameter should be set to False, and the implementation will convert the raw logits to probabilities using a sigmoid function. If the raw logits have already been converted to probabilities, this conversion is skipped. Since most networks return logits, it is generally preferable to pass logits to the loss and perform the sigmoid inside of it.
- The soft counts are summed across all samples in the mini-batch, so a single Dice coefficient is calculated for the entire mini-batch. An alternative is to calculate the Dice coefficient for each sample and average the results. The batch-level version is more stable when some samples contain few or no positive pixels, which is common in geospatial semantic segmentation.
class DiceLoss(nn.Module):
def __init__(self, smooth=1.0, fromProbs=False):
super().__init__()
self.smooth = smooth
self.fromProbs = fromProbs
def forward(self, inputs, targets):
if not self.fromProbs:
inputs = torch.sigmoid(inputs)
inputs = inputs.reshape(-1)
targets = targets.reshape(-1).to(inputs.dtype)
TP = (inputs * targets).sum()
FP = ((1-targets) * inputs).sum()
FN = (targets * (1-inputs)).sum()
dice = (2*TP + self.smooth) / (2*TP + FP + FN + self.smooth)
return 1 - diceOnce the Dice loss is defined, it can be used the same as the BCELoss(). Since the example data provide probabilities as opposed to logits, I set fromProbs to True.
target = torch.tensor(b1['class_code'].to_numpy(), dtype=torch.double)
pred = torch.tensor(b1['prob_fail'].to_numpy())loss = DiceLoss(fromProbs=True)
loss(pred, target)4.4.2 Tversky Loss
The Tversky loss is similar to the Dice loss except that it accepts alpha (\(\alpha\)) and beta (\(\beta\)) parameters that control the relative weight or influence of FP and FN errors, respectively:
\[L_{Tversky} = 1 - \frac{\sum \hat{p}_{\text{TP}} + \epsilon}{\sum \hat{p}_{\text{TP}} + \alpha \sum \hat{p}_{\text{FP}} + \beta \sum \hat{p}_{\text{FN}} + \epsilon}\]
When \(\alpha = \beta = 0.5\), the Tversky loss is equal to the Dice loss. Increasing \(\beta\) relative to \(\alpha\) penalizes FNs more heavily, which encourages higher recall, or fewer omission errors. Increasing \(\alpha\) relative to \(\beta\) penalizes FPs more heavily, which encourages higher precision, or fewer commission errors. The terms are commonly set so that they sum to 1. I follow the convention used in the paper that introduced the loss (Salehi et al., 2017) and in the Segmentation Models package, which we use in Chapter 21: alpha weights FPs and beta weights FNs. Be aware that some implementations reverse this convention, so it is important to check the documentation when using a loss implemented by someone else.
Salehi, S.S.M., Erdogmus, D. and Gholipour, A., 2017. “Tversky loss function for image segmentation using 3D fully convolutional deep networks.” In International Workshop on Machine Learning in Medical Imaging (pp. 379-387). Springer.
class TverskyLoss(nn.Module):
def __init__(self, alpha=0.5, beta=0.5, smooth=1.0, fromProbs=False):
super().__init__()
self.alpha = alpha
self.beta = beta
self.smooth = smooth
self.fromProbs = fromProbs
def forward(self, inputs, targets):
if not self.fromProbs:
inputs = torch.sigmoid(inputs)
inputs = inputs.reshape(-1)
targets = targets.reshape(-1).to(inputs.dtype)
TP = (inputs * targets).sum()
FP = ((1-targets) * inputs).sum()
FN = (targets * (1-inputs)).sum()
tversky = (TP + self.smooth) / (TP + self.alpha*FP + self.beta*FN + self.smooth)
return 1 - tverskyI then apply the loss to the binary classification result using an alpha of 0.3 and a beta of 0.7. This results in a higher weight or penalty applied to incorrectly labeling a positive sample to the negative class (FNs) in comparison to incorrectly labeling a negative sample to the positive class (FPs).
loss = TverskyLoss(alpha=0.3, beta=0.7, fromProbs=True)
loss(pred, target)4.4.3 Focal Tversky Loss
The focal Tversky loss (Abraham and Khan, 2019) raises the Tversky loss to a power defined using a gamma (\(\gamma\)) term:
\[L_{FT} = \left(1 - TI\right)^{\frac{1}{\gamma}}, \quad TI = \frac{\sum \hat{p}_{\text{TP}} + \epsilon}{\sum \hat{p}_{\text{TP}} + \alpha \sum \hat{p}_{\text{FP}} + \beta \sum \hat{p}_{\text{FN}} + \epsilon}\]
where \(TI\) is the Tversky index. When \(\gamma = 1\), the focal Tversky loss is equal to the Tversky loss. The authors suggest values of \(\gamma\) between 1 and 3 and used \(\gamma = 4/3\). Since \(1 - TI\) is between 0 and 1, an exponent less than 1 (i.e., \(\gamma > 1\)) increases the loss and its gradient when the Tversky index is already high. The authors’ intent was to keep the model focused on improving predictions for difficult cases, such as small objects or regions, even after much of the image is predicted well, and to avoid the loss becoming very small too quickly. Because the effect of \(\gamma\) depends on how the exponent is defined, be careful when comparing implementations: some implementations raise the loss to \(\gamma\) as opposed to \(1/\gamma\), in which case values less than 1 have the effect described here.
Abraham, N. and Khan, N.M., 2019. “A novel focal Tversky loss function with improved attention U-Net for lesion segmentation.” In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) (pp. 683-687). IEEE.
class FocalTverskyLoss(nn.Module):
def __init__(self, alpha=0.5, beta=0.5, gamma=1.0, smooth=1.0, fromProbs=False):
super().__init__()
self.alpha = alpha
self.beta = beta
self.gamma = gamma
self.smooth = smooth
self.fromProbs = fromProbs
def forward(self, inputs, targets):
if not self.fromProbs:
inputs = torch.sigmoid(inputs)
inputs = inputs.reshape(-1)
targets = targets.reshape(-1).to(inputs.dtype)
TP = (inputs * targets).sum()
FP = ((1-targets) * inputs).sum()
FN = (targets * (1-inputs)).sum()
tversky = (TP + self.smooth) / (TP + self.alpha*FP + self.beta*FN + self.smooth)
return (1 - tversky)**(1/self.gamma)loss = FocalTverskyLoss(alpha=0.3, beta=0.7, gamma=4/3, fromProbs=True)
loss(pred, target)4.4.4 Combination Losses
Lastly, it is possible to combine losses to create a combo loss. In the example below, I am creating a combo loss from Dice and BCE. The user can also specify the relative weight of the BCE component in the final metric. Combining a region-based loss, such as Dice, with a distribution-based loss, such as BCE, is common since BCE provides smooth, well-behaved gradients for every sample while Dice reduces the impact of class imbalance.
Note that I use F.binary_cross_entropy_with_logits() when logits are provided. This function combines the sigmoid and BCE calculations in a way that is more numerically stable than applying a sigmoid and then calculating BCE from the resulting probabilities, since it avoids taking the log of probabilities that are very close to 0 or 1.
class DiceBCELoss(nn.Module):
def __init__(self, bceWght=1.0, smooth=1.0, fromProbs=False):
super().__init__()
self.bceWght = bceWght
self.smooth = smooth
self.fromProbs = fromProbs
def forward(self, inputs, targets):
inputs = inputs.reshape(-1)
targets = targets.reshape(-1).to(inputs.dtype)
if self.fromProbs:
probs = inputs
bce = F.binary_cross_entropy(probs, targets)
else:
probs = torch.sigmoid(inputs)
bce = F.binary_cross_entropy_with_logits(inputs, targets)
TP = (probs * targets).sum()
FP = ((1-targets) * probs).sum()
FN = (targets * (1-probs)).sum()
dice = (2*TP + self.smooth) / (2*TP + FP + FN + self.smooth)
return (1 - dice) + self.bceWght*bceloss = DiceBCELoss(bceWght=0.75, fromProbs=True)
loss(pred, target)4.5 Assessment Metrics with torchmetrics
Similar to loss metrics, you can define custom assessment metrics functionally or by subclassing nn.Module. However, I will not demonstrate that here. This is partially because there is some preferred functionality that is difficult to code on your own, such as adapting classification metrics to work with both binary and multiclass classification problems or accumulating metric results with each mini-batch update. Given this complexity, I will provide a brief introduction to the torchmetrics package. This package provides access to a wide variety of metrics for both regression and classification. It also provides metrics for specific tasks, such as text and audio classification.
It should also be noted that assessment should be conducted using a withheld validation set as opposed to the training set. This is because, due to overfitting, assessment metrics calculated from the training data can be overly optimistic. It is generally preferred to calculate assessment metrics on the withheld validation data at the end of each training epoch. A third subset, the testing data, is often used to evaluate the final model.
Since torchmetrics is not a standard package, it may need to be installed, as demonstrated in the cell below. Following installation, I am importing the package with an alias of tm.
#!pip install torchmetricsimport torchmetrics as tmIn later modules, you will see applications of metrics implemented with torchmetrics within training loops with aggregation performed across mini-batches. In such cases, I will use the class-based implementation of the metrics. For demonstration purposes, here I will use the functional implementations, which do not require instantiation.
4.5.1 Binary Classification Metrics
For a binary classification, the assessment metrics are calculated from the counts of true positives (TP; positive samples predicted as positive), false positives (FP; negative samples predicted as positive), true negatives (TN; negative samples predicted as negative), and false negatives (FN; positive samples predicted as negative). Commonly used metrics are summarized in Table 4.3. In torchmetrics, binary metrics are calculated by setting task="binary". The predictions can be provided as positive class probabilities or logits, which are converted to hard classifications using a threshold of 0.5 by default (the threshold argument), or as hard classifications coded to 0 and 1. The reference labels must be coded to 0 (negative) and 1 (positive).
| Metric | Description | Equation | torchmetrics |
|---|---|---|---|
| Overall accuracy | Proportion of all samples that are correctly classified | \(\frac{TP + TN}{TP + TN + FP + FN}\) | tm.functional.accuracy(pred, target, task="binary") |
| Precision | Proportion of samples predicted as positive that are actually positive; 1 minus the commission error rate for the positive class | \(\frac{TP}{TP + FP}\) | tm.functional.precision(pred, target, task="binary") |
| Recall (sensitivity) | Proportion of positive samples that are predicted as positive; 1 minus the omission error rate for the positive class | \(\frac{TP}{TP + FN}\) | tm.functional.recall(pred, target, task="binary") |
| F1-score | Harmonic mean of precision and recall | \(\frac{2 \times Precision \times Recall}{Precision + Recall} = \frac{2TP}{2TP + FP + FN}\) | tm.functional.f1_score(pred, target, task="binary") |
| Specificity | Proportion of negative samples that are predicted as negative | \(\frac{TN}{TN + FP}\) | tm.functional.specificity(pred, target, task="binary") |
| Negative predictive value (NPV) | Proportion of samples predicted as negative that are actually negative | \(\frac{TN}{TN + FN}\) | tm.functional.negative_predictive_value(pred, target, task="binary") |
Precision, recall, and the F1-score focus on the positive class and do not consider TNs, which makes them informative when the positive class is rare. In contrast, overall accuracy can be high for an imbalanced dataset even if few positive samples are correctly identified. Specificity and NPV describe the accuracy for the negative class and are the negative class equivalents of recall and precision, respectively.
The code below calculates each metric for the binary landslide data using the positive class probabilities and the default threshold of 0.5.
predB = torch.tensor(b1['prob_fail'].to_numpy())
targetB = torch.tensor(b1['class_code'].to_numpy())
binaryMetrics = {"Overall Accuracy": tm.functional.accuracy,
"Precision": tm.functional.precision,
"Recall": tm.functional.recall,
"F1-Score": tm.functional.f1_score,
"Specificity": tm.functional.specificity,
"NPV": tm.functional.negative_predictive_value}
for name, metric in binaryMetrics.items():
print(f"{name} = {metric(predB, targetB, task='binary'):.3f}")Overall Accuracy = 0.875
Precision = 0.853
Recall = 0.906
F1-Score = 0.879
Specificity = 0.844
NPV = 0.9004.5.2 Multiclass Classification Metrics
For a multiclass classification, the metrics are calculated from the confusion matrix, in which each cell stores the number of samples from a reference class that were predicted as a given class. In the equations in Table 4.4, \(n_{kk}\) is the number of correctly classified samples for class \(k\) (the diagonal of the confusion matrix), \(n_{k+}\) is the total number of samples predicted as class \(k\), \(n_{+k}\) is the total number of reference samples for class \(k\), \(N\) is the total number of samples, and \(K\) is the number of classes. Precision and recall are commonly referred to as user’s accuracy and producer’s accuracy, respectively, in the remote sensing literature. Class-level metrics are returned by setting average="none", and a single class-aggregated value is returned by setting average="macro" or average="micro", as discussed below.
average="macro" returns the mean of the class-level precision, recall, or F1-score values.
| Metric | Description | Equation | torchmetrics |
|---|---|---|---|
| Overall accuracy | Proportion of all samples that are correctly classified | \(\frac{\sum_{k=1}^{K} n_{kk}}{N}\) | tm.functional.accuracy(pred, target, task="multiclass", num_classes=K, average="micro") |
| Kappa | Agreement between the predictions and reference labels after accounting for the agreement expected by chance (\(p_e\)); generally not recommended as a primary metric | \(\frac{p_o - p_e}{1 - p_e}\), where \(p_o\) is overall accuracy and \(p_e = \sum_{k=1}^{K}\frac{n_{k+} n_{+k}}{N^2}\) | tm.functional.cohen_kappa(pred, target, task="multiclass", num_classes=K) |
| Precision (user’s accuracy) | For each class, the proportion of samples predicted as that class that actually belong to it; 1 minus the commission error rate | \(\frac{n_{kk}}{n_{k+}}\) | tm.functional.precision(pred, target, task="multiclass", num_classes=K, average="none") |
| Recall (producer’s accuracy) | For each class, the proportion of the reference samples of that class that are correctly predicted; 1 minus the omission error rate | \(\frac{n_{kk}}{n_{+k}}\) | tm.functional.recall(pred, target, task="multiclass", num_classes=K, average="none") |
| F1-score | For each class, the harmonic mean of precision and recall | \(\frac{2 \times Precision_k \times Recall_k}{Precision_k + Recall_k}\) | tm.functional.f1_score(pred, target, task="multiclass", num_classes=K, average="none") |
The first example demonstrates obtaining overall accuracy, the class-aggregated F1-score, and the full confusion matrix for a multiclass problem. This requires setting the task parameter to ‘multiclass’ and defining the number of classes being differentiated. Overall accuracy and F1-score generate a single metric while the confusion matrix is a square matrix. It should be noted that most classification assessment metrics are derived from the confusion matrix, other than those calculated from class probabilities using multiple thresholds (i.e., area under the receiver operating characteristic curve (AUC ROC) and area under the precision-recall curve (AUC PR)). So, if you obtain a confusion matrix, you can derive other metrics from it.
Again, you will see more examples of classification metric implementations using torchmetrics in later modules.
pred = torch.tensor(c1[c1.columns[1:10]].to_numpy())
label_encoder = preprocessing.LabelEncoder()
c1['class_code'] = label_encoder.fit_transform(c1['reference'])
target = torch.tensor(c1['class_code'].to_numpy())acc = tm.functional.accuracy(pred, target, task='multiclass', num_classes=9)
f1 = tm.functional.f1_score(pred, target, task='multiclass', num_classes=9)
cm = tm.functional.confusion_matrix(pred, target, task='multiclass', num_classes=9)
print(f'Accuracy = {acc:.3f}, Average F1-Score: {f1:.3f}')Accuracy = 0.803, Average F1-Score: 0.803print(cm)tensor([[38, 0, 2, 1, 0, 0, 3, 1, 0],
[ 0, 68, 3, 21, 0, 0, 2, 3, 0],
[ 0, 0, 20, 1, 0, 0, 0, 0, 0],
[ 0, 4, 5, 83, 0, 0, 0, 1, 0],
[ 0, 1, 1, 0, 70, 0, 0, 4, 7],
[ 0, 0, 0, 0, 1, 13, 0, 0, 0],
[ 4, 0, 1, 0, 0, 0, 39, 0, 1],
[ 0, 3, 3, 1, 0, 0, 0, 13, 0],
[ 0, 0, 0, 0, 23, 0, 3, 0, 63]])In the confusion matrix returned by torchmetrics, the rows represent the reference classes and the columns represent the predicted classes. So, the row totals are the reference totals (\(n_{+k}\) in Table 4.4) and the column totals are the predicted totals (\(n_{k+}\)). Note that this orientation is the transpose of the convention commonly used in the remote sensing literature, in which the rows represent the predictions. The Kappa statistic can be calculated as follows.
kappa = tm.functional.cohen_kappa(pred, target, task='multiclass', num_classes=9)
print(f'Kappa = {kappa:.3f}')Kappa = 0.770For metrics that are calculated at the class-level, it is possible to generate an average to obtain a single metric. For example, the class-level F1-scores can be aggregated using either macro or micro averaging. In the case of macro averaging, each F1-score is calculated separately for each class and then averaged. In the case of micro averaging, the number of TPs, FNs, and FPs are calculated for each class and then summed. The summed TP, FN, and FP counts are then used to calculate the class-aggregated metric. In torchmetrics, the average parameter controls if and how averaging is applied. The default is micro averaging. If this argument is set to “none” then no averaging is applied and the class-level metrics are returned. The code block below demonstrates these different behaviors.
Macro and micro averaging will generally yield similar results if the classes are not heavily imbalanced. However, the results will diverge with increasing class imbalance. Since macro averaging simply calculates the metric separately for each class and then averages them, each class is equally weighted in the calculation. When micro averaging is used, the more abundant classes will have greater weights in the aggregated metric. In other words, micro averaging is more sensitive to class imbalance. Note that micro-averaged F1-score, recall, and precision are all equivalent. They are also equivalent to overall accuracy. As a result, I generally recommend calculating overall accuracy and the macro-averaged versions of the F1-score, precision, and recall.
f1Micro = tm.functional.f1_score(pred, target, task='multiclass', average = "micro", num_classes=9)
f1Macro = tm.functional.f1_score(pred, target, task='multiclass', average = "macro", num_classes=9)
f1NoAgg = tm.functional.f1_score(pred, target, task='multiclass', average = "none", num_classes=9)
print("Class-Aggregated F1-Score with Micro Averaging = " + str(f1Micro))Class-Aggregated F1-Score with Micro Averaging = tensor(0.8028)print("Class-Aggregated F1-Score with Macro Averaging = " + str(f1Macro))Class-Aggregated F1-Score with Macro Averaging = tensor(0.8014)print("F1-Score with No Averaging = " + str(f1NoAgg))F1-Score with No Averaging = tensor([0.8736, 0.7861, 0.7143, 0.8300, 0.7910, 0.9630, 0.8478, 0.6190, 0.7875])The code blocks below demonstrate the different averaging techniques for the recall and precision metrics.
reMicro = tm.functional.recall(pred, target, task='multiclass', average = "micro", num_classes=9)
reMacro = tm.functional.recall(pred, target, task='multiclass', average = "macro", num_classes=9)
reNoAgg = tm.functional.recall(pred, target, task='multiclass', average = "none", num_classes=9)
print("Class-Aggregated Recall with Micro Averaging = " + str(reMicro))Class-Aggregated Recall with Micro Averaging = tensor(0.8028)print("Class-Aggregated Recall with Macro Averaging = " + str(reMacro))Class-Aggregated Recall with Macro Averaging = tensor(0.8208)print("Recall with No Averaging = " + str(reNoAgg))Recall with No Averaging = tensor([0.8444, 0.7010, 0.9524, 0.8925, 0.8434, 0.9286, 0.8667, 0.6500, 0.7079])preMicro = tm.functional.precision(pred, target, task='multiclass', average = "micro", num_classes=9)
preMacro = tm.functional.precision(pred, target, task='multiclass', average = "macro", num_classes=9)
preNoAgg = tm.functional.precision(pred, target, task='multiclass', average = "none", num_classes=9)
print("Class-Aggregated Precision with Micro Averaging = " + str(preMicro))Class-Aggregated Precision with Micro Averaging = tensor(0.8028)print("Class-Aggregated Precision with Macro Averaging = " + str(preMacro))Class-Aggregated Precision with Macro Averaging = tensor(0.7999)print("Precision with No Averaging = " + str(preNoAgg))Precision with No Averaging = tensor([0.9048, 0.8947, 0.5714, 0.7757, 0.7447, 1.0000, 0.8298, 0.5909, 0.8873])4.5.3 Regression Metrics
The example below demonstrates calculating root mean square error (RMSE) and R-squared for a regression problem. RMSE is calculated by using the mean_square_error() function and setting the squared parameter to False. To obtain MSE, you would set the squared parameter equal to True.
target = torch.tensor(r1['truth'].to_numpy())
pred = torch.tensor(r1['m1'].to_numpy())rmse = tm.functional.mean_squared_error(pred, target, squared=False)
r2 = tm.functional.r2_score(pred, target)
print(f'RMSE = {rmse:.3f}, R-Squared: {r2:.3f}')RMSE = 0.370, R-Squared: 0.7714.6 Concluding Remarks
The goal of this module was to provide an introduction to using and calculating loss and assessment metrics. I used PyTorch to calculate loss metrics and demonstrated how to create custom metrics by subclassing nn.Module. For assessment metrics, I demonstrated the torchmetrics module. You will see example implementations of loss and assessment metrics within the training loop throughout the remainder of this course. We will also apply the loss and assessment metrics to the training set and a withheld validation set, which is consider best practice. We will generally assess the final model using a third testing set.
In the next section, we will discuss preparing data for input into a modeling process using the DataSet and DataLoader classes.
4.7 Questions
- What two properties must all loss metrics have? Why must a measure of error, as opposed to a measure of accuracy, be used as a loss metric?
- Compare the mean square error (MSE), mean absolute error (MAE), Huber, and log-cosh losses in regard to how strongly they penalize samples with large residuals. In what situation might the Huber loss be preferred to MSE, and why must its \(\delta\) parameter be selected relative to the scale of the target variable?
- Explain how binary cross entropy (BCE) loss is calculated. Does the
nn.BCELoss()implementation expect logits or probabilities? - As implemented in PyTorch, explain the relationship between
nn.CrossEntropyLoss()andnn.NLLLoss(). Which of these expects raw logits? - Explain how the Dice loss is calculated. Why are predicted probabilities, as opposed to “hard” counts of true positives, false positives, and false negatives, used in the calculation? Why is the Dice loss less impacted by class imbalance than BCE loss?
- Explain the purpose of the alpha and beta terms in the Tversky loss and the gamma term in the focal Tversky loss.
- Explain the difference between macro-averaging and micro-averaging when aggregating class-level metrics to a single metric. Which method is more impacted by class imbalance, and why?
- Why should assessment metrics be calculated using withheld data as opposed to the training data? Explain the roles of the training, validation, and testing data partitions.
4.8 Exercises
Exercise 4.1: Classification Losses
Introduction
This exercise explores loss metrics for binary and multiclass classification problems. You will calculate the losses manually in Excel and then check your work using the loss functions implemented in PyTorch.
Deliverables
- A Quarto (.qmd) file containing your answers to the background questions and the Python code for the PyTorch tasks, and the HTML webpage rendered from it
- Edited Excel spreadsheet
Data
The data for this exercise are provided in the data/files/lossMetrics folder.
binaryClassification.csv: Table of class labels and predictions for a binary classification problem. These data are synthetic and do not represent a real classification problem.
| Variable | Description |
|---|---|
| sampleID | Unique ID for each sample |
| refLabel | Reference class label |
| refCode | Reference numeric code |
| predLogit | Predicted class logit |
multiclassPrediction.csv: Table of class labels and predictions for a 4-class problem. These data are synthetic and do not represent a real classification problem.
| Variable | Description |
|---|---|
| sampleID | Unique ID for each sample |
| refLabel | Reference class label |
| refCode | Reference numeric code |
| logitA | Predicted logit for Class A |
| logitB | Predicted logit for Class B |
| LogitC | Predicted logit for Class C |
| logitD | Predicted logit for Class D |
| predCode | Predicted numeric class code |
| predLabel | Predicted class label |
Background Questions
Question 1: Write the formula for binary cross entropy loss. Explain each term in the equation.
Question 2: Write the formula for cross entropy loss. Explain each term in the equation.
Question 3: Explain the difference between a logit and a class probability.
Question 4: A binary classification can be framed as a two-class problem (e.g., “presence” and “absence”) or a one-class problem (e.g., “presence”). Explain how this decision impacts (1) the design of the neural network architecture (specifically, the final output) and (2) the appropriate loss metric to apply.
Question 5: As implemented in PyTorch, explain the difference between negative log likelihood loss and cross entropy loss.
Tasks
Excel
Perform the following calculations in Excel.
Task 1: For the binary classification results, convert the raw logits to estimated class probabilities relative to the positive case.
Task 2: Use the estimated positive class probability for each sample to calculate the binary cross entropy loss for the classification.
Task 3: For the multiclass classification, convert the class logits to estimated probabilities for each class.
Task 4: Use the class probabilities and class labels to calculate the cross entropy loss for the classification. Note that you will need to generate one-hot encodings for the correct class labels.
PyTorch
Perform the following calculations using Python and PyTorch.
Task 5: Convert the data into PyTorch tensors and edit them as necessary. Calculate the binary cross entropy loss for the binary classification and the cross entropy loss for the multiclass classification using the loss functions implemented in PyTorch to check your work in Excel.
Exercise 4.2: Regression Losses
Introduction
This exercise explores loss metrics for regression problems. You will calculate the losses manually in Excel and then check your work using the loss functions implemented in PyTorch.
Deliverables
- A Quarto (.qmd) file containing your answers to the background questions and the Python code for the PyTorch tasks, and the HTML webpage rendered from it
- Edited Excel spreadsheet
Data
The data for this exercise are provided in the data/files/lossMetrics folder.
regressionProblem.csv: Table of reference and predicted values for a regression problem. These data are synthetic.
| Variable | Description |
|---|---|
| sampleID | Unique ID for each sample |
| reference | Actual value |
| predicted | Predicted value |
Background Questions
Question 1: Explain the difference between mean square error (MSE) and mean absolute error (MAE) regression losses in regard to the relative weight applied to samples with a large prediction error or residual.
Tasks
Excel
Perform the following calculations in Excel.
Task 1: Calculate the MSE loss.
Task 2: Calculate the MAE loss.
Task 3: Calculate the Huber loss.
Task 4: Calculate the log-cosh loss.
PyTorch
Perform the following calculations using Python and PyTorch.
Task 5: Convert the data into PyTorch tensors and edit them as necessary. Calculate the MSE and MAE losses using the PyTorch implementations to check your Excel-based calculations.
Exercise 4.3: Classification Accuracy Assessment
Introduction
This exercise explores assessment metrics for binary and multiclass classification problems. You will calculate binary classification metrics manually in Excel and then calculate binary and multiclass metrics using torchmetrics.
Deliverables
- A Quarto (.qmd) file containing your answers to the background questions and the Python code for the PyTorch tasks, and the HTML webpage rendered from it
- Edited Excel spreadsheet
Data
This exercise uses the same binaryClassification.csv and multiclassPrediction.csv files that were used in Exercise 4.1, which are provided in the data/files/lossMetrics folder. See Exercise 4.1 for descriptions of the columns in each table.
Background Questions
Question 1: State the equations for recall and precision relative to true positive (TP), false negative (FN), and false positive (FP) counts. Explain how these measures relate to commission and omission errors.
Question 2: State the equations for specificity and negative predictive value relative to true negative (TN), false negative (FN), and false positive (FP) counts. Explain how these measures relate to commission and omission errors.
Question 3: Explain the difference between macro- and micro-averaging for aggregating class-level metrics to a single metric.
Tasks
Excel
Perform the following calculations in Excel.
Task 1: For the binary classification results, count the number of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) samples when samples with a predicted positive case probability larger than 0.5 are mapped to the positive case.
Task 2: Use the counts from Task 1 to calculate overall accuracy, precision, recall, F1-score, specificity, and negative predictive value.
PyTorch
Perform the following calculations using Python and PyTorch.
Task 3: Use torchmetrics to calculate the following metrics for the binary classification: precision, recall, F1-score, overall accuracy, area under the precision-recall curve, and area under the receiver operating characteristic curve.
Task 4: Use torchmetrics to calculate the following aggregated metrics for the multiclass classification: macro-averaged recall, precision, and F1-score; micro-averaged recall, precision, and F1-score; and overall accuracy.
Task 5: Use torchmetrics to calculate the following class-level metrics for each class in the multiclass classification: precision, recall, and F1-score.