********** Summary **********
The main update is the new distances module, which adds an extra level of modularity to loss functions. It is a pretty big design change, which is why so many arguments have become obsolete. See [the documentation](https://kevinmusgrave.github.io/pytorch-metric-learning/distances/) for a description of the new module.
Other updates include support for half-precision, new regularizers and mixins, improved documentation, and default values for most initialization parameters.
********** Breaking Changes **********
Dependencies
This library now requires PyTorch >= 1.6.0. Previously there was no explicit version requirement.
Losses and Miners
All loss functions
normalize_embeddings has been removed
- If you never used this argument, nothing needs to be done.
- normalize_embeddings = True: just remove the argument.
- normalize_embeddings = False: remove the argument and instead pass it into a distance object. For example:
python
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(normalize_embeddings=False))
ContrastiveLoss, GenericPairLoss, BatchHardMiner, HDCMiner, PairMarginMiner
use_similarity has been removed
- If you never used this argument, nothing needs to be done.
- use_similarity = True: remove the argument and:
python
if you had set normalize_embeddings = False
from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = ContrastiveLoss(distance=DotProductSimilarity(normalize_embeddings=False))
otherwise
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = ContrastiveLoss(distance=CosineSimilarity())
squared_distances has been removed
- If you never used this argument, nothing needs to be done.
- squared_distances = True: remove the argument and instead pass power=2 into a distance object. For example:
python
from pytorch_metric_learning.distances import LpDistance
loss_func = ContrastiveLoss(distance=LpDistance(power=2))
- squared_distances = False: just remove the argument.
ContrastiveLoss, TripletMarginLoss
power has been removed
- If you never used this argument, nothing needs to be done.
- power = 1: just remove the argument
- power = X, where X != 1: remove the argument and instead pass it into a distance object. For example:
python
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(power=2))
TripletMarginLoss
distance_norm has been removed
- If you never used this argument, nothing needs to be done.
- distance_norm = 2: just remove the argument
- distance_norm = X, where X != 2: remove the argument and instead pass it as p into a distance object. For example:
python
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(distance=LpDistance(p=1))
NPairsLoss
l2_reg_weight has been removed
- If you never used this argument, nothing needs to be done.
- l2_reg_weight = 0: just remove the argument
- l2_reg_weight = X, where X > 0: remove the argument and instead pass in an LpRegularizer and weight:
python
from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = NPairsLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)
SignalToNoiseRatioContrastiveLoss
regularizer_weight has been removed
- If you never used this argument, nothing needs to be done.
- regularizer_weight = 0: just remove the argument
- regularizer_weight = X, where X > 0: remove the argument and instead pass in a ZeroMeanRegularizer and weight:
python
from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = SignalToNoiseRatioContrastiveLoss(embedding_regularizer=ZeroMeanRegularizer(), embedding_reg_weight=0.123)
SoftTripleLoss
reg_weight has been removed
- If you never used this argument, do the following to obtain the same default behavior:
python
from pytorch_metric_learning.regularizers import SparseCentersRegularizer
weight_regularizer = SparseCentersRegularizer(num_classes, centers_per_class)
SoftTripleLoss(..., weight_regularizer=weight_regularizer, weight_reg_weight=0.2)
- reg_weight = X: remove the argument, and use the SparseCenterRegularizer as shown above.
WeightRegularizerMixin and all classification loss functions
- If you never specified regularizer or reg_weight, nothing needs to be done.
- regularizer = X: replace with weight_regularizer = X
- reg_weight = X: replace with weight_reg_weight = X
Classification losses
- For all losses and miners, default values have been set for as many arguments as possible. This has caused a change in ordering in positional arguments for several of the classification losses. The typical form is now:
python
loss_func = SomeClassificatinLoss(num_classes, embedding_loss, <keyword arguments>)
See the documentation for specifics
Reducers
ThresholdReducer
threshold has been replaced by low and high
- Replace threshold = X with low = X
Regularizers
All regularizers
normalize_weights has been removed
- If you never used this argument, nothing needs to be done.
- normalize_weights = True: just remove the argument.
- normalize_weights = False: remove the argument and instead pass normalize_embeddings = False into a distance object. For example:
python
from pytorch_metric_learning.distances import DotProductSimilarity
loss_func = RegularFaceRegularizer(distance=DotProductSimilarity(normalize_embeddings=False))
Inference
MatchFinder
mode has been removed
- Replace mode="sim" with either distance=CosineSimilarity() or distance=DotProductSimilarity()
- Replace mode="dist" with distance=LpDistance()
- Replace mode="squared_dist" with distance=LpDistance(power=2)
********** New Features **********
Distances
Distances bring an additional level of modularity to building loss functions. Here's an example of how they work.
Consider the TripletMarginLoss in its default form:
python
from pytorch_metric_learning.losses import TripletMarginLoss
loss_func = TripletMarginLoss(margin=0.2)
This loss function attempts to minimize [d<sub>ap</sub> - d<sub>an</sub> + margin]<sub>+</sub>.
In other words, it tries to make the anchor-positive distances (d<sub>ap</sub>) smaller than the anchor-negative distances (d<sub>an</sub>).
Typically, d<sub>ap</sub> and d<sub>an</sub> represent Euclidean or L2 distances. But what if we want to use a squared L2 distance, or an unnormalized L1 distance, or completely different distance measure like signal-to-noise ratio? With the distances module, you can try out these ideas easily:
python
TripletMarginLoss with squared L2 distance
from pytorch_metric_learning.distances import LpDistance
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(power=2))
TripletMarginLoss with unnormalized L1 distance
loss_func = TripletMarginLoss(margin=0.2, distance=LpDistance(normalize_embeddings=False, p=1))
TripletMarginLoss with signal-to-noise ratio
from pytorch_metric_learning.distances import SNRDistance
loss_func = TripletMarginLoss(margin=0.2, distance=SNRDistance())
You can also use similarity measures rather than distances, and the loss function will make the necessary adjustments:
python
TripletMarginLoss with cosine similarity
from pytorch_metric_learning.distances import CosineSimilarity
loss_func = TripletMarginLoss(margin=0.2, distance=CosineSimilarity())
With a similarity measure, the TripletMarginLoss internally swaps the anchor-positive and anchor-negative terms: [s<sub>an</sub> - s<sub>ap</sub> + margin]<sub>+</sub>. In other words, it will try to make the anchor-negative similarities smaller than the anchor-positive similarities.
All **losses, miners, and regularizers** accept a distance argument. So you can try out the MultiSimilarityMiner using SNRDistance, or the NTXentLoss using LpDistance(p=1) and so on. Note that some losses/miners/regularizers have restrictions on the type of distances they can accept. For example, some classification losses only allow CosineSimilarity or DotProductSimilarity as their distance measure between embeddings and weights. To view restrictions for specific loss functions, see [the documentation](https://kevinmusgrave.github.io/pytorch-metric-learning/losses/)
There are four distances implemented (LpDistance, SNRDistance, CosineSimilarity, DotProductSimilarity), but of course you can extend the BaseDistance class and write a custom distance measure if you want. See [the documentation](https://kevinmusgrave.github.io/pytorch-metric-learning/distances/) for more.
EmbeddingRegularizerMixin
All loss functions now extend EmbeddingRegularizerMixin, which means you can optionally pass in (to any loss function) an embedding regularizer and its weight. The embedding regularizer will compute some loss based on the embeddings alone, ignoring labels and tuples. For example:
python
from pytorch_metric_learning.regularizers import LpRegularizer
loss_func = MultiSimilarityLoss(embedding_regularizer=LpRegularizer(), embedding_reg_weight=0.123)
WeightRegularizerMixin is now a subclass of WeightMixin
As in previous versions, classification losses extend WeightRegularizerMixin, which which means you can optionally pass in a weight matrix regularizer. Now that WeightRegularizerMixin extends WeightMixin, you can also specify the weight initialization function [in object form](https://kevinmusgrave.github.io/pytorch-metric-learning/common_functions/#torchinitwrapper):
python
from ..utils import common_functions as c_f
import torch
use kaiming_uniform, with a=1 and mode='fan_out'
weight_init_func = c_f.TorchInitWrapper(torch.nn.kaiming_uniform_, a=1, mode='fan_out')
loss_func = SomeClassificationLoss(..., weight_init_func=weight_init_func)
New Regularizers
For increased modularity, the regularizers hard-coded in several loss functions were separated into their own classes. The new regularizers are:
- LpRegularizer
- SparseCentersRegularizer
- ZeroMeanRegularizer
Support for half-precision
In previous versions, various functions would break in half-precision (float16) mode. Now all distances, losses, miners, regularizers, and reducers work with half-precision, float32, and double (float64).
New collect_stats argument
All distances, losses, miners, regularizers, and reducers now have a collect_stats argument, which is True by default. This means that various statistics are collected in each forward pass, and these statistics can be useful to look at during experiments. However, if you don't care about collecting stats, you can set collect_stats=False, and the stat computations will be skipped.
Other updates
- You no longer have to explicitly call .to(device) on classification losses, because their weight matrices will be moved to the correct device during the forward pass if necessary. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/139
- Reasonable default values have been set for all losses and miners, to make these classes easier to try out. In addition, equations have been added to many of the class descriptions in the documentation. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/140
- Calls to torch.nonzero have been replaced by torch.where.
- The documentation for ArcFaceLoss and CosFaceLoss have been fixed to reflect the actual usage. (The documentation previously indicated that some arguments are positional, when they are actually keyword arguments.)
- The tensorboard_folder argument for utils.logging_presets.get_record_keeper is now optional. If you don't specify it, then there will be no tensorboard logs, which can be useful if speed is a concern.
- The loss dictionary in BaseTrainer is now cleared at the end of each epoch, to free up GPU memory. See issue https://github.com/KevinMusgrave/pytorch-metric-learning/issues/171