$\color{black}\rule{365px}{3px}$

1. Contrastive Loss

$\color{black}\rule{365px}{3px}$

A metric-learning loss function used in machine learning, particularly in tasks like face verification, image retrieval, and Siamese neural networks. Its primary purpose is to learn a feature space where similar samples are pulled closer together and dissimilar samples are pushed further apart. The loss is designed to enforce a contrast between "positive pairs" (similar samples) and "negative pairs" (dissimilar samples).

Formulation

For a pair of input samples $(x_1,x_2)$, their corresponding labels $y$ indicate whether the pair is similar $(y=1)$ or dissimilar $(y=0)$. Let $D$ be the distance between their embeddings in the feature space (usually computed as the Euclidean distance or cosine similarity).

$$ L=(1−y)⋅max(0,m−D)^2+y⋅D^2 $$

Where:

$D$: Distance between the embeddings of $x_1$ and $x_2$ (e.g., $D=∥f(x_1)−f(x_2)∥$).
$y$: Binary label ($1$ if similar, $0$ if dissimilar).
$m$: Margin that defines the minimum distance required between embeddings of dissimilar pairs. If the distance $D$ is greater than $m$, the loss for the dissimilar pair becomes zero.

IF Positive Pairs $(y=1)$

The term $y⋅D^2$ minimizes the distance $D$ between embeddings of similar pairs, encouraging them to be closer in the feature space.

</aside>

IF Negaive Pairs $(y=0)$

The term $(1−y)⋅max(0,m−D)^2$ penalizes embeddings of dissimilar pairs only when their distance $D$ is less than the margin $m$. This encourages dissimilar pairs to be at least $m$ units apart.

When $D<m$:
- The dissimilar pairs are "too close" in the feature space, which violates the margin constraint.
- This leads to a non-zero loss, $(m−D)^2$, to push the embeddings further apart.
When $D≥m$:
- The dissimilar pairs are sufficiently far apart (beyond the margin $m$).
- In this case, $max(0,m−D)=0$, resulting in zero loss. </aside>