Dataset with subgroups and no labels

[PabloLeon]

Hi there! I have a slightly unusual dataset where several raters grouped items together. That is, labels between raters do not correspond to each other, and we only have information about which items an individual rater deemed the same/different. For example, say rater 1 puts items a and b in group 1, and item c in group 2, and rater 2 puts items b and c in group 1: we would not want to consider group 1 across raters the same grouping. So far, I have used my own implementation of contrastive learning, but I would like to use this library to try more sophisticated techniques like mining or different losses.
What would you recommend as the most convenient approach to using this library in my setting? My plan so far was to accumulate the gradients across rater batches and mine items within raters? Alternatively, I could relabel the groups across raters or extend the loss classes?
Any advice would be greatly appreciated!

[KevinMusgrave]

In your example, are the positive pairs (a,b) and (b,c), and the negative pairs (a,c) and (b,c) ?

[PabloLeon]

Yes, in the example those are the pairs.

[KevinMusgrave]

If you don't need to use a mining function, then you can create arrays of indices that describe the positive and negative pairs. For your example:

embeddings = torch.cat([a,b,c], dim=0)
pos_pairs = [(0, 1), (1, 2)] # (a,b) and (b,c)
neg_pairs = [(0, 2), (1, 2)] # (a,c) and (b,c)

a1, p = zip(*pos_pairs) # a1 has all positive anchors, p has all positives
a2, n = zip(*neg_pairs) # a2 has all negative anchors, n has all negatives
indices_tuple = [torch.tensor(x) for x in [a1, p, a2, n]]

# In current version of code, labels are always required
# But they won't be used if indices_tuple is specified
labels = torch.arange(len(embeddings))
loss = loss_fn(embeddings, labels, indices_tuple=indices_tuple)

If you need to use a mining function, then you'll need labels. But I can't think of a way to generate labels when it's possible for a single pair to be both positive and negative (like (b,c) is in your example.)

There is one miner that can filter existing pairs:

from pytorch_metric_learning.miners import HDCMiner
miner = HDCMiner(filter_percentage=0.25) # will keep top 25% most difficult pairs

# as long as you're passing in pairs, then "labels" won't be used
miner.set_idx_externally((a1, p, a2, n), labels)

# since set_idx_externally was called, labels won't be used
hard_pairs = miner(embeddings, labels)

Another alternative is to rely on reducers instead of a miner:

from pytorch_metric_learning import losses, reducers

# only keep pairs that have a loss greater than 0.2
reducer = reducers.ThresholdReducer(low=0.2)
loss_fn = losses.ContrastiveLoss(reducer=reducer)
loss = loss_fn(embeddings, labels, indices_tuple=indices_tuple)

In the above examples, labels are required, but they won't be used (related issue: #412)

[PabloLeon]

Thank you for this detailed answer! I was thinking that I could use the full range of miners by having each batch deal with the subgroups individually and then accumulating the error/gradients (since I know what the subgroups there are). Both of your suggestions look promising though, and I will check that out. Thanks so much!

[KevinMusgrave]

Starting in v1.5.0, losses like ContrastiveLoss and TripletMarginLoss no longer require labels if indices_tuple is passed in:

loss = loss_fn(embeddings, indices_tuple=triplets)

(Posting here for future readers.)