AUM

class AUM(tasks, n_classes, model, criterion, optimizer, n_epoch, topk=False, verbose=False, use_pleiss=False, **kwargs)

AUM (Pleiss et. al, 2020)

Measures the AUM per task given the ground truth label.

Using:

  • Margin estimation

  • Trust score per task

__init__(tasks, n_classes, model, criterion, optimizer, n_epoch, topk=False, verbose=False, use_pleiss=False, **kwargs)

Compute the AUM score for each task. Given a classifier \(\mathcal{C}\) and tasks \(x_i\) with hard labels from an aggregation \(y_i^\text{agg}\), the AUM writes

\[\mathrm{AUM}(x_i)=\frac{1}{T}\sum_{t=1}^T \left(\sigma(\mathcal{C}(x_i)){y_i^{\text{agg}}} - \sigma(\mathcal{C}(x_i))_{[2]}\right)\]
Parameters:

  • tasks (torch Dataset) – Dataset of tasks as \((x_i, _, y_i^{\text{agg}}, i)_{(i,j)}\)

  • n_classes (int) – Number of possible classes, defaults to 2

  • model (torch Module) – Neural network to use

  • criterion (torch loss) – loss to minimize for the network

  • optimizer (torch optimizer) – Optimization strategy for the minimization

  • n_epoch (int) – Number of epochs (should be the first learning rate scheduler step drop or lower than half the training epochs)

  • verbose (bool, optional) – Print details in log, defaults to False

  • use_pleiss (bool, optional) – Use Pleiss margin instead of Yang, defaults to False

make_step(batch)

One optimization step

Parameters:

batch (batch) –

Batch of tasks

Batch:

  • index 0: tasks \((x_i)_i\)

  • index 1: placeholder

  • index 2: labels

  • index 3: tasks index \((i)_i\)

Returns:

Tuple with length, logits, targets, ground turths and index

Return type:

tuple

get_aum()

Records prediction scores of interest for the AUM during n_epoch training epochs

compute_aum()

Compute the AUM for each task

cut_lowests(alpha=0.01)

Computes the tasks with the lowest AUM scores. The index of such tasks are stored in the .too_hard attribute.

Parameters:

alpha (float, optional) – quantile order to identify as ambiguous, defaults to 0.01

run(alpha=0.01)

Run AUM identification