Trace_confusion

class Trace_confusion(answers, **kwargs)

Use Dawid and Skene confusion matrices to obtain a scalar indicator of the worker’s confusion

__init__(answers, **kwargs)

Get the trace of the confusion matrices for each worker. The closer to K (the number of classes), the better the worker.

\[\mathrm{Tr}_j = \sum_{k=1}^{K} \pi_{k,k}^{(j)}\]

Parameters:

answers (dict) –

Dictionary of workers answers with format

{
    task0: {worker0: label, worker1: label},
    task1: {worker1: label}
}

The number of classes n_{classes} and the number of workers n_{workers} should be specified as keyword argument. If the matrices are known and stored in a npy or pth file, it can be specified as matrix_file. Otherwise, the model will run the DS model to obtain the matrices.

run(path)

From workers confusion matrices, compute the trace and save it.

Parameters:

path (str) – path to save the results <path>/identification/trace_confusion.npy