Dawid and Skene

Model

The Dawid and Skene’s model estimates one confusion matrix $\pi\in\mathbb{R}^{K\times K}$ per worker. The model assumes that the probability for a task $x_i$ to have true label $y_i^\star=k$ follows a multinomial distribution with probabilities $\pi^{(j)}_{k,\bullet}$ for each worker.

The likelihood to maximize is:

\[\displaystyle\prod_{i\in [n_{\texttt{task}}]}\prod_{k \in [K]}\bigg[\rho_k\prod_{j\in [n_{\texttt{worker}}]} \prod_{\ell\in [K]}\big(\pi^{(j)}_{k, \ell}\big)^{\unicode{x1D7D9}_{\{y_i^{(j)}=\ell\}}} \bigg]^{T_{ik}},\]

with $\rho_k=\mathbb{P}(y_i^\star=k)$ and $T_{i,k}=\unicode{x1D7D9}_{\{y_i^{\star}=k \}}.$

CLI

With peerannot in a terminal located in the directory of answers, the DS model can be used as follows.

peerannnot aggregate . --strategy DS --answers-file answers.json

Note that by default, if the answers are in a file names answers.json the --answers-file argument can be omitted.

API

Import the aggregation model in the current session

from peerannot.models import Dawid_Skene as DS

Assuming the answers are in a dictionary names answers then run:

ds = DS(answers, n_workers=n_workers, n_classes=n_classes)
ds.run()
yhat = ds.get_probas()

Estimated abilities

To access the estimated confusion matrices in a variable pi, run:

pi = ds.pi
print(pi.shape)
# (n_worker, n_classes, n_classes)

m_step()

Maximizing log likelihood (see eq. 2.3 and 2.4 Dawid and Skene 1979)

Returns (as attributes):

  • p: (p_j)_j probabilities that instance has true response j if drawn at random (class marginals)
  • pi: number of times worker k records l when j is correct

e_step()

Estimate indicator variables (see eq. 2.5 Dawid and Skene 1979)

Returns (as attributes):

  • T: New estimate for indicator variables (n_task, n_worker)
  • denom: value used to compute likelihood easily

log_likelihood()

Computes the logarithm of the likelihood.

run(epsilon=1e-6, maxiter=50, verbose=False)

Run the EM algorithm

Parameters:

  • epsilon:(float) stopping value for the absolute difference between the log likelihood of two consecutive steps
  • maxiter:(int) Total number of steps in the EM algorithm
  • verbose:(bool) Print if the EM algorithm did not converge.

Aggregate into hard labels

After running the aggregation strategy, instead of soft labels one can recover hard labels by running:

yhat_hard = df.get_answers()

Note that this is an argmax on the first dimension with a random split in case of equalities.

API details: class models.Dawid_Skene

Dawid and Skene model class that herits from CrowdModel

__init__(answers, n_classes,**kwargs)

Parameters:

  • answers:(dict) Dictionnary of workers answers with format 
    
          {
              task0: {worker0: label, worker1: label},
              task1: {worker1: label}
          }
  • n_classes: (int) Number of possible classes
  • kwargs: (dict) Dictionnary that should contain at least $n_{\texttt{worker}}$ the number of workers. Other arguments are path_remove to remove tasks identified from the WAUM or another method.

get_crowd_matrix()

From the dictionnary of answers generates a tensor of size $(n_{\texttt{task}},n_{\texttt{worker}},n_{\texttt{class}})$ with $0$ where there is no collected annotation.

init_T()

Empirical distribution initialization for the variables $T_{i,k}$