Crowdlayer

class Crowdlayer(tasks_path, answers, model, n_classes, optimizer, n_epochs, scale=0, verbose=True, pretrained=False, output_name='crowdlayer', **kwargs)

Crowdlayer (Rodrigues et. al 2018)

End-to-end learning strategy with multiple votes per task

Using: - Crowd layer added to network

Code: - Tensorflow original code available at https://github.com/fmpr/CrowdLayer - Code adaptated in Python

__init__(tasks_path, answers, model, n_classes, optimizer, n_epochs, scale=0, verbose=True, pretrained=False, output_name='crowdlayer', **kwargs)

CrowdLayer deep learning strategy. Learn a classifier with crowdsourced labels by modeling worker-specific confusions.

During training, given the classifier \(\mathcal{C}\) with output scores \(z_i\), local confusions \(\pi^{(j)}\), the model computes the following:

\[h_i^{(j)} = \sigma((\pi^{(j)})\sigma (z_i )),\]

The final loss is the crossentropy between the worker-specific prediction and the given label.

\[\mathcal{L} = \frac{1}{n_{\texttt{task}}}\sum_{i=1}^{n_{\texttt{task}}}\sum_{j\in\mathcal{A}(x_i)} \mathrm{CE}(h_i^{(j)}, y_i^{(j)}).\]
Parameters:

  • tasks_path (path) – Path to images to train from

  • answers (path) –

    Path to answers (json format)

    {
    task0: {worker0: label, worker1: label},
    task1: {worker1: label}
}

  • model (string) – Backbone classifier architecture: should be from torchvision.models

  • n_classes (int) – Number of classes

  • optimizer (string) – Pytorch optimizer name (either sgd or Adam)

  • n_epochs (int) – Number of training epochs

  • scale (float) – Regularization parameter in the loss, defaults to 1e-5 (Regularization not implemented yet)

  • verbose (bool, optional) – Verbosity level, defaults to True

  • pretrained (bool, optional) – Use the pretrained version of the backbone classifier, defaults to False

  • output_name (str, optional) – Generated file prefix, defaults to “conal”

The batch size, learning rate, scheduler and milestones can be specified as keyword arguments. Visit the computo paper or the tutorial for examples.

setup(**kwargs)

Create training, validation and test dataloaders from the dataset.

run(**kwargs)

Train the CrowdLayer model and evaluate on the test set.

Uses a gpu by default is torch.cuda.is_available() is True.

Results are stored in <tasks_path>/results/<output_name>.json and the best model in <tasks_path>/best_models/<output_name>.pth. Results contain the train, validation and test loss as well as the validation and test accuracy.

run_epoch(model, trainloader, criterion, optimizer, optimizer2, logger)

Run one epoch and monitor metrics