Peerannot library

The peerannot library was created to handle crowdsourced labels in classification problems.

Installation

To install the peerannot library and reproduce results, use directory of the setup.py file and then

$ pip install peerannot
  

Installing the library gives access to the Command Line Interface using the keyword peerannot in a bash terminal.

Example

We demonstrate how peerannot works with the cifar10H dataset. We assume that the current working directory is the cifar10H directory containing the cifar10H.py file.

First, install the dataset with

$ peerannot install ./cifar10h.py
  

Then, we can try classical label aggregation strategies as follows:

for strat in MV NaiveSoft DS GLAD WDS
  do
  echo "Strategy: ${strat}"
  peerannot aggregate . -s $strat
  done
  

This will create a new folder names labels containing the labels in the labels_cifar10H_${strat}.npy file. Once the labels are available, we can train a neural network with PyTorch as follows. In a terminal:

for strat in MV NaiveSoft DS GLAD WDS
  do
  echo "Strategy: ${strat}"
  declare -l strat
  strat=$strat
  peerannot train . -o cifar10H_${strat} \
                    -K 10 \
                    --labels=./labels/labels_cifar-10h_${strat}.npy \
                    --model resnet18 \
                    --img-size=32 \
                    --n-epochs=1000 \
                    --lr=0.1 --scheduler=multistep -m 100 -m 250 \
                    --num-workers=8
  done
  

As the WAUM purpose is to identify ambiguous tasks, the command to run the identification is:

$ peerannot identify . -K 10 --method WAUM \
                       --labels ./answers.json \
                       --model resnet18 --n-epochs 50 \
                       --lr=0.1 --img-size=32 \
                       --maxiter-DS=50 \
                       --alpha=0.01
  

Then, one can train on the pruned dataset with any aggregation strategy as follows:

# run WAUM + DS strategy
$ peerannot train . -o cifar10H_waum_0.01_DS \
                  -K 10 \
                  --labels= ./labels/labels_cifar-10h_ds.npy \
                  --model resnet18 --img-size=32 \
                  --n-epochs=150 --lr=0.1 -m 50 -m 100 --scheduler=multistep \
                  --num-workers=8 \
                  --path-remove ./identification/waum_0.01_yang/too_hard_0.01.txt
  

Finally, for the end-to-end strategies using deep learning (as CoNAL or CrowdLayer), the command line is:

$ peerannot aggregate-deep . -o cifar10h_crowdlayer \
                           --answers ./answers.json \
                           --model resnet18 -K=10 \
                           --n-epochs 150 --lr 0.1 --optimizer sgd \
                           --batch-size 64 --num-workers 8 \
                           --img-size=32 \
                           -s crowdlayer
  

For CoNAL, the hyperparameter scaling can be provided as -s CoNAL[scale=1e-4].

Peerannot and crowdsourcing formatting

In peerannot, one of our goals is to make crowdsourced datasets under the same format so that it is easy to switch from one learning or aggregation strategy without having to code once again the algorithms for each dataset.

So, what is a crowdsourced dataset? We define each dataset as:

dataset
      ├── train
      │     ├── class0
      │     │     ├─ task0-<vote_index_0>.png
      │     │     ├─ task1-<vote_index_1>.png
      │     │     ├─ ...
      │     │     └─ taskn0-<vote_index_n0>.png
      │     ├── class1
      │     ├── ...
      │     └── classK
      ├── val
      ├── test
      ├── dataset.py
      ├── metadata.json
      └── answers.json
  

The crowdsourced labels for each training task are contained in the anwers.json file. They are formatted as follows:

{
      0: {<worker_id>: <label>, <another_worker_id>: <label>},
      1: {<yet_another_worker_id>: <label>,}
  }
  

Note that the task index in the answers.json file might not match the order of tasks in the train folder... Thence, each task's name contains the associated votes file index. The number of tasks in the train folder must match the number of entry keys in the answers.json file.

The metadata.json file contains general information about the dataset. A minimal example would be:

{
      "name": <dataset>,
      "n_classes": K,
      "n_workers": <n_workers>,
  }
  

The dataset.py is not mandatory but is here to facilitate the dataset's installation procedure. A minimal example:

class mydataset:
      def __init__(self):
          self.DIR = Path(__file__).parent.resolve()
          # download the data needed
          # ...

      def setfolders(self):
          print(f"Loading data folders at {self.DIR}")
          train_path = self.DIR / "train"
          test_path = self.DIR / "test"
          valid_path = self.DIR / "val"

          # Create train/val/test tasks with matching index
          # ...

          print("Created:")
          for set, path in zip(
              ("train", "val", "test"), [train_path, valid_path, test_path]
          ):
              print(f"- {set}: {path}")
          self.get_crowd_labels()
          print(f"Train crowd labels are in {self.DIR / 'answers.json'}")

      def get_crowd_labels(self):
          # create answers.json dictionnary in presented format
          # ...
          with open(self.DIR / "answers.json", "w") as answ:
              json.dump(dictionnary, answ, ensure_ascii=False, indent=3)