API identify

The identification has three levels of exploration:

• dataset: measure the reliability of the collected votes

• per worker: measure the reliability of the worker

• per task: measure the reliability of the taks

All strategies are available running:

peerannot identificationinfo

We created an interactive tool to compare the WAUM and AUM identifications on the CIFAR-10H dataset. Check it out by clicking this link!

Dataset exploration

krippendorff_alpha.Krippendorff_Alpha

Per worker exploration

Spam_score.Spam_Score	Spammer score (Raykar and Yu, 2011)
trace_confusion.Trace_confusion	Use Dawid and Skene confusion matrices to obtain a scalar indicator of the worker's confusion

Per task exploration

AUM.AUM	AUM (Pleiss et. al, 2020)
entropy.Entropy	Computes the votes's distribution entropy per task.
WAUM_perworker.WAUM_perworker	WAUM per worker (Lefort et al. 2024 in TMLR)
WAUM.WAUM	WAUM (Lefort et al., 2024 in TMLR)