peerannot

–Handling your crowdsourced datasets easily–

The peerannot library was created to handle crowdsourced labels in classification problems.

Getting started

Start here to get up and running

• Get started

Tutorials and additional examples

Want to dive deeper into the library? Check out the tutorials You will find resources to add your own datasets, strategy, and run your first label aggregations.

• Tutorials

• More examples can be found in the published paper in Computo Journal

• An interactive tool to compare the AUM and WAUM identification scores is available here

API and CLI Reference

Want to deep dive into the library? In addition to the tutorials, you can find the full API and CLI reference here.

Run peerannot from a python script

API Reference

API Reference

Run peerannot from your terminal

CLI Reference

CLI interface

Glossary

Name	Definition	Mathematical Definition
$n_{task}$	The total number of tasks in a dataset
$n_{worker}$	The total number of workers in a dataset
$[K]$	The set of labels a task can take	$[K]=\{1,...,K\}$
${\mathrm{\Delta }}_K$	The simplex of dimension $K-1$, used to represent soft labels (ie. labels as a probability vector along $[K]$)	${\mathrm{\Delta }}_K=\{p\in [K]:\sum _{k=1}^K{p}_k=1,p_k\ge 0\}$
$\mathcal{A}\mathcal{(}{\mathcal{x}}_{\mathcal{i}}\mathcal{)}$	The set of workers that answered the task $i$	$\{j\in [n_{worker}:w_j\text{ answered }{x}_i\}$
$\mathcal{T}\mathcal{(}{\mathcal{w}}_{\mathcal{j}}\mathcal{)}$	The set of tasks answered by the worker $j$	$\{i\in [n_{task}:w_j\text{ answered }{x}_i\}$
$\mathcal{Lab}\mathcal{(}{\mathcal{x}}_{\mathcal{i}}\mathcal{)}$	The vector of answered labels of the task $i$	$(y_i^{(j)}{)}_{j\in \mathcal{A}\mathcal{(}{\mathcal{x}}_{\mathcal{i}}\mathcal{)}}$
$y_i^{\ast }$	The true label of the task $i$	$y_i^{\ast }\in [K]$
${\hat{y}}_i^{agg}$	The computed label of the task $i$ given the aggregation $agg$ method	$\{\begin{array}{l}{\hat{y}}_i^{agg}\in [K]\text{ if a hard label}\\ {\hat{y}}_i^{agg}\in {\mathrm{\Delta }}_K\text{ if a soft label}\end{array}$
$y_i^{(j)}$	The label (hard) that the worker $j$ assigned to the task $i$
${\pi }^{(j)}$	The confusion matrix of the worker $j$	${\pi }_{k,\ell }^{(j)}=\mathbb{P}(y_i^{(j)}=\ell \mid y_i^{\star }=k),\mathrm{\forall }(\ell ,k)\in [K{]}^2$
$AccTrain(\mathcal{D})$	A metric that measure aggregation strategies’ accuracies	$AccTrain(\mathcal{D})=\frac{1}{|\mathcal{D}|}\sum _{i=1}^{|\mathcal{D}|}{\mathbf{1}}_{\{y_i=\underset{k\in [K]}{\mathrm{argmax}}({ŷ}_i{)}_k\}}$

Citation

Cite us, join us, and let us collaboratively improve our toolbox!

@article{lefort2024,
   author = {Lefort, Tanguy and Charlier, Benjamin and Joly, Alexis and Salmon, Joseph},
   publisher = {French Statistical Society},
   title = {Peerannot: Classification for Crowdsourced Image Datasets with {Python}},
   journal = {Computo},
   date = {2024-04-04},
   url = {https://computo.sfds.asso.fr/published-202402-lefort-peerannot/},
}