The Honeypot (or gold standard) is a tool to measure the accuracy of annotations. It helps to audit the work of annotators throughout the project. The Honeypot works by interspersing data instances for which you have defined a groundtruth label in the annotation queue so that you can measure the disagreement level between your gold standards and the annotations made by annotators.
The assets used for the Honeypot are given to be annotated to all project annotators. They are intelligently distributed throughout the annotation process.
Activate the Honeypot
You can set the Honeypot in the Settings tab, Quality section, below the consensus. Check the box.
Set the Honeypot
The creation of the Honeypot is done either through the API or the Review interface.
- API: by uploading assets with their golden label
- Review interface: by clicking on the honeypot button, close to Reviewed button. It becomes yellow when activated.
The Honeypot score, expressed as a %, is calculated as follows:
At the label level
For NER: two annotations are equal if they refer to the same class, and that they cover the same portion of text.
For IMAGES: two annotations are equal if they refer to the same class, and that they cover the same pixels.
At the asset level: this is the average of the Honeypots at the label level.
At the annotator level: this is the average of the annotator labels Honeypots.