Evaluation metrics

All metrics compare a detected label cube \(P\) with a true label cube \(G\) (shape time × y × x, 0 = no event). Cells are weighted by their area \(w\), so on a latitude–longitude grid the scores are area-weighted.

Space–time overlap (IoU)

For a true event \(g\) and a detected event \(p\),

\[\mathrm{IoU}(g,p) = \frac{\sum_{v \in g \cap p} w_v}{\sum_{v \in g \cup p} w_v}.\]

Object matching — driftdrought.metrics.object_match()

True and detected events are paired one-to-one by the Hungarian algorithm on the IoU matrix. A pair is a true positive when its IoU reaches the threshold (0.5 by default; f1_iou25 uses 0.25):

\[\mathrm{precision} = \frac{TP}{N_P}, \qquad \mathrm{recall} = \frac{TP}{N_G}, \qquad F_1 = \frac{2\,\mathrm{precision}\cdot\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}.\]

It also returns the mean IoU of the matches and the relative volume error of matched events, \(|V_p - V_g| / V_g\). The number of detected events (n_pred) against true events (n_gt) exposes fragmentation (too many events) and chaining (too few) that a single score can hide.

HOTA — driftdrought.metrics.hota()

Higher Order Tracking Accuracy (Luiten et al., 2021) balances detection and association. Every time step is a frame and every event slice a detection. For a localization threshold \(\alpha\), slices are matched per frame (Hungarian, IoU \(\ge \alpha\)) into true positives (TP), false negatives (FN) and false positives (FP):

\[\mathrm{DetA}_\alpha = \frac{|TP|}{|TP| + |FN| + |FP|}, \qquad \mathrm{AssA}_\alpha = \frac{1}{|TP|}\sum_{c \in TP} \frac{|TPA(c)|}{|TPA(c)| + |FNA(c)| + |FPA(c)|},\]

where, for a matched pair \(c=(g,p)\), \(TPA(c)\) counts the frames in which \(g\) and \(p\) are matched, \(FNA(c)\) the frames in which \(g\) exists but is not matched to \(p\), and \(FPA(c)\) the frames in which \(p\) exists but is not matched to \(g\). Then

\[\mathrm{HOTA}_\alpha = \sqrt{\mathrm{DetA}_\alpha\,\mathrm{AssA}_\alpha}, \qquad \mathrm{HOTA} = \frac{1}{19}\sum_{\alpha=0.05}^{0.95} \mathrm{HOTA}_\alpha .\]

Reading the scores. DetA is high when drought cells are found in the right months; AssA is high when each true drought keeps one identity through time (no fragmentation, no chaining of separate droughts). A detector that merges two droughts into one long object can have a high DetA and a low AssA; one that breaks a drought into monthly pieces has the opposite.

Partition agreement — driftdrought.metrics.partition_agreement()

Over the cells labelled by either cube, the adjusted Rand index (ARI; 1 = identical partitions, about 0 = random) and the variation of information,

\[\mathrm{VI}(G,P) = H(G) + H(P) - 2\,I(G;P) \quad \text{(bits; 0 = identical)}.\]

Event confidence and significance

These are attributes of each detected event, not scores against a truth; see Method and the driftdrought.confidence and driftdrought.significance modules.