Graphical representation of each method. For simplicity, each panel shows the first self-attention block, from the input representation (F F(x*) + P E(x*)) to the output of multi-head attention. Panel a shows the DEs approach, which relies on standard multi-head attention; attention scores α_i are derived from the key (K) and query (Q) matrices (Eq. (2)), which are used to calculate the output h_i (Eq. (4)). Panel b shows MC Dropout, which activates dropout both after the input and within the attention blocks (dashed blue lines). As a result, the query (Q), key (K) and value (V) matrices change with each forward pass. Panel c corresponds to the HSA approach, which introduces a hierarchical component, specifically, the key representation is redefined using a set of centroids (C) to produce a new key matrix ($\tilde{\mathit{K}}$); attention weights (${\tilde{\mathit{\alpha }}}_i,{\hat{\mathit{\alpha }}}_i$) are sampled from a Gumbel-softmax distribution, introducing stochasticity into the attention computation. Panel d corresponds to the HA-MC Dropout approach, which combines dropout (as in b) and the hierarchical component (as in c with no sampling from a Gumbel-softmax distribution); this configuration retains the hierarchical component and applies dropout within the attention block, resulting in varying attention weights across forward passes.