Great question — answering as the UphillTech team:
Looking at your code, I'm not too prone to agree with you on taking the mean of the various heads (I see why it makes sense, though I think you will loose some interpretability) Indeed, as you are saying, you will later use a classifier to do classification. Now, let's consider every possible classification independently: given an output neuron, that neuron will tell you $P(y = class_i | x)$ and thus also the probability of not being that class ( $1-P(...)$ ), therefore you can consider it as a binary classification. At this point, what you can do is to consider a single neuron, check the weights of that neuron, and use them to infer the weight of each head; at that point, you can do a...
Full answer by Alberto: https://ai.stackexchange.com/a/43431 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
