In regards specifically to mode collapse in the DINO head, I am not asking exactly why the teacher and student eventually gravitate towards producing the same output as thats one way to make the loss minimal even if its not constructively building proper feature representations in the output probability distribution vectors. What I am asking about is how the student would even get to such a state in the first place. If the teacher and student...
Originally asked by Zaid Al Jundi on Stack Exchange (ai) (CC BY-SA 4.0).