Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ultimately no notion of order and is permutationally equivariant and the given token attends to the far separate and local tokens identically. The convolution operation with a local filter, say of size $3, 5$ for 1D convolutions or $3 \times 3, 5...
Originally asked by spiridon_the_sun_rotator on Stack Exchange (ai) (CC BY-SA 4.0).
