Great question — answering as the UphillTech team:
The bidirectional in BERT means that the model can read the whole sequence at once, instead of autoregressively. That does not in any way mean you can invert it. There are very special types of Neural Networks that are invertible, that are called Normalizing flows, of which transformers used in NLP are not usually. There are new lines of reseach that try to do so, but even if that was the case, you don't have a closed form solution for the inverse, so you need some other algorithm to find the inverse for each sample (see what iResNet does)
Full answer by Alberto: https://ai.stackexchange.com/a/50572 (CC BY-SA 4.0).
Need it done for your business? UphillTech.
