How do BPE Tokenizers with `add_prefix_space` handle natural language, such...

ubuntu_ulysses

BPE Tokenizers are the standard for modern LLMs. By default, most add_prefix_space , so that John went away is pretokenized to [_John][_went][_away] . To preserve reversibility, on roundtrip, the leading space is removed. This allows the tokenizer to map a word in the beginning of the sentence to the same word anywhere. But what about cases where there is no leading space in natural language? Examples include poetry ( John went away\nJohn will...

Originally asked by SRobertJames on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

Prefix spaces are not added in the beginning of the first words of a sentence. So John went away\nJohn will come back another day\n will become [John] [_went] [_away] [John] [_will] [_come] [_back] [_another] [_day]. There are various implementations of BPE which handle punctuation symbols like " differently. Some consider them as a part of the word. There is also a variation called Byte-level BPE to handle this. The purpose of adding whitespaces is to let the model that "John" appearing at the beginning of sentence is different from "John" appearing in the middle of a sentence.

Full answer by Ritil: https://ai.stackexchange.com/a/49031 (CC BY-SA 4.0).

Need it done for your business? UphillTech.