Tokenizer
The algorithm (typically BPE) splitting text into tokens. Different per model family, which is why the same text costs different amounts on different providers.
The algorithm (typically BPE) splitting text into tokens. Different per model family, which is why the same text costs different amounts on different providers.