Signal & Noise
Menu

Tokenizer

The algorithm (typically BPE) splitting text into tokens. Different per model family, which is why the same text costs different amounts on different providers.

← Back to the full glossary