Speaker
Andrew Janco
(Princeton University)
Description
Transformer-based language models have a fixed vocabulary of tokens used to represent words and word parts. Which languages can and cannot be represented with current model vocabularies? When native support does not exist, what are the possibilities and drawbacks of byte pair encoding? This short talk presents a tool to check how well current model vocabularies support the language(s) you’re working with. It joins data for more than 8,000 languages from the Unicode CLDR and Glottolog, with common vocabularies in tiktoken and HuggingFace AutoTokenizer.