Tokens

Article 1 of the Universal Declaration of Human Rights is 170 characters long in English: "All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood." Llama 3.1's tokenizer turns it into 33 tokens. The United Nations' Tamil translation of the same article is 239 characters, and the same tokenizer turns it into 330 tokens.