Tokens in LLMs explained!
https://itsfoss.com/llm-token/
@itsfoss Token is each words output. Not sure if anyone doesn't know that. "This is a boy" 4 tokens
@isola @itsfoss Not necessarily. A word can be multiple tokens, especially in a language like German. See the article for more possibilities.
Librem Social is an opt-in public network. Messages are shared under Creative Commons BY-SA 4.0 license terms. Policy. Stay safe. Please abide by our code of conduct. (Source code)
