2026-10-27 04:47 UTC
Replies (3)
-
@YetAnotherNerd@sopuli.xyz 2026-10-27 04:53
It’s the smallest unit an LLM can process. If you write a long florid prompt, it’s more tokens and therefore costs more for the LLM to interpret. There’s literally a “caveman” mod where it makes the LLM respond in caveman-speak, which means loss tokens and therefore cheaper. What’s odd (did not RTFA) is that some languages are cheaper, IIRC even has a super-caveman mode where it responds in Chinese/Japanese because that costs fewer tokens.
-
@brsrklf@jlai.lu 2026-10-27 06:18
In the end LLM are just predicting what is statistically the most likely follow-up for a sentence, according to what they’ve been trained on. That’s how they work. For this they cut sentences into bits called tokens. The text they are working on is a sequence of tokens, and they predict the next one. A token could be a whole word (generally if it’s short and usual), often it’s a syllable, and if you’re trying to use uncommon words/names they might even cut them down letter-by-letter in part to try and fit them into something they can swallow. Of course tokens would be very language dependent, and some languages just use more characters/longer words and sentences than others on average.
-
@southsamurai@sh.itjust.works 2026-10-27 20:26
I’m going to add to the previous responses that a token is also the billing unit used. If you hire a lawyer, they charge a rate based on their predetermined “billable hours” arrangement. They beak down their jobs into small time segments, often much smaller than an actual hour. Similar to how a mechanic might use a predetermined labor cost based on a time estimate chart; swapping a battery takes X time, but rebuilding an engine takes Y time, with the actual number of hours and minutes not being useful in terms of billing because neither lawyers or mechanics can precisely track exactly how much of the time they’re digging working on your specific job is being spent “hands on” vs the time necessary to do all the little things that come up to interrupt the work. So they bill by the abstracted time a job takes when stripped of real world necessity. Tokens are, as others said, chunks of data. When you enter a prompt, and get a response, those chunks get tallied up and billed for. From what I’ve read (and I’m not in the field, not even as s hobbyist/amateur; I just wanted to know exactly what you asked) the various companies bill by large chunks, usually per million tokens, or a similar chunks. As an example, a given word might only need three tokens to process as input, and another three as output. But, because the cost of doing that work is more intensive to output (generate) a response, output costs more per unit. Now, I’m not saying the capitalist hellscape reality of that is a good thing, I’m just saying that’s the reasoning of the capitalist twats that run the companies. Anywhere other than lemmy, I might not need that disclaimer lol. Now, why they chose token as the jargon, I’ve never looked to see if I could find an answer, but that’s just me wondering just enough to type it, but not enough to go looking ;)