Why we use the term “Tokens” with AI today… before AI, software engineers used assemblers and compilers. The most common tools are LEX and YACC. LEX would take parts of a written language (specifically a programming language) and convert them into “tokens” which then went to YACC to be processed for context.
But even before the 1970’s (when LEX was created at Bell Labs) the term “token” was used as a place-holder or representation of something greater … as in a “token of affection”. The term “token” dates back to old English tæcen, meaning a sign, mark, symbol or indication but had been more historically used in Germanic languages (Zeichen) which means “sign”… more on that in a bit.
Long before we used it in computers, in the middle ages, people would give brooches and rings to loved ones as symbols of commitment (where we got the custom of the engagement ring) . One common trend at the time, for those who couldn’t afford brooches and rings, was to bend or engrave a coin to give as a gift to represented a value of something else. In the Old English tongue, during the Middle Ages, they didn’t have a name other than “a bent coin presented to my love” or “a pledge”. But in High-German of the middle ages, they would be called “Zeichen”. They were no longer spendable coins. As signs, they represented something specific and special to its giver and bearer.
So when you use a GPT today, even though computers can’t love, it was a lovely custom that over the centuries gave us the terminology we feed into it today.
Taking the dark and bitter, then making it light and sweet.
