I improved Semantle with an AI that can’t talk

Among the many ways I’ve found to waste time in front of a screen are daily games. One of them is called Semantle.

Every day, a word is picked at random and you have to guess it with a semantic game of hot and cold. At first you type random words, then you eventually get lucky and land on a word closer than the others. Then you follow the trail, testing similar words until you find the word of the day. The game is fun but repetitive, and finding the right word sometimes takes hundreds of guesses for a reward that doesn’t always live up to the effort. “Congratulations, you found the word cloakroom in 245 guesses. See you tomorrow.” The game was supposed to distract me from my own mortality. In the end, Semantle mostly sharpened my sense of wasted time and of the emptiness of my existence. Thanks, Semantle.

The semantic value of a word on its own isn’t enough to understand what it expresses, and that may be what Semantle is missing. All that’s left is to figure out how to measure the semantic distance between words depending on their context. Anyway, I ended up creating my own daily game: Whippin AI.

01Embedding

Let’s first look at how Semantle calculates the distance between words. A distance is a numerical value, so we have to find a way to express the meaning of words in numerical form too. Basically, expressing complex information as numbers has a name: it’s called an embedding. We could, for example, give words coordinates, which would then make it easy to measure how close two words are.

The idea looks good, but the more words we add to our plane, the harder it gets to spread them out intelligently, and two dimensions won’t be enough for long. So we’ll go from 2 to, say, 300 dimensions. That’s much harder to draw, but far more practical for filing away hundreds of thousands of words. The real problem finally takes shape. How do we choose the 300 coordinates of each word?

Skip-gram / word2vec

We start by assigning a random vector to each word. At this stage, our coordinates mean absolutely nothing, and to give them meaning we’ll ask a neural network to do something fairly simple: given a word, predict the ones likely to appear around it. We give it cat and it assigns a probability to every word it knows. Since everything is still more or less random, its first answers look like this:

1. glass5.0%
2. funny4.8%
3. engine4.6%
4. clavicle4.5%
5. September4.4%
6. casserole4.3%

Now we need a way to tell it that it’s talking nonsense. To do that, we build a large reference corpus from carefully selected texts: books, Wikipedia pages, academic papers, for example. Let’s take an ambitious sentence from our corpus:

The cat eats a mouse

We pick cat and look at what appears around it: the, eats, a. That’s what the network is supposed to predict. If it considers engine far more likely than eats, it has obviously made a mistake. We still have to put a number on how wrong it is, so we sum up all its errors in a score we call the loss: the higher it is, the worse its predictions.

We now know how to measure the problem, but not yet how to fix it. The network contains a huge number of parameters, including the coordinates of our vectors, and we need to know which ones to move and in which direction. For each one, we calculate the effect a tiny change would have on the loss. Basically, for each vector we answer the question:

If I change this coordinate very slightly, does the loss go up or down, and by how much?

All these variations make up the gradient, a vector that points in the direction in which the loss would increase the most, so we do something very sophisticated: we go the other way. We slightly change the parameters, make a new prediction, recalculate the error, then start over. That’s gradient descent.

1. predicting
2. comparing with the actual context
3. computing the loss
4. computing the gradients
5. nudging the vectors

After just a few billion repetitions, the loss starts to come down in earnest. But what we really care about is what all these corrections did to our vectors. Say the network regularly comes across sentences like these:

The cat sleeps on the couch
The dog sleeps on the rug

I give my cat something to eat
I give my dog something to eat

The cat runs in the garden
The dog runs in the garden

cat and dog obviously live fairly similar lives. They sleep, eat and run, so they appear surrounded by comparable words. To predict their contexts correctly, the network ends up giving them representations that look alike. Nobody told it that a cat and a dog are two fairly similar domestic animals. In fact, it still doesn’t know what a cat is. It simply noticed that cat often hangs out with the same words as dog, and that’s enough.

As it keeps correcting its predictions, the completely random vectors we started with end up organizing themselves. Words used in similar contexts occupy similar regions of our 300-dimensional space. At the end of training, we take these vectors and use them directly as the semantic coordinates of our words: we’ve just created an embedding.

02One word, one vector

Now that we have the Semantle recipe, here’s how to make a version with context, a distraction from death worthy of Pascal: take a lovely sentence from a book or a song, hide a few words and have the player find them. And there you go, we’ve created Semantle with context.

Except our embedding has a birth defect: it’s static. It assigns a single vector to each word, whatever the sentence. But words don’t have just one meaning. During training, bank hung out as often with ducks as with tax advisers, and its vector ended up somewhere in between, a place that’s neither quite the shore nor quite an offshore account.

As long as we’re guessing a word without context, this fuzziness isn’t really a problem, since the word of the day itself has no precise meaning. But let’s take this sentence:

Down beside where the waters flow, down by the banks of the Ohio

No robbers in sight. And yet, here are the neighbors of banks according to the embedding:

• shore
• loan
• creek
• vault
• slope
• deposit
• riverside
• robbery
• heist
• teller

Half the list by the river, the other half in the biography of Bonnie and Clyde. But let’s take another, more subtle example:

She treats me with a nurse’s gentleness

• address
• document
• cover
• examine
• dissect
• discuss

The embedding read it as “to treat a subject.” The sentence, though, is about treating someone. A player who types tend or coddle has perfectly understood the sentence, yet the game tells them they’re far off. This injustice breeds frustration, and frustration leads to war. So giving the player a sentence isn’t enough. The embedding, too, has to read the sentence.

03First brilliant idea: reading ChatGPT’s mind

For all the vastness of the universe, as far as we know there are only two kinds of beings capable of understanding human language. Humans themselves and, very recently, GPT and its fellow LLMs. The latter also happen to be better at handling 300-dimensional vectors, so that’s where we’ll turn.

Transformers

Before talking about a full LLM, let’s start with the building block it’s mostly made of, the Transformer. We’ve seen how to represent words as vectors. A Transformer receives several of these vectors and transforms them so that each one can take in information coming from the other words in the sentence. Then all it takes is to run a sentence through several layers of Transformers to get vectors that no longer represent just the words themselves, but the words in their context. On paper, Transformers look like the ideal tool to solve our problem.

The mechanism that lets words pick up information from one another is called attention. For each word, the Transformer builds three new vectors from its current representation:

Q (query): what I’m looking for
K (key):   when I might be relevant
V (value): the information I pass on if anyone’s listening

Let’s take the following sentence and focus on the word bank:

The duck swims to the bank

The Transformer receives its vector and computes its Q (query). We can imagine it expresses something like:

What in the context matters for understanding my role here?

Each of the other words has a K (key), and to determine which ones are interesting, the Transformer compares the query of bank with each of those keys using a dot product. The dot product depends on both the length of the two vectors and the angle between them. For a given length, the more they point in the same direction, the bigger the result: at right angles it’s zero, and in opposite directions it goes negative. We can therefore use it as a compatibility score between a Q (query) and a K (key), and these scores are then converted to percentages that add up to 100% (via a softmax, for those who care).

1. The4%
2. duck70%
3. swims20%
4. to3%
5. the3%
6. bank

These percentages are used to weight the corresponding values, which then enrich the vector of bank.

beforeafter

So bank has just taken in a lot of information from duck, without anyone giving the model a handwritten rule saying:

If the subject is a duck, then bank probably means the side of a river.

The model simply learned that, in this kind of context, the information contained in duck is very useful for correctly representing bank. The matrices that produce Q, K and V are themselves parameters of the network. At the start of training, they produce essentially nonsense, then they are gradually adjusted by gradient descent, like our embeddings.

Once attention is done, each word has a new representation, enriched by what it picked up from its context. It then goes through a more conventional neural network before becoming the input of the next Transformer layer. And the whole thing repeats a few dozen times. At each layer, new Q, K and V are computed from the representations produced by the previous one. An early layer might learn a fairly simple relationship between duck and bank, and later layers build on that to form progressively richer representations.

To train a model like GPT, we give it a task fairly close to the one we used for embeddings: predicting what comes next in a text. We show it, for example, The duck, and it has to predict swims, then The duck swims, and it has to predict to, and so on over billions of pieces of text. At each prediction, we calculate the loss, then the gradients, and we slightly change all the parameters. After predicting what comes next in billions of sentences, the model gradually learns which words should listen to each other and which information should flow between them. That’s how a huge stack of Transformer layers, trained with the rather basic goal of guessing the next word, ends up building something that looks dangerously like a nuanced understanding of context.

Opening the hood

At each layer of the network, each word therefore has a vector that has listened to its context. This intermediate vector has a name: the hidden state. It’s exactly what Semantle was missing: an embedding that has read the sentence and can tell the bank a duck swims to from the bank that finances land grabs in the West Bank. Normally we don’t look at these vectors directly: we let the model run all the way and read the text it produces. But nothing prevents us from stopping it partway and retrieving the vector produced by the layer we’re interested in.

Nothing, except OpenAI. ChatGPT won’t let us pop the hood, so we need a model whose parameters we can download and run ourselves. Fortunately, market communism works in mysterious ways, and Alibaba hands out the weights of its Qwen model for free.

The plan

Let’s go back to our sentence:

The duck swims to the bank

1. We take the 10,000 words closest to bank according to our standard Semantle-style embedding.

• shore
• loan
• creek
• vault
• slope
• deposit
• riverside
• robbery
• heist
• teller

2. We write 10,000 versions of our sentence, replacing bank with each of these words.

The duck swims to the shore
The duck swims to the loan
The duck swims to the creek
...

3. We feed these 10,000 sentences to the LLM, retrieve the hidden state of the swapped-in word each time and calculate its distance from that of bank in the original sentence.

4. We sort the list again, but by this contextual distance.

• shore
• riverside
• embankment
• slope
• waterside

We’ve got our solution. We keep Semantle to quickly find candidates, then we use an LLM to eliminate the ones with the wrong meaning. We no longer compare bank and loan in a vacuum, we first let them read their respective sentences, then we compare what they’ve become. In theory, the idea is perfect.

And yet the article isn’t even halfway through. The theory had left out three important details.

Problem 1: causal mask

First detail I glossed over. During attention, a word can only listen to what comes before it. This is called the causal mask, and it makes sense: the model is trained to guess what comes next, so we’re hardly going to give it away. Let’s take a new sentence:

The fisherman’s bank account was empty

1. The
2. fisherman’s
3. bank
4. account
5. was
6. empty

When the model computes the hidden state of bank, it has heard The and fisherman’s, but it hasn’t heard account yet. The most useful part of the sentence is invisible to it, and as far as the model knows, the fisherman is still by the river. For a word to have read the whole sentence, we have to place it after the sentence:

Context:
The fisherman’s <TARGET> account was empty

Lexeme:
bank

This time, bank has read everything. Unfortunately, nothing really pushes it to make the connection with the sentence above, and its vector falls back on a fairly general meaning, close to that of the static embedding:

• loan
• shore
• vault
• creek
• deposit
• riverside
• teller

Problem 2: word ≠ token

So far, I’ve been pretending that an LLM reads words. It doesn’t. It reads tokens, pieces of text chosen to be reusable. A common word can fit in a single token, while a rarer one may be split into several, in which case it gets one hidden state per token. We have to choose which one to use, and since each token can only read what comes before it, the last one is a fairly natural candidate: it has at least seen all the previous pieces of the same word.

With Qwen 3, for example, bedbug is split into bed- and -bug. Here’s what turns up among the nearest neighbors of its last token, -bug:

it’s getting harder and harder to tell the bedbug from the landlord

• rug
• mug
• thug
• slug
• drug
• shrug

Not a single insect, whereas the static embedding offered flea, cockroach and tick. We just turned Semantle into a rhyming dictionary.

For the record, the tick isn’t an insect but an arachnid.

Problem 3: LLMs predict tokens, not distances

Third detail, and the deepest. Our Semantle-style embedding was trained so that words used in similar contexts get close vectors. An LLM, on the other hand, is trained with a different objective: predicting the next token. So there’s no guarantee that the raw geometry of its hidden states makes a good semantic distance. These vectors are above all intermediate representations, there to help the model carry on with its calculation.

First symptom: the vectors look far too much alike. With our static embedding, we measured how close two words were with cosine similarity: the dot product of two vectors first scaled to a length of 1. That leaves only the angle: two vectors pointing in the same direction are considered very similar, regardless of their length.

In an ideal embedding, the vectors would be fairly well spread out in all directions. But the hidden states of an LLM tend to pile up in the same region of space. As a result, even two words that have nothing to do with each other already point in largely the same direction. In my tests, the similarities are almost always between 0.45 and 0.88, on a scale where 1 means two vectors point in exactly the same direction and 0 that they’re perpendicular. This phenomenon has a name: anisotropy.

We can try to remove part of it. If all the vectors tend to point to the same place, we calculate their mean vector, then subtract it from each of them. Basically, we strip out what they all have in common before measuring their differences again. That spreads the candidates out a bit better and makes their similarities more discriminating. But not enough.

rawcentered

Second symptom: in the last layers, the model isn’t just thinking about the meaning of the word anymore. Mostly, it’s getting ready for whatever would make sense at that point in the sentence:

until their retirement

• graduation
• wedding
• strike

Events that fit very well after “until their,” but have little to do with retirement. The model largely measures how well the candidate works with what surrounds it, whereas we want to measure how well its meaning matches that of the hidden word. We wanted a semantic distance, we built a fill-in-the-blank exercise.

The project seemed doomed to fail, but poking around in the guts of an LLM was still more fun than going back to Semantle with my tail between my legs to guess the word “wiener” in 328 tries. So I kept looking, and eventually I found something (sort of).

Turning the problem against itself

Since the model is obsessed with what comes next in the sentence, we might as well turn the problem against it and make sure that what comes next is precisely the answer we’re looking for:

Context:
A gold digger and a broke guy just don’t go together

In this context, a word close to “broke”:

This time, instead of the vector of broke, we take the vector of the colon, at the very end of the prompt. That token has read everything: the sentence, the word we’re interested in and the instruction. And since its role is to prepare what comes right after, we can imagine that at that moment the model is asking itself a question fairly close to ours:

I’m going to have to write a word that means roughly the same thing as broke in this sentence.

So we’re no longer measuring the representation of the word itself, with its spelling and its possible split into several tokens, but the state of the model just before it offers an answer. As a bonus, this solves our problem with split words like bedbug: whether they take up one token or five, by the time it reaches the colon the model has read them all. And if we look layer by layer, on a model with 36 of them, we can even watch the answer come into focus:

layer 22

• broken
• broker
• brokerage
• brook
• brake

layer 28

• broken
• broker
• penniless
• cash-strapped
• poor

layer 33

• cash-strapped
• penniless
• impecunious
• indigent
• destitute

In the middle of the network, the representation is still very sensitive to the form of broke. A few layers later, the spelling neighbors progressively disappear in favor of words tied to its meaning in the sentence. The model has finally understood that we’re talking about a guy with no money, and a few other examples, chosen precisely for their ambiguity, now start working properly:

Down beside where the waters flow, down by the banks of the Ohio

• shore
• riverside
• waterside

She treats me with a nurse’s gentleness

• handle
• pamper
• tend

Better, but not quite there yet

Except this solution generalizes rather poorly. Spelling lookalikes keep climbing the rankings (angle, for example, ends up as the 4th neighbor of angel), and the top of the lists is sometimes downright noisy. In they were bundles of nerves, nerves ends up surrounded by organs, thymus and, for good measure, the French viscères.

Above all, I have no way of really talking to the model. A vector gives me a distance, but it doesn’t let me explain to it what shouldn’t count. If a word is rewarded simply because it rhymes with the answer, I can’t write “enough with the rhymes” anywhere. I can only change the prompt and pray. I did try several phrasings: they all produced different rankings, but none was clearly more relevant than the others.

To avoid unfairly blaming LLMs on the basis of a single model, I repeated the experiment across several generations and architectures:

Qwen 3

I first tested Qwen 3, which uses the standard Transformer architecture we just described, in several sizes from 0.6 to 14 billion parameters, without quantization. Increasing the size didn’t always improve the results, and the same underlying flaws kept coming back.

Qwen 3.5

Qwen 3.5 uses a more recent architecture: in three out of four layers, standard attention is replaced by a cheaper linear memory mechanism (Gated DeltaNet, for those who still care). It made no difference to our problem. bedbug keeps rhyming, and on six ambiguous words this model even proved less effective than its predecessor. Best part: for the banks of the Ohio, the 4-billion-parameter version understood the river, and the 9-billion one, the vault. I’d made the model bigger and the answers worse. My approach was clearly wrong.

Qwen 3 – Embedding

As it happens, there are LLMs retrained to produce good vectors. This time, they are explicitly taught that semantically close texts must produce close representations, and that unrelated texts must move apart. These are embedding models, and that’s exactly the property our cobbled-together hidden states were missing. Qwen, for instance, offers Qwen3-Embedding, a family built from Qwen 3 and then specially trained to produce representations intended for search and semantic comparison. I tested it in several ways. The results were just as mediocre as my hidden states.

Qwen 3 – Reranker

Qwen also distributes Qwen3-Reranker, perhaps the most promising candidate on paper. When you do a Google search, the engine first gathers thousands of potentially relevant pages, then reranks them to put the best ones first. That second sort is the job of a reranker, a model that rereads the query alongside each page and judges whether the page answers it. That looks like our problem, phrased differently. The sentence, with its hidden word, is our Google search, and the 10,000 potentially relevant words the Semantle embedding hands us are the results to rerank. This time there’s no distance to calculate and no Transformer guts to poke around in: we simply ask the reranker to do its job.

It understands broke and bedbug the way their sentences mean them, but a reranker is designed first and foremost to judge how relevant a result is to a search. So it brings up words that go well with the search, not necessarily ones that mean the same thing: unemployed comes second for broke. Which makes sense, but it’s not quite what we’re after.

In my tests, it also had an annoying habit of treating the instruction as part of the search. Even when I explicitly told it that, in the strength to stifle my emotions, stifle should not be considered close to emotions, it kept ranking it near the very top. A bit like typing “recipe without peanuts” into Google and watching it zero in on recipe and peanut.

So that’s it? I’m just a sorry excuse for a tinkerer who can’t even improve the dreaded Semantle, doomed to be tormented, day after day, by ever more absurd words?

04The solution: stop measuring, start asking

That’s when I came across Jev, a decision model released in September 2026 by TypeSafe. So what is this wonder model, the one that succeeded where LLMs failed, actually made of? We don’t know exactly: TypeSafe mentions a new architecture but hasn’t published the details. But we can still get a fairly good idea of the principle. Take a model that understands text, then make it switch careers: from writer to judge. A standard LLM ends its calculation by trying to answer this question:

Which token should I write next?

A decision model can use a representation of the same kind, but end with a small specialized layer, a classifier, which answers this instead:

Among these answers, which one seems correct, and with what probability?

Grossly oversimplified, it looks something like this:

LLM

1. text
2. representation
4. probabilities over all the next tokens
5. text

Decision model

1. text + question
2. representation
3. classifier
4. probabilities over the proposed answers

The bulk of the work stays the same: reading language, understanding the relationships between words and building a representation of the context. Only at the end do we ask it for something else, and that’s precisely what we need. We spent all this time retrieving the intermediate vectors of a model trained to write, hoping their geometry would somehow serve our purpose. This time, we can simply ask the question:

In this sentence, how close is this word to that one?

Let’s look at the new plan.

A rating for each candidate

Semantle stays on as the talent scout. It rounds up the 10,000 neighbors of the hidden word, without worrying about context, and Jev then rates each one:

0: no relation in meaning to the secret word as used in the sentence
1: distant relation, same very general domain
2: related, same semantic field or neighboring idea
3: very close, near-synonym
4: same meaning, direct synonym in this context

The instructions are written in plain English: only meaning counts, not grammar, not spelling, not whether the candidate can replace the word in the sentence. Let’s take nerves again:

Qwen

• organs
• thymus
• minds
• medical
• marrows
• viscères

Jev

• nervousness
• excitability
• nervous
• irritation
• irritable

Jev understood we were talking about temperament and not neurology. As for rhymes and spelling neighbors, they’re gone for good. At first, though, Jev also made mistakes:

the strength to stifle my emotions

28th place

• stifle

stifle isn’t a word close to emotions, it’s a word from the sentence. So I added a line to the instructions:

A candidate that appears in the sentence, or that describes what the sentence does with the secret word, isn’t any closer because of it.

stifle went from 28th to 1,182nd place. When I was poking around in the hidden states, I could see that a result was bad, but I couldn’t explain to the vector why. Here, if Jev gets it wrong for a reason I can put into words, I can simply tell it.

Not everything can be fixed with words, though. I had also written that a candidate that isn’t English deserved the lowest rating, and Jev blithely ignored it. retraite stayed the first neighbor of retirement. I ended up fixing this with code, by demoting any word that’s three times more frequent in a French corpus than in an English one.

A tournament to sort out the winners

Jev isn’t perfectly stable either. If I ask it several times to rate the same candidate, its answer can vary by about 0.1 points. At the bottom of the ranking, that doesn’t change much. If the 7,400th word becomes the 7,500th, it won’t shake up the game. At the top of the ranking, it’s different. That’s where the order matters most, and if the words ranked 50th and 150th have almost identical ratings, such a variation is enough to make them interchangeable. So Jev is pretty good at telling which words belong in the top 200, but much worse at putting them in order.

Time for a tournament. Each of the top 200 candidates faces every other one, for a total of 19,900 duels, all on a single question:

Which of these two words is closer to the secret word in this sentence?

For each duel, Jev gives both candidates a probability of winning, and a word’s final score is simply its average probability of winning across all its matchups. A standard sorting algorithm would need far fewer comparisons. But a sort assumes we can trust the function that compares two elements. If A > B and B > C, we’d reasonably like to be able to conclude that A > C. Jev, on the other hand, may very well answer that A < C: it’s neither perfectly deterministic nor perfectly transitive. In a standard sort, one bad comparison could send a word to the wrong place and influence everything that follows. In a full tournament, each candidate faces the other 199, and if Jev makes a strange call, it gets diluted among 198 other matchups.

And it really works. For the secret word coldly, impassivity goes from 8th to 2nd place. For beauty, fascination climbs from 152nd to 13th.

Finally, past 300th place or so, I just keep Semantle’s original order. Nobody will notice, but now you know.

Why does it work?

Let’s be fair to LLMs, though. Nothing I ask of Jev seems out of their reach. I could take a good model, show it the sentence and two candidates, then ask it which one is semantically closer to the secret word. It would probably answer very well. The problem is that I’d have to ask it the question tens of thousands of times, and an idea that’s perfectly reasonable on paper becomes much less appealing when each little decision costs a few seconds and a request to a big model.

Jev can’t do anything an LLM couldn’t, but it can make a small semantic decision extremely fast and for almost nothing. For one secret word, my system has it process about 3.1 million tokens, and with the fixed part of the prompt cached, the whole thing costs me about $0.13. So I can afford a strategy of remarkable brutality: have 10,000 candidates judged one by one, keep the best 200, then organize 19,900 additional duels just to rank them better. With a slower or more expensive model, I’d have spent my time looking for a clever algorithm to reduce the number of requests. Jev, though, is cheap enough that we can ask it the question thirty thousand times.

When an operation gets a hundred or a thousand times cheaper, you don’t just get the same solution for less. You get the ones you’d never have bothered to try.

05Thanks anyway

A few embeddings, several billion parameters, hundreds of thousands of comparisons and a reasonable amount of stubbornness later, the problem is more or less solved and Semantle can finally leave my daily life for good.

But in the end, no daily game has ever distracted me from my mortality as well as doing everything I could to never have to play Semantle again.

So thanks, Semantle.