.png)
1 month ago
22
Keywords matter. When you enter a search in Google or a prompt in Gemini, you expect the answer to reflect the question. Consider an extreme example. If you entered a search for “best suvs of 2026,” would you expect to see the result below?
How much do keywords matter?
Google has had some ability to understand synonyms for years now. Consider this search for “cell phone” in 2012 (via the Internet Archive):
Three ways to measure similarity
To better understand Google’s current capabilities, we compared the queries to organic result titles (as displayed by Google, not <title> tags) using three metrics: (1) exact-match*, (2) partial match with Jaccard similarity, and (3) semantic match with cosine similarity.
1. Exact-match*
Exact-match is pretty self-explanatory, but we chose to be a bit forgiving, normalizing case and punctuation, removing plurals, and allowing any title that contained the full query.
2. Jaccard similarity
To analyze partial matches, we used Jaccard similarity, which measures the number of shared elements (in this case, words) across two sets vs. the unique elements of both sets. Put simply, it’s the proportion of shared words across the two strings to the total, unique words. This is measured on a 0.0-1.0 scale.
3. Cosine similarity
Finally, we calculated vector embeddings and cosine similarity between the two strings. This captures semantic relationships – in a word, “meaning.” Specifically, we used 768-dimensional Nomic embeddings. Cosine similarity also measures similarity on a 0.0-1.0 scale. Let’s look at the stats and some examples.
Exact-match data and examples
Even with our more forgiving exact-match*, only 43 display titles (0.49%) contained the full query. Here’s an example that only differs by a hyphen (-):
… and here’s an example where the title includes the query and a bit more:
Flipping that first statistic, 99.51% of display titles did not contain the full query. Given the data set of long-tail queries, I don’t think this will shock most of you, but it does illustrate how much SEO has evolved since the keyword-stuffing days.
Partial match / Jaccard similarity
Here’s where things get more interesting. The mean Jaccard similarity for the 8,703 display titles was 0.23 (note that Jaccard similarity is pretty unforgiving). To put that in context, here’s what a 0.23 score actually looks like:
I’ve highlighted the matching words – as you can see, the mean value represents a pretty limited overlap. Here’s a higher Jaccard score (0.75) that isn’t an exact-match:
Putting aside word-order (which Jaccard ignores completely), this is a substantial overlap. Note that a true exact-match would also have a Jaccard similarity of 1.0.
Semantic match / cosine similarity
The mean value of cosine similarity across the data set was 0.76 – cosine similarity is much more forgiving than Jaccard similarity. Here’s an example of a 0.76:
This one’s interesting because the display title is a much more structured, SEO-style title. While it’s specific to Minecraft and maybe not quite what the searcher intended, we can certainly see that there’s semantic overlap. Let’s look at a high-similarity example:
This result has a cosine similarity of 0.90, and you can see that the vector embeddings are probably equating concepts like “US” and “America” and ignoring some of the minor differences (like “8 of the”) that don’t impact the relevance to the query.
Bonus: high cosine, low Jaccard
A word-for-word exact-match is going to be ones across the board, and high Jaccard similarity almost always means high cosine similarity. What about cases where the word overlap is low but the semantic overlap is high?
Google isn’t just recognizing synonyms and semantic similarity here – they’re actually highlighting possible answers. While this highlighting is a layer that happens after results are retrieved and ranked, it’s worth exploring in your own search results to understand the kind of information that Google is rewarding.
.png)
English (US) ·