4 Prompt Tracking Mistakes — Whiteboard Friday

Speaking of which, I think the second big mistake that people make is thinking about this in traditional SEO metrics, like ranking.

So here, I could say, "Oh, fine, I'm mentioned first. Therefore, that's great. I've won." But actually, the Samsung brands have been mentioned more times.

Over a large number of prompts, I might be far more interested in how many times I'm mentioned at all, what percentage of responses mentioned me at all, how I'm mentioned, what are the snippets in which I'm mentioned, and what kind of adjectives are used.

There are all sorts of metrics. I think this is quite a young sort of space right now. So this is emerging in terms of what metrics people prefer. But I really think you shouldn't focus on the traditional SEO ones because they're not that helpful in this space.

Sure, it is a piece of contextual information who is mentioned first. It is something you might wish to look at, but it shouldn't be your be-all and end-all in the same way that it might have been traditionally.

I would be looking over a large number of prompts as my sort of bread and butter, my equivalent to ranking, before I get into the more subtle stuff. I would be looking over a large number of prompts and just looking at what percentage of the responses mention me at all as my main sort of gauge to get started, which brings me on to my third big mistake, and that is quantity.

If you are a brand like Apple, like I've just been talking about, you probably have been tracking. I assume that Apple does have an active SEO program, and they probably are tracking thousands, if not tens of thousands, perhaps hundreds of thousands of keywords.

So it wouldn't make sense for a brand of that size to go and say, "Okay, across all of our products and all of our brands and all of our markets, we're going to track 50 prompts."

If anything, there is more variance in prompts. The outputs vary more from day to day. They are less deterministic. They are longer tail. If anything, you would be tracking more prompts than you were doing top-of-funnel keywords.

Now, don't get me wrong. I think if you're a small business, I think if you're a local business, I think if you're a business that does a very specific thing, maybe 50 prompts can be enough in some cases. But this is the same kind of business that would have been tracking a smaller number of keywords as well.

So I think you need to have a scale that is appropriate, and that's what we're trying to support in our products as well, making sure that customers can have a scale that is similar to their keyword tracking without it being sort of astronomically expensive or something like this. 

The last thing I want to speak about is head terms. So when you're deciding what prompts to track, as I mentioned, they're going to be longer tail typically. I don't want you to take whatever your existing keyword list was and just dump it into whatever your prompt tracking tool of choice might be.

Very few people are opening ChatGPT, typing "laptops," and then hitting Enter. I'm sure it's been done. It's probably been done by me for an example on a slide. But this is a weird thing to do. This isn't really how people interact with these tools. So you should make sure you reflect that in the kind of prompts that you're researching and including.

Now, prompt volume, it can be very difficult right now. There are long-standing stats that people like to use, which I think is something like 15% of searches of keywords that Google sees every day are brand new, as in never seen before.

One has to imagine that for prompts, this is 90-something percent. I don't know the exact stat. 

But if I were to guess, it's probably 90-something percent. I think the overwhelming majority of prompts probably have a volume of roughly one. So, really, you just need to identify the topics that are important and then have some realistic prompts. 

We have some tools to help with this. My colleague, Dr. Pete, has written a great guide, which will be referenced below, I hope. People Also Ask could also be a good starting point because those are sort of real long-tail keywords that Google seems to think exist and might be useful. There are lots of different things you can do here to try and get some kind of representative, more specific, longer-tail terms. Those are the four sorts of mistakes I wanted to talk about.

1. Consider specifying locations or personas

So one of the big ones that you could play with is locations and personas. 

So, in your traditional rank tracking, something that a lot of people do and something that I've done in the past is take the same keyword and then maybe track it from say 50 different postcodes, or something like that, to see if I get different responses. 

For different kinds of business, this may be more or less useful. Or if you're not a business that deals in real-world locations, then maybe this would be more about personas. So instead of saying, "What is the best phone for me?" you might say, "What is the best phone for a university student who has to do document editing on the go?" or something like this. That's a pretty strange example, but for example. Then you might have various other personas, or you might have various other locations, depending on the type of business. 

2. Play around with different languages

The second thing you can do is play around with languages. 

So at the moment, most prompt-tracking tools are somewhat language-ambivalent because most of the interfaces that they deal with are also language-ambivalent. 

ChatGPT basically responds in whatever language you use. You can open ChatGPT. The interface is in English. If you type to it in Italian, it will reply to you in Italian. So you can take advantage of that. If you do exist in multiple markets where you want to see what the responses are like in different languages, you can take advantage of that and just track prompts in whatever languages you see fit. 

Obviously, that could be combined with the location point. 

Then these last two ideas are slightly different. So this is less about sort of prompt expansion in terms of getting the prompt you want to track to get more detail and to be more representative, and more about getting sort of different types of insights rather than just performance.

3. Specify qualities

So, for example, instead of saying what is the best iPhone for blah, blah, blah, you could say what is the cheapest phone, or the most durable phone, or the most trustworthy phone, or the most privacy-respecting phone. You could have these as sort of different campaigns that you track in order to get different types of insights. 

4. Identify and focus on market gaps

Or the flip side of that, this would take a little bit more processing on your part to go through these insights, but you could ask, "What is it that the Samsung phone does better than the iPhone," or something like this, and then get sort of more qualitative insights out that you could then track over time to see whether you're making a difference in your marketing with that perception. 

All right. I hope you like these ideas. I hope you're not making these mistakes. If you are, I hope this was useful. Thank you very much. See you in the next Whiteboard Friday.