Introduction
Click Residuals is a clever technique for finding failing search queries first described by Walter Underwood in 2022 as a query success metric. I’ve used Click Residuals (CR) in several client engagements to identify both individual and groups of queries that don’t seem to perform as well as they could – and in recent engagements worked with my clients to develop the concept to help them and you to zero in on the reasons for this low performance.
Why do we need CR
One of the biggest challenges when tuning search is to identify what to focus on for the largest benefit. Many search teams are forced into a reactive pattern – a colleague or customer finds a query that doesn’t work, and this is immediately looked at (perhaps because the colleague was a senior executive, or perhaps just because it’s top of the pile). Seldom is any impact assessment carried out or any evaluation of the business value of this failing query. These queries are also considered individually rather than symptomatic of any wider issues. This leaves the team little time to come up with more proactive strategies around finding and fixing search problems.
Deriving metrics for the business value of search is also difficult. Successful user actions (an add-to-basket, purchase, article download etc.) can be tracked, but attributing these to particular queries is hard – we don’t know if the results from a query actually caused this action, and many search sessions involve multiple queries, refinements, clicks back and forth and abandonments. Perhaps the user even did the search task last week and then came back and bought a bookmarked item today.
Leaders however will often ask search teams to improve conversion metrics, especially if they can find broken queries. “I know we sell i Pads,” [yes, the space is deliberate!] “but I searched for them this Sunday and all I saw were accessories. Fix this and we’ll make lots more money!”. Seldom is it so simple as the HIPPOs gather.
So, we need a simple and straightforward way to find which of our search queries are underperforming, to move us to a more proactive pattern – we need to find and fix these issues preferably before others tell us about them.
How to get CR data from your analytics
As Underwood writes: “Click residual” is the difference between the expected number of clicks and the actual number of clicks. When that is negative, you can see how many times a customer did a search, but wasn’t satisfied enough with the results to click, relative to the overall performance of the search system. “
What CR thus gives us is a measure of unhappiness with the results of a query. It’s important to remember that there may be lots of reasons for this, not all of them resting with the search team. Searching for ‘red shoes’ in an electronics store is unlikely to lead to satisfaction (and people type far weirder queries than this). If a particular query triggers a huge, glaring banner advertisement that could also make users unhappy and unlikely to click.
You can start with CR by asking your analytics team to give you the clickthrough rates for each user query – how many times the results for this query were clicked. The formula you need is in the blog above, and you should end up with a spreadsheet with lines like this:
| My Online Chip Shop | ||||
| query | frequency | Actual clicks | Expected clicks | Click Residual |
| fish | 885 | 2708 | 1985 | 723 |
| chips | 2207 | 6650 | 4950 | 1700 |
| vinegar | 320 | 223 | 717 | -494 |
| sosages | 187 | 102 | 419 | -317 |
| … | … | … | ||
| Totals | 45803 | 102750 | ||
| Average | CTR/freq. | 2.243 |
The lines with the negative CR might indicate a problem – in this toy example, my instinct would be to investigate how the search system deals with typos, and whether we actually provide a good selection of vinegar options.
However, we’re also looking for overall trends, not just individual problems. Sort the sheet by the residuals and frequency, and we can see which popular queries aren’t performing as they should. Carry out some simple clustering of the queries using your favourite ML or AI tool and you might be able to find whole groups of terms that underperform. In past engagements I’ve found problems with misspelled queries, a lack of good results for queries about animals and football teams and problems with what I call functional queries e.g. “returns policy” or “where is my nearest store”.
Business benefit & prioritisation
As you might now have lots of queries with low CR and need to decide what is more important, you can add more data into the mix to try and derive a measure of business benefit. If you find an interesting group of failing queries, look at the total query volume for these queries so you can consider how important they may be. Perhaps so few people search for football teams that you should de-prioritise any fix.
If you know that a particular subject area is important to your business (e.g. mobile phones are more profitable than kitchen appliances) than you can also use this to order your failing searches in terms of priority. Commercial measures like Average Order Value can help if you can derive this on a per-query basis.
Exit rate is another good thing to add – did the user immediately leave the site after their query? A query with a negative CR and high exit rate is another good flag for user dissatisfaction.
Adding more data to our CR spreadsheet
This technique makes some assumptions that may not fit how your site actually operates, in particular that the average CTR is the same across all of your site. Because user behaviour is different depending on their search intent, it might be better to calculate it on a per-category basis.
Consider how people buy kitchen appliances (rarely and after a lot of consideration of the cost and how it will fit into their kitchen) against cleaning supplies (regularly, probably at the lowest cost with lots of repeat buying once they’ve found a good option). The CTR might be very different in these two cases.
If you have some kind of query intent classification or category prediction pipeline this can be used to tag each query with a category. Alternatively, AI tools can help you guess that “washing machine” and “wash cloth” are likely to apply to very different categories. You might then find out more about what queries are really underperforming in each category and combined with the business value measures described above, decide further on prioritising what needs fixing.
You might also have data on which parts of your query pipeline fired on each query – did spelling correction happen, or was there a business rule in play? All of these give you further context on what parts of your system may not be performing as well as they could.
If you have some kind of relevance judgement process this might produce metrics like NDCG for some queries. This might indicate where some queries produce irrelevant results, but also might lead you to ask why users aren’t clicking results as much as they should even if they are all highly relevant. Perhaps there are other reasons – like price points or stock availability.
Acting on issues found with CR
Now you’ve identified some queries or groups of queries to fix, the next challenge is to work out some potential solutions. Your first steps should be logging the problem in some kind of issue tracker, and having a conversation with your business team about what is the best approach – carry more stock, improve the UX or fix the search algorithm. In the latter case, offline testing is your friend here, as it allows you to try out these solutions on a small set of queries (drawn from the CR spreadsheet) before you go live.
Of course, you won’t know if your chosen solution has improved your CR numbers until you’ve gathered enough new click data, so a constant online analysis and monitoring process should be set up.
Summary
Click Residuals is a remarkably simple technique for identifying underperforming search queries. It doesn’t necessarily tell you why a query is underperforming, but it’s a great way to move to a more proactive process for fixing search issues. Enhancing CR with extra data can give you more ways to prioritise these fixes and understand the underlying reasons behind the user dissatisfaction that is indicated.
I’d love to hear your ideas about how to use or enhance Click Residuals – get in touch if you’d like to discuss how it can be implemented at your organisation.