As a Product Manager or engineering lead within a search team, it’s often hard to know where to start fixing search. You’ll have a list of complaints from users, management, those producing content, customer support teams; a backlog of things that you know don’t work very well, plus a nagging feeling that the competition are doing search ‘better’, whatever that means.
Getting started is often the hardest part – deciding what is most important and how to prioritise potential fixes. I’m going to assume for the purposes of this blog that you have at least some data on what users search for – query logs – and some web analytics data showing which of these queries are performing well or badly. Here’s some places to begin:
Zero results
Search queries that produce zero results are a great starting point – after all, if you can’t produce any results at all, your user is probably going to assume you can’t help them (even if they mistyped the query, or used a unusual term to describe what they want). They may just decide your website is ‘broken’ and go to a competitor. Zero Result Rates (ZRR) can commonly be 3-5% of all query traffic (although once I saw a horrendous 35% ZRR on the website of a well known UK brand). This number should be easy to extract from your query logs.
There are various ways to reduce ZRR, but the most important thing is not to leave your user staring at a page with a short message that implies ‘We can’t help you’. Typo tolerance or did-you-mean spelling suggestions, query relaxation, query rewriting, vector search or even displaying recommendations are all useful things to explore.
Low or zero-click results
Queries that produce results nobody clicks on are obviously concerning. There’s a cunning technique to identify these queries called Click Residuals, described in this classic blog from ex-Netflix search lead Walter Underwood – calculate an average level of clicks across all your queries and then work out which are much lower than this, with ‘residual’ – unharvested – clicks.
This technique will quickly show you underperforming queries, but it gets more interesting when you cluster groups of queries together (not particularly hard to do with a bit of ML). You can then end up with groups of underperforming queries – perhaps you’re not doing so well at queries about animals, or 3-word queries. This helps you identify problems that affect many queries, and if you can then come up with a fix this may have a large impact.
High and low traffic
Some of your queries will be used very often, and some will be rare, further down the ‘long tail’ of the classic power curve. Fix those high frequency queries and you’ll be helping a lot of users – but don’t ignore the low frequency queries. If you identify classes of queries to fix – perhaps using the Click Residuals technique above – then you can fix a lot of these low frequency queries in one go, and the total volume of these can be significant.
Common techniques to fix high frequency queries include rule-based redirection, synonyms and query rewriting (e.g. if a common query is ‘where is head office’ then just redirect to a map, or if the query includes a term that doesn’t exist in your source data use synonyms). Low frequency queries might be helped with vector search or typo tolerance (which could fix many mis-spelled queries).
Calculating value
Just considering query frequency doesn’t give you a full picture of the value of each query. If you get many queries for red shoes, but you don’t make much profit on shoes, then this perhaps isn’t as important to fix as queries for handbags where your margin is much more significant. This is a great example of why you should work with your commercial team to identify which queries, or classes of queries, generate the most value for your business.
Working with colleagues closer to the ‘business’ is vital – they know what your customers/users are actually trying to achieve with search, and what results should appear for a particular query. They can help you gather judgment data (this result is a good match for that query, this other one isn’t) which you can use to tune search. It’s important to remember that you are not your user – you’re probably too close to the technical implementation, whereas your user has no idea how you’ve organised your data, decided on your categories or how your search engine actually works.
Conversely, commercial teams may have priorities that are hard for you to understand – a seasonal promotion, boosting a certain category – that are difficult to implement, not obvious from the web analytics data or can cause unwanted side effects. Explaining to them the downsides of a particular fix or why something works the way it does may be necessary.
Deciding priorities for fixing search
Most search teams have a long backlog, experience pressure from above and from other teams and can struggle to keep up with fast moving new developments and operational issues. Search Product Managers are often stuck in the middle between business and technical teams.
The importance of good processes cannot be overstated here, including a clear way to identify, record and triage issues, and offline testing allowing for rapid experimentation with potential improvements. This helps prevent the team from being distracted by every panicked email with the subject “This search doesn’t work!!”, or by an executive who has read the latest LinkedIn post about how AI can fix everything automatically.
Take small but effective steps
Search is never ‘finished’ – as user behaviour, language, source data and technology constantly changes, one is always playing catch-up. However, this doesn’t mean that progress can’t be made – a team that is constantly improving search in small ways will be happier than one constantly chasing a big fix for everything (that almost certainly doesn’t exist).
We can fix search – but we can’t fix all of it today!
If you need help creating processes and methods to fix your search from someone who has worked with many leading search teams, get in touch today.
Fix Stock photos, Zero Stock photos, Arrow Stock photos, Decrease Vectors, Hand Money stock photos, Priority Vectors all by Vecteezy