Search users have long been divided into two camps. One group expresses their needs in a handful of words, assuming your search team has as many resources as Google to figure out what they actually want (no matter if they haven’t described this very well, that’s your problem!). The other is happy learning complex query syntaxes, selecting from hundreds of filters and selecting from a myriad of ways to sort the results. These rarer, expert users are the fans of Advanced Search – and their ranks may shortly be swelled by a new breed of AI Agents.
The job of Advanced Search
Advanced Searchers often work with search engines as part of their job: they may be patent analysts, legal librarians or recruitment consultants. Some of them have been searching for decades, way before search engines became available to all. They remember some searches taking minutes or even longer to complete – and this was fine, because the task was complex and the processing necessary. They may have spent years learning the quirks of a particular information system – how it implemented Boolean operators, how these interacted with word proximity, where to use wildcards (and where not to do so in case the system attempted to return every single document in the index). Unlike most casual searchers they have a deep understanding of how the source information has been categorised and filed, which helps them decide which filters to use, what documents types to choose to see and in what order.
The search expressions they create (sometimes pages long) become themselves a form of intellectual property: they may be stored, developed and enhanced over time. If the underlying search engine is upgraded or changed it’s vital these expressions are preserved and continue to work in the same way. People may even take their expressions with them when they move jobs (I’ve seen recruitment consultants jealously guard their LinkedIn search strings).
Another quality that defines our Advanced Searchers is they are focused on recall – they’re happy to scroll through pages of search results, to make sure they don’t skip past a relevant legal case, interesting candidate or relevant patent, unlike our casual searchers who will probably select one of the first few results shown.
In short, Advanced Searchers carry out complex, specialised tasks that may take some time. They may have to use multiple information systems, repeatedly choose and adjust query parameters and examine a long list of results. They don’t want to miss anything.
Making things too simple
Search engine development often fails to deal well with the Advanced Search use case. We are more likely to focus on the casual user, making their search journey as swift and simple as possible. We want them to be able to type a couple of words and get relevant results on the first page, or even short-circuit this entirely with auto-complete and auto-suggest in the search bar itself. Behind this simplicity may be complex re-ranking technology, driven by machine-learned signals, but the user will never see it – they’ll just be happy we figured out what they wanted and quickly put it at the top of the pile. We care more about precision than recall in this case, we don’t expect anyone to click through to page 2 of the results.
This approach makes sense for social media, e-commerce and web search with a high volume of search traffic. Our Advanced Searchers are probably still stuck with the same search technology they’ve had for the last decade, given that this is a smaller, more specialised market sector. I’ve also seen teams discard this use case, as ‘no-one ever uses Advanced Search’ – often a mistake, as those who do can be important, valuable (and sometimes voluble) users.
The rise of AI Agents
An AI Agent attempts to carry out a task, often using reasoning steps to achieve this. First it has to break down the user query (and in this case, the query may be long and complex and require enhancement), creating a plan for how it will answer. Next it will interrogate (search!) a variety of information systems, iterating until it has enough information, perhaps returning to the first planning stage if this isn’t easily achieved. Lastly it will summarise and synthesise some kind of response.
Obviously all of this can take significant time, factoring in the various latencies of LLMs and querying. Like a manual Advanced Search, we don’t mind if it takes some time, as it’s doing something complicated, like finding all the candidates matching a particular job specification, retrieving all their CVs, extracting some key metadata from these and creating an ordered list in a spreadsheet. Even our casual user may not care if it takes a few minutes to carry out the task “find me and the family a short holiday some time in April, but not over Easter, in a cottage on the South Coast near a good pub. GIve me a list of the options sorted by price with links to booking”.
As some have suggested, perhaps it’s time we re-thought how to build search systems, given that our users may increasingly be machines, not people:
Bringing back Advanced Search for AI
So how can we best support our new AI users? My suggestion is that we consider them as the ultimate Advanced Searchers, hungry for as many features as we can provide, but also prepared to wait for our complex queries to complete. Here’s some starting points.
All of the options, all of the time
Not all search engines have a rich set of features, or make these features easily accessible. For example, Solr often has several different ways of achieving the same thing, and it’s not always obvious which to use (see this classic post on how Solr boosting works for an example). Users of Elasticsearch and OpenSearch may end up with many pages of JSON describing their queries, which can be hard to grasp. Vespa has its own YQL syntax that may be the most flexible of all the current engines, allowing one to combine traditional data search, text search, vector search, geographical search and more. Some of the newer vector-focused search engines don’t support some of the ‘old school’ text search features very well, if at all. To support the advanced search use case, we need advanced options – so let’s make these visible, accessible and extensive!
Use AI to help AI
Finding a way to expose all of this functionality to AI Agents will be a challenge. We could consider an AI-powered approach that can ‘learn’ how to access a set of search engine features, perhaps trained on documentation: however the existing human-readable resources for learning the features of various search engines are focused on the simple use cases, with the detail of seldom-used features often buried in the documentation, or found only by painful experience. Interestingly Weaviate have an AI trained on their documentation answering questions in the #ask-ai channel of their Slack.
Connecting Advanced Search to AI
So given we can ask AI to help, what should this interface look like? Are there already standards we can adopt? There are at least some interesting candidates:
Search-R1
The Search-R1 paper published a few weeks ago proposes a new way to interact with search engines during the steps of a LLM reasoning process. Wrapping the query and response in certain tags it allows an engine to be called multiple times, illustrated by this prompt:Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search>, and it will return the top searched results between <answer> and </answer>. You can search as many times as you want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer> without detailed illustrations. For example, <answer> xxx </answer>.
The associated Github project turns what is supplied in the <search> block into a query for a number of engines including Lucene, FAISS and Google. However I can see no support for any kind of advanced search – this is just a simple text string as a query – not that this couldn’t be done by expanding the idea above. Could Search-R1 be the genesis of a generic Advanced Search interface?
Interestingly Perplexity are already using Search-R1 for their Perplexity Pro offering – and Perplexity is a Vespa customer.
Machine language for search
At the Search Solutions conference a couple of years ago, Grace Lee from Reuters asked an interesting question: why do RAG systems have to communicate in text with the underlying retrieval engine?
As LLMs are built on human language, the way we integrate them with other systems is by default verbose and readable. Most search engines take a text-based query and return a text-based response, perhaps with some structure e.g. JSON. However, do our AI Advanced Searchers need to do this? Perhaps it’s time to think about a machine-readable language for search queries and responses, although this will obviously have an impact on debugging and explainability.
To MCP or not MCP
Model Context Protocal (MCP) is a standard proposed as a way to connect AI to data systems. Although it was created in late 2024 it’s only now that it’s gaining momentum (and six months is a long time in AI!). There are however some significant concerns around security and scaling – and with the speed that everything around AI is being built, corners may often be cut. However with big hitters like Anthropic, Microsoft and IBM supporting this work, MCP may become the default and it would be interesting to see some prototype implementations of search-backed MCP servers.
The end of Advanced Search?
I wonder if the rise of AI Agents also foretells the end of the era of Advanced Searchers. If a casual user (say a jobseeker) can now use an AI-powered system with all the underlying power of advanced search, do we even need the expertise of a recruitment agent?
Even with our current rush to AI, replacing decades of human experience in using information systems is a huge and perhaps impossible task. I prefer to think of AI agents as assistants rather than wholesale replacements, enabling our Advanced Searchers to carry out their tasks faster and with more accuracy. I think we’ll need those Advanced Search forms for some time yet.
Need to make your search Advanced with AI? Let me help.