# Charlie Hull - The Search Juggler

> Expert Consulting in Search and AI

> Admin Email: thesearchjuggler@gmail.com


## Posts
### Why replacing your search engine may not fix your search
As a search manager or search team lead, you'll occasionally be asked to consider whether you should migrate to a new technology platform. In fact, this may come up on a regular basis, as salespeople "just check in" (in case things have changed since the last time you politely declined), or the latest Forrester or Gartner analyst report appears on your executive team's desk. 
If you're generally happy with your technology platform and unwilling to move, these approaches can be a distraction from the real work of improving search. Yes, there can be good reasons to move to a new technology, but don't discount the advantages of sticking with your current system - you know it well and you've probably made significant improvements to it. However if you are thinking of moving, here's some things to consider.
Buy versus Build
This is a never-ending question: should you build your own platform around open source technology, or buy a pre-existing commercial solution? Proponents of the former will point to how open source is freely available, whereas fans of the latter will tell you how great it is that everything comes ready out of the box. The real picture is more complicated: open source requires a significant amount of initial and ongoing engineering effort and (hopefully) active participation in the community, whereas commercial solutions will also require some engineering effort as whatever is in the box won't completely solve your problem - and there's also a fee both at the start and for ongoing support to consider. Open source isn't free, but closed source locks you into the vendor's roadmap and pricing plan.
(I've long been a proponent of open source search engines such as Apache Solr, OpenSearch or Vespa - but I can also recognise that open source isn't for everyone).
The cost of search migration
Platform migrations are risky, no matter if you're moving from open source to/from closed, or between different providers of either option. 
They always take much longer than you think: it's common to see estimates of a few months for a migration but the reality may be counted in years. At worse, you may have to back off and stop the process to prevent escalating costs.
They may reveal hidden problems: you'll be digging deep into where your data comes from, finding out those customisations done in the past that no-one remembers the reason for and finding new requirements that probably weren't considered when you decided to migrate. Beware dragons!
Don't forget learning time: your team should know your current system well, but will need time to be trained on anything new.
They may not even make search much better: your current solution may already be pretty good and the benefit of a migration (in terms of user satisfaction and business performance) less than you originally thought. Introducing more effective offline and online testing or a proper search issue triage process may be of more benefit overall. 
To reduce the risk, you need to be very clear about expectations; plan incremental, small steps - and be pessimistic and realistic about timelines.
Don't rely on analysts
Analyst reports are in my opinion not a reliable source of information about technology options. It's well known that the larger analysts operate a model where they consider only some vendors, and the ones they focus on most tend to have a commercial relationship with the analyst (it can come dressed up as 'partnership' but let's call it for what it is: pay to play, or at least pay to access). This is why most analyst reports fail to consider open source options, as these projects tend to have few financial resources (unless they're single-vendor open source, which has its own disadvantages). Startups with little history of providing effective solutions may suddenly appear in a high position as they spend some of their funding dollars. 
Analyst companies often name and create new technology categories: "Cognitive Search" was popular some years ago and of course nowadays "AI" themes are common.  Strangely the same companies seem to appear in these new categories, although they're still basically a search engine provider: it's a mutually beneficial relationship for marketing for both the featured companies and the analyst, where a new exciting market appears, the companies are shown to be leading this new market and the analyst to be providing insight into it, even if the latter invented it in the first place.
Avoiding the time sink
If you're regularly asked by management or others to consider a new technology choice, here are some ways to reduce any wasted effort:
Have a clear feature list - what you need the platform to do and what is not required. Keep this updated as your product evolves, data and user needs change. 
Ask for proof of the supplier or project's capabilities - case studies, reference customers (that hopefully you can talk to directly rather than through a vendor), performance data.
When a supplier is asked to provide a demo, remember the demo is never the product - it's carefully set up to make the product look its very best.
Ask the supplier to run a Proof of Concept, with your data not theirs. Use your own testing systems to verify this, not the suppliers.
Be suspicious of any new 'magical' capabilities - it's rare that any new platform has something that no other platform has. The supplier should be able to explain the detail (under NDA if necessary).
Get independent advice - there are plenty of experts out there to advise you, and the best have no 'skin in the game', e.g. have no relationship with any vendor that might cause them to be less than neutral.
So why move at all, ever?
It's unlikely that any piece of technology will suit you forever - even if you follow a sensible upgrade path. The world changes and we have to change with it. You should keep up with the release cycle of your current platform, even if you lag behind a little in upgrading - make sure you know what's coming next (and maybe what's a little too new to adopt). Keeping an eye on the competitors of your current platform is also a good idea - maybe they've concentrated more on something you really do need, or their pricing strategy is more favourable (this also helps when you negotiate with your current supplier!).
When and if you do decide to investigate migration to a new platform, or carry out a significant version upgrade, there is also the opportunity for a wider conversation about search. If you don't have good enough test data to evaluate new technology options, you've now got a good reason to create it. If you've not analyzed customer feedback to find out which parts of their search journey they can struggle with, now's the time. If you don't have good analytic data showing how search succeeds and fails, why not? If your team don't have the core knowledge to question the usefulness of any promised new features, buy them the books and/or training they need and send them to some relevant conferences or Meetups. Perhaps this is a better way to 'fix your search' - fix your search process and support your search people!
If you need help figuring out if, when and why to move search platforms, contact me today
Wheels Stock photos by Vecteezy

### It's your responsibility - but how do you even start fixing search?
As a Product Manager or engineering lead within a search team, it's often hard to know where to start fixing search. You'll have a list of complaints from users, management, those producing content, customer support teams; a backlog of things that you know don't work very well, plus a nagging feeling that the competition are doing search 'better', whatever that means.
Getting started is often the hardest part - deciding what is most important and how to prioritise potential fixes. I'm going to assume for the purposes of this blog that you have at least some data on what users search for - query logs - and some web analytics data showing which of these queries are performing well or badly. Here's some places to begin:
Zero results
Search queries that produce zero results are a great starting point - after all, if you can't produce any results at all, your user is probably going to assume you can't help them (even if they mistyped the query, or used a unusual term to describe what they want). They may just decide your website is 'broken' and go to a competitor. Zero Result Rates (ZRR) can commonly be 3-5% of all query traffic (although once I saw a horrendous 35% ZRR on the website of a well known UK brand). This number should be easy to extract from your query logs.
There are various ways to reduce ZRR, but the most important thing is not to leave your user staring at a page with a short message that implies 'We can't help you'. Typo tolerance or did-you-mean spelling suggestions, query relaxation, query rewriting, vector search or even displaying recommendations are all useful things to explore.
Low or zero-click results
Queries that produce results nobody clicks on are obviously concerning. There's a cunning technique to identify these queries called Click Residuals, described in this classic blog from ex-Netflix search lead Walter Underwood - calculate an average level of clicks across all your queries and then work out which are much lower than this, with 'residual' - unharvested - clicks. 
This technique will quickly show you underperforming queries, but it gets more interesting when you cluster groups of queries together (not particularly hard to do with a bit of ML). You can then end up with groups of underperforming queries - perhaps you're not doing so well at queries about animals, or 3-word queries. This helps you identify problems that affect many queries, and if you can then come up with a fix this may have a large impact.
High and low traffic
Some of your queries will be used very often, and some will be rare, further down the 'long tail' of the classic power curve.  Fix those high frequency queries and you'll be helping a lot of users - but don't ignore the low frequency queries. If you identify classes of queries to fix - perhaps using the Click Residuals technique above - then you can fix a lot of these low frequency queries in one go, and the total volume of these can be significant. 
Common techniques to fix high frequency queries include rule-based redirection, synonyms and query rewriting (e.g. if a common query is 'where is head office' then just redirect to a map, or if the query includes a term that doesn't exist in your source data use synonyms). Low frequency queries might be helped with vector search or typo tolerance (which could fix many mis-spelled queries).
Calculating value
Just considering query frequency doesn't give you a full picture of the value of each query. If you get many queries for red shoes, but you don't make much profit on shoes, then this perhaps isn't as important to fix as queries for handbags where your margin is much more significant. This is a great example of why you should work with your commercial team to identify which queries, or classes of queries, generate the most value for your business. 
Working with colleagues closer to the 'business' is vital - they know what your customers/users are actually trying to achieve with search, and what results should appear for a particular query. They can help you gather judgment data (this result is a good match for that query, this other one isn't) which you can use to tune search. It's important to remember that you are not your user - you're probably too close to the technical implementation, whereas your user has no idea how you've organised your data, decided on your categories or how your search engine actually works.
Conversely, commercial teams may have priorities that are hard for you to understand - a seasonal promotion, boosting a certain category - that are difficult to implement, not obvious from the web analytics data or can cause unwanted side effects. Explaining to them the downsides of a particular fix or why something works the way it does may be necessary.
Deciding priorities for fixing search
Most search teams have a long backlog, experience pressure from above and from other teams and can struggle to keep up with fast moving new developments and operational issues. Search Product Managers are often stuck in the middle between business and technical teams.
The importance of good processes cannot be overstated here, including a clear way to identify, record and triage issues, and offline testing allowing for rapid experimentation with potential improvements. This helps prevent the team from being distracted by every panicked email with the subject "This search doesn't work!!", or by an executive who has read the latest LinkedIn post about how AI can fix everything automatically.
Take small but effective steps
Search is never 'finished' - as user behaviour, language, source data and technology constantly changes, one is always playing catch-up. However, this doesn't mean that progress can't be made - a team that is constantly improving search in small ways will be happier than one constantly chasing a big fix for everything (that almost certainly doesn't exist). 
We can fix search - but we can't fix all of it today!
If you need help creating processes and methods to fix your search from someone who has worked with many leading search teams, get in touch today.
Fix Stock photos, Zero Stock photos, Arrow Stock photos, Decrease Vectors, Hand Money stock photos, Priority Vectors all by Vecteezy

### A new world of search engine tuning tools
Tuning search engines to return the best results for users is a constant process of experimentation, measurement and adjustment. We hope that we have enough data about what results users prefer - perhaps derived from which results they click on - but if not we often have to create our own, making judgements about which results are relevant. We use this data to test each new configuration, and iterate until we achieve a satisfoctory metric. 
Many search engines don't provide the tooling required to carry out this tuning process. A long time Lucene contributor recently admitted to me than when they worked for a search engine company some years ago they didn't spend much time on end user applications. Their world was internal, creating features on the company roadmap, rolling out that next major version - not focused on the day to day needs of those users tuning their search in a fast changing world.
The right tools for the job
Although most search engine suppliers offer consulting and support for their (paying) users, more often this is provided by external consultants (hello!) and agencies, who sometimes end up building extra tooling and processes to do so. A very good example of this is Quepid, built by my ex-colleagues at OpenSource Connections, which lets you collect relevance judgements and thus measure the quality of a set of search engine queries - it works with a range of search engines including Solr, Elasticsearch, OpenSearch and Algolia and is both open source and available as a free hosted service. Here's some manual judgements on searches for bread from a previous post, with an overall nDCG measurement for the query 'loaf':
Another open source example is Rated Ranking Evaluator from the Sease team, which sadly hasn't been updated for a while, new player SearchTweak, and there are commercial tools from companies like SearchHub. This landscape of tools is somewhat patchy: not everything works with every search engine, keeping up with new versions and API changes is a huge challenge. Often the tools used for search tuning are built in-house, a collection of scripts, hacks, spreadsheets and macros - hard to maintain and not always easy to use.
New search tuning tools in OpenSearch
I've been working with the OpenSearch team for a few years now, both while at OpenSource Connections and afterwards (I was recently proud to be asked to become an OpenSearch Ambassador). My most recent consulting engagement has also focused on OpenSearch. This has given me an opportunity to contribute to some exciting developments around search tuning tools.
The OpenSearch team includes some people who have been in the search business almost as long as I have - they have worked at both ends of the business, as product owners or search engineers for companies using search engines and at companies creating and supporting search platforms. They thus have a unique perspective and have collaborated with the team at OpenSource Connections to create an extensive suite of tools to help users experiment with, measure and tune search - all open source, but they are also available on the AWS hosted service from version 3.1. These include:
User Behaviour Insights (UBI) - an open standard for how to record user interactions with search results (what is clicked on, what is hovered over, what happens next) - so we can see the user's whole search journey. As this is an open standard it will also work with any search engine - but with OpenSearch you can also import this analytic data back into an OpenSearch index, so it's close to where you need it (rather than the more common situation where search analytic data is provided by different systems and a different team, which can be hard to get, slow and out of date). I've spoken about UBI at a few recent search conferences and as a champion of the project it's hugely encouraging to see it supported in OpenSearch.
Search Relevance Workbench (SRW) - a set of OpenSearch dashboards showing you useful metrics like zero result searches, the difference between the result lists from two different search configurations, etc. Importantly SRW can use data provided by UBI and can also run search relevance experiments : provide a set of queries and relevance judgements and you can see if a new search configuration does better than the old, with metrics including nDCG. Although it doesn't let you create manual relevance judgements, it's compatible with Quepid so can import these on demand.
Elasticsearch joins the search tuning party
This week also saw an announcement from an Elastic engineer that he had created Relevance Studio, which looks like a similar project to Search Relevance Workbench. It's apparently not an 'official' project or Elastic product and unfortunately uses Elastic's own not-exactly-open-source license, but it has some interesting features including an MCP server and the ability to automate the tuning lifecycle. Unlike OpenSearch's SRW it also lets you make human relevance judgments.
I hope that Elastic see the value of this project and support it further. Compatibility with UBI and with Quepid's export format would be a nice to have (making human judgments is tedious, so re-using them where possible is essential).
Ready to self-tune? Not quite
As someone who regularly helps companies develop processes and methods for effective search tuning, these developments are very exciting. Finally we have tuning tools as fully integrated parts of the search engine platform, at least for OpenSearch. Grab that click data, use it to figure out which configuration is best or even feed it into a ML system that learns the best combination of settings automatically.
We're not quite at the point where we can point our search engine at some content, some example queries, a user analytics framework etc. and let it figure out how to configure itself - but we're making some significant steps to that particular Nirvana. In the meantime we can be much more effective, spend less time hacking together internal tooling and focus more on what's important - giving our users the very best results for their queries to drive our businesses forward.
If you'd like to find out how any of these tools could help you build better search, let me know.
Tool Stock photos by Vecteezy

### A holistic approach to search engine consulting
I'm currently working as a fractional Search Lead at Moonpig, the first company to sell greeting cards and related gifts online in the UK. The job title is somewhat vague, so I thought I'd give you a general insight into how I approach consulting around search & AI - and show you it's not just about technology. What I write below isn't specific to Moonpig and is based on my wider experience in search engine consulting.
Learning the World
The first thing any consultant has to do when starting a new engagement is to learn the world of the organisation they've been brought in to help. Every organisation is different, even those operating in the same sector and of relatively the same size. You need to understand the organisation's purpose (selling greeting cards, or providing medical information, or assisting lawyers), rough structure, the internal language and terms, business processes and how people around search interact. I'm not a lawyer, doctor or greeting card expert, but I need to know enough about these sectors to figure out how great search can help.
There are several ways to learn: from structured approaches where you systematically ask a series of questions of the search team, to reading background documents and/or shadowing key individuals. The principle that 'no question is a stupid question' applies here - and you may need to ask many stupid questions! I make copious notes during this process, often record online calls and I'm beginning to use AI transcription tools to assist me. I'm relaxed about the fact that I'll probably repeat myself occasionally and I try to stay humble: after all, my new colleagues know much, much more about their organisation than I do, especially about the special cases, workarounds, shortcuts and hacks that always exist. Being judgemental ('why on earth have you done it that way?') isn't at all helpful - there are usually good reasons why things are set up a particular way, and there is always technical debt, missed opportunities and an endless backlog of things to fix one day, when there is time. 
If appropriate, one might provide a list of written questions to be answered ahead of the start date - but remember the answers may not be definitive, or can be just plain wrong, as you might discover when you actually arrive. 
Plugging in
Fitting into an existing team is always difficult: there may be suspicion of your consultant status, existing social norms you don't understand, processes and rituals that seem to make no sense - my approach is always to try to disturb things as little as possible. If there's a regular standup or other meetings then I start attending them, but mainly to listen. No-one wants to hear your grand opinions when there's a Kanban board to be got through. 
If you're lucky (and I have been at Moonpig) the onboarding process will be smooth, and you'll have the various logins, equipment and other things you'll need to participate on day one. You'll also be grateful for friendly greetings, occasional check-ins to make sure you're settling well and invitations to meet the team in more informal settings. This 'social glue' helps immensely, but some organisations struggle with it - often because there are many more urgent things to do. Politeness, patience - and an understanding that yours are not always the most urgent questions to answer - are vital. 
Wait and see
It's quite tempting to come up with a Grand Plan within days of arriving at a new engagement. In glorious technicolor detail, this will show how your recommendations will change everything, please all the customers, make more money and generally Do Search Right. However the reality is that you simply don't know enough yet to go into any detail. I expect it to take a good few weeks before I can recommend anything more than general initiatives - for example, I'll certainly recommend offline relevance testing (if it's not being done already) but I have no idea what queries we'll test, what tools we'll be able to use and how often we'll be able to do it. That said, a general outline of the areas you hope to work it will be useful, if for nothing else to check against your client's expectations.
Get your hands dirty
At some point you'll want to figure out what's really going on with search. With some experience, you can try out various types of queries and identify whether or how certain features have been implemented - for example, how does search cope with typos, are there some dead ends where you hit zero results, do the facets make sense. You can do most of this with the main search interface, but at some point you're going to have to go deeper.
I have a good general understanding of how search works at a low level - terms, frequencies, indexes etc. - and can find my way around most search engine specific query options (usually Domain Specific Languages or DSLs, or collections of named parameters). With a suitable test bed (for example Quepid or the OpenSearch DevTools console) I can try out actual search queries using a provided example, fiddle with parameters, chop bits out and see what happens. This can be a great way to find odd bugs, part-implemented features and ways in to improve search quality: but remember there may also be very good reasons why something is how it is. This will lead you to more questions for the team: why is that boost level 15.5? Where do the vector embeddings come from? Why aren't we searching this text field?
You'll also need to see the code that actually calls the search engine - I'm lucky that over a long career in IT I've worked with everything from assembler to Java, C++ to scripting languages, so I can usually figure out what code is doing (just don't ask me to read LISP). Once I've pulled things apart (and asked more stupid questions) I can make some more concrete recommendations for improvement. I'm unlikely to write any code - the team should do this, as they'll have to maintain it - but I can show examples and general patterns.
It's also worth looking into how the index is generated from content, as content quality is the Achilles heel of so many search applications. Wear your protective goggles for this task!
Education, education, education
One of the main tasks of a consultant is to teach - and this is necessary at many different levels. You may have to explain high-level search concepts to an executive team, how spelling correction differs from typo tolerance to a product owner or how the Snowball stemmers work to an engineer.  If you don't know the answers they seek, do some research rather than glossing over the subject. No-one will expect you to be the all-seeing oracle but you are a search expert so you should at least know where to look! 
More formal training is also an option - if you have experience and can provide this (and the team have time available) it can be a great way to level up search knowledge, especially for those relatively new to search and information retrieval. 
I like to provide a rolling list of useful education resources - conferences to go to (I maintain a calendar of these), books to read, relevant links to follow, people to be inspired by - and remind people about this regularly, mentioning any exciting new additions. I'm very willing to recommend resources created by other search consultants, many of which I hugely admire - here's a great example from Peter Fries, the 'penny farthing' of offline and online testing from his Haystack 2018 talk:
Be enthusiastic 
One of the most vital things in my view is to be enthusiastic about what can be done to improve search - perhaps with better testing, or an upgraded infrastructure, or (of course) with AI. No-one wants to hear that everything is broken, they're years behind the competition and nothing can be done. Be realistic about the challenges and don't over-promise, but with your experience guiding them any search team should be re-invigorated by your suggestions and ideas. There's always something more that can be done to improve search - it's a never-ending journey!
The search consulting endgame
The aim of any search consultant should be to leave things in a better state than before they arrived - but remember not all your suggestions will be implemented, often for very good reasons of time and budget. If you have made a real difference to search quality, then hopefully both you and the client will be able to announce this publically, which can be hugely satisfying. 
However one should also remember that it's people that really matter - so if you've helped others gain knowledge and experience that will serve them in the future as they move towards search Nirvana, this can be a great reward.
If you need help building better search, then get in touch to discuss a future engagement.
Audit Stock photos by Vecteezy

### AI Agents are the new Advanced Searchers
Search users have long been divided into two camps. One group expresses their needs in a handful of words, assuming your search team has as many resources as Google to figure out what they actually want (no matter if they haven't described this very well, that's your problem!). The other is happy learning complex query syntaxes, selecting from hundreds of filters and selecting from a myriad of ways to sort the results. These rarer, expert users are the fans of Advanced Search - and their ranks may shortly be swelled by a new breed of AI Agents.
The job of Advanced Search
Advanced Searchers often work with search engines as part of their job: they may be patent analysts, legal librarians or recruitment consultants. Some of them have been searching for decades, way before search engines became available to all. They remember some searches taking minutes or even longer to complete - and this was fine, because the task was complex and the processing necessary. They may have spent years learning the quirks of a particular information system - how it implemented Boolean operators, how these interacted with word proximity, where to use wildcards (and where not to do so in case the system attempted to return every single document in the index). Unlike most casual searchers they have a deep understanding of how the source information has been categorised and filed, which helps them decide which filters to use, what documents types to choose to see and in what order.
Google's Advanced Search options
The search expressions they create (sometimes pages long) become themselves a form of intellectual property: they may be stored, developed and enhanced over time. If the underlying search engine is upgraded or changed it's vital these expressions are preserved and continue to work in the same way. People may even take their expressions with them when they move jobs (I've seen recruitment consultants jealously guard their LinkedIn search strings). 
Another quality that defines our Advanced Searchers is they are focused on recall - they're happy to scroll through pages of search results, to make sure they don't skip past a relevant legal case, interesting candidate or relevant patent, unlike our casual searchers who will probably select one of the first few results shown.
In short, Advanced Searchers carry out complex, specialised tasks that may take some time. They may have to use multiple information systems, repeatedly choose and adjust query parameters and examine a long list of results. They don't want to miss anything.
Making things too simple
Search engine development often fails to deal well with the Advanced Search use case. We are more likely to focus on the casual user, making their search journey as swift and simple as possible. We want them to be able to type a couple of words and get relevant results on the first page, or even short-circuit this entirely with auto-complete and auto-suggest in the search bar itself. Behind this simplicity may be complex re-ranking technology, driven by machine-learned signals, but the user will never see it - they'll just be happy we figured out what they wanted and quickly put it at the top of the pile. We care more about precision than recall in this case, we don't expect anyone to click through to page 2 of the results.
This approach makes sense for social media, e-commerce and web search with a high volume of search traffic. Our Advanced Searchers are probably still stuck with the same search technology they've had for the last decade, given that this is a smaller, more specialised market sector. I've also seen teams discard this use case, as 'no-one ever uses Advanced Search' - often a mistake, as those who do can be important, valuable (and sometimes voluble) users.
The rise of AI Agents
An AI Agent attempts to carry out a task, often using reasoning steps to achieve this. First it has to break down the user query (and in this case, the query may be long and complex and require enhancement), creating a plan for how it will answer.  Next it will interrogate (search!) a variety of information systems, iterating until it has enough information, perhaps returning to the first planning stage if this isn't easily achieved. Lastly it will summarise and synthesise some kind of response. 
Obviously all of this can take significant time, factoring in the various latencies of LLMs and querying. Like a manual Advanced Search, we don't mind if it takes some time, as it's doing something complicated, like finding all the candidates matching a particular job specification, retrieving all their CVs, extracting some key metadata from these and creating an ordered list in a spreadsheet. Even our casual user may not care if it takes a few minutes to carry out the task "find me and the family a short holiday some time in April, but not over Easter, in a cottage on the South Coast near a good pub. GIve me a list of the options sorted by price with links to booking". 
As some have suggested, perhaps it's time we re-thought how to build search systems, given that our users may increasingly be machines, not people:
https://twitter.com/jobergum/status/1909687138532176195
Bringing back Advanced Search for AI
So how can we best support our new AI users? My suggestion is that we consider them as the ultimate Advanced Searchers, hungry for as many features as we can provide, but also prepared to wait for our complex queries to complete. Here's some starting points.
All of the options, all of the time
Not all search engines have a rich set of features, or make these features easily accessible. For example, Solr often has several different ways of achieving the same thing, and it's not always obvious which to use (see this classic post on how Solr boosting works for an example). Users of Elasticsearch and OpenSearch may end up with many pages of JSON describing their queries, which can be hard to grasp. Vespa has its own YQL syntax that may be the most flexible of all the current engines, allowing one to combine traditional data search, text search, vector search, geographical search and more. Some of the newer vector-focused search engines don't support some of the 'old school' text search features very well, if at all. To support the advanced search use case, we need advanced options - so let's make these visible, accessible and extensive!
Use AI to help AI
Finding a way to expose all of this functionality to AI Agents will be a challenge. We could consider an AI-powered approach that can 'learn' how to access a set of search engine features, perhaps trained on documentation: however the existing human-readable resources for learning the features of various search engines are focused on the simple use cases, with the detail of seldom-used features often buried in the documentation, or found only by painful experience. Interestingly Weaviate have an AI trained on their documentation answering questions in the #ask-ai channel of their Slack. 
Connecting Advanced Search to AI
So given we can ask AI to help, what should this interface look like? Are there already standards we can adopt? There are at least some interesting candidates:
Search-R1
The Search-R1 paper published a few weeks ago proposes a new way to interact with search engines during the steps of a LLM reasoning process. Wrapping the query and response in certain tags it allows an engine to be called multiple times, illustrated by this prompt:Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, if you find you lack some knowledge, you can call a search engine by <search> query </search>, and it will return the top searched results between <answer> and </answer>. You can search as many times as you want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer> without detailed illustrations. For example, <answer> xxx </answer>. 
The associated Github project turns what is supplied in the <search> block into a query for a number of engines including Lucene, FAISS and Google. However I can see no support for any kind of advanced search - this is just a simple text string as a query - not that this couldn't be done by expanding the idea above. Could Search-R1 be the genesis of a generic Advanced Search interface?
Interestingly Perplexity are already using Search-R1 for their Perplexity Pro offering - and Perplexity is a Vespa customer.
Machine language for search
At the Search Solutions conference a couple of years ago, Grace Lee from Reuters asked an interesting question: why do RAG systems have to communicate in text with the underlying retrieval engine? 
As LLMs are built on human language, the way we integrate them with other systems is by default verbose and readable. Most search engines take a text-based query and return a text-based response, perhaps with some structure e.g. JSON. However, do our AI Advanced Searchers need to do this? Perhaps it's time to think about a machine-readable language for search queries and responses, although this will obviously have an impact on debugging and explainability.
To MCP or not MCP
Model Context Protocal (MCP) is a standard proposed as a way to connect AI to data systems. Although it was created in late 2024 it's only now that it's gaining momentum (and six months is a long time in AI!). There are however some significant concerns around security and scaling - and with the speed that everything around AI is being built, corners may often be cut. However with big hitters like Anthropic, Microsoft and IBM supporting this work, MCP may become the default and it would be interesting to see some prototype implementations of search-backed MCP servers. 
The end of Advanced Search?
I wonder if the rise of AI Agents also foretells the end of the era of Advanced Searchers. If a casual user (say a jobseeker) can now use an AI-powered system with all the underlying power of advanced search, do we even need the expertise of a recruitment agent? 
Even with our current rush to AI, replacing decades of human experience in using information systems is a huge and perhaps impossible task. I prefer to think of AI agents as assistants rather than wholesale replacements, enabling our Advanced Searchers to carry out their tasks faster and with more accuracy. I think we'll need those Advanced Search forms for some time yet.
Need to make your search Advanced with AI?  Let me help.
Silhouette Vectors by Vecteezy

### A new Search Meetup for London
I've been involved in running search-themed events for many years - from helping with Enterprise Search Europe (2011-2016), Haystack in the US, London and Berlin (2018-2024) and in between many smaller events such as Search Hackdays and Search Meetups. The latter remain my favourite kind of event  - small, friendly, local and a great place to build a community of search professionals and enthusiasts. Over the years I've run the Cambridge Enterprise Search Meetup (highlights included a visit by father-of-BM25 Stephen Robertson and the regular blues jam in the pub downstairs afterwards), assisted with the Enterprise Search London Meetup and most successfully ran the Lucene/Solr London Meetup. Some of these events would regularly attract over a hundred people to venues including Bloomberg's amazing lecture theatre (where we had actual waiters serving snacks!) and Barclay's Canary Wharf building with an amazing view over the Thames. 
The COVID-19 pandemic had a huge impact on in-person events - even now I feel attendance has never really recovered - and so far I've not restarted a UK Search Meetup, although I've attended a few such as Elastic London and the London Information Retrieval Meetup run by the Sease team. I've also failed to find many that cover that particular intersection of AI and search we're living through - where search folks are adding AI, and AI folks are rediscovering search and information retrieval. I want to bring these two groups together to learn from each other and make connections.
A team effort
It's thus been really exciting to find some others keen to explore this space, in particular the OpenSearch team and Eliatra (who support OpenSearch with hosting and services). Although a London OpenSearch Meetup group has existed for a short while, together we decided to widen the focus of this group to cover not just OpenSearch, but other search engines, and also AI subjects (that have some relation to search, of course). We hope by doing this to attract a wider audience, allow some cross-fertilisation between topics of interest to us all, introduce new (and perhaps old) technologies, tips & tricks and find out what we really mean by AI-powered Search. Thus we're very happy to invite you to join the London Search & AI User Group!
At our events we'll have:
Two amazing talks on search & AI subjects
A chance for Q&A on both the talks and on more general subjects - need help with your search project? Want to find out what's coming next? Ask away!
Drinks and snacks (usually pizza)
Networking - find a friend, meet people facing similar challenges, even find career opportunities or potential collaborators
Swag (from some of our hosts and speakers)
For those who would like to, a chance to meet in a pub nearby afterwards. 
If possible we'll stream the event for those who can't make it in person, and we'll also try to record the talks (this will depend on the venue).
How you can help
We're going to aim to run an event every two or three months - but to make this happen, we need your help in two ways:
We need hosts - somewhere to hold a Meetup, for at least 50-60 people, in central London, on an evening in the week between 630pm-9pm (ish). We'll need a screen or projector to borrow for our talks. Hosts are welcome to suggest a talk or at least give a short introduction themselves (it's a great place to recruit!). If you can provide some snacks/pizza and drinks that would also be great - but if not, we may be able to find someone to sponsor these separately. You'll be able to put your logo and link on the Meetup group of course.
We need talks - on any aspect of search or search & AI - on indexing, data conditioning, relevance, operations, interesting projects, amazing applications, scaling, different search engines (open source or commercial), RAG, vector search, search with or without LLMS, old school tricks and cutting edge ideas. Just don't make it a sales pitch! (If you've never given a talk before, a Meetup is a great place to start - and I'm happy to give advice and help to new presenters, just ask).
If you can help with either speaking or hosting please let us know using this form (I'll also be asking for help at our first event).
Our first Search Meetup - April 3rd 2025
The first meeting of our new group will be at 112/114 Middlesex Street, near Liverpool Street Station, from 6.30pm-9pm on April 3rd 2025, kindly hosted by EPAM. Speakers will be myself - showing you how to test search on any website, and Nate Boot of OpenSearch. We'll have Q&A too, so get those questions ready. Space may be limited so register soon!I'm hoping to see some old friends and meet many new ones - see you there!
If you need help planning, hosting or running a search-related event - or a speaker, let me know.

### Making search relevance judgements is BORING - so let's get AI to do it!
I recently wrote about a way to make Quepid, the relevance testing tool, work with pretty much any website. I did this by hacking Quepid's ability to work with a nice tidy Search API that returns JSON, instead using Javascript to pull field data out of raw HTML. The next step is to create relevance judgements - manually score the returned results as good, bad or something in the middle - a task best done by experts in whatever field your website or service covers.
There's a minor problem with this: judging relevance is not very exciting. In a perfect world, we'd have an army of domain experts able to devote their time to marking up thousands of search results. We'd be able to return to these experts regularly, to ask them to judge any new results that appear due to our improvements to the search algorithm. We'd have enough of them to figure out biases due to their differing opinions, because of course they wouldn't always agree.
In the real world however, this is a difficult thing to achieve. If you're running legal search, you probably need people with a legal qualification - and these people are busy (and can charge clients a lot of money for every 15 minutes of their time). If you're selling DIY equipment you need someone who understands the difference between emulsion and gloss paint, but they may be be serving customers in a shop. I've seen plenty of teams struggle with finding domain experts and convincing them to do some judgements (although rewarding them sometimes helps - a prize for the most productive judge?). Quite often it's the search team themselves that end up doing the majority of the judging, which comes with a risk of confirmation bias and again, isn't very exciting for them.
An AI Judge to the rescue
Perhaps we can use an AI to create some relevance judgements? This isn't a new idea of course, it's quite common to use one AI system to evaluate the output of another system. There's a risk of 'marking your own homework' of course, so definitely don't mark one LLM with the same LLM (I'm sure Claude is a great fan of Claude) - but with sufficient guardrails and some manual validation it seems a valid strategy to massively expand how many relevance judgements we can carry out.
Quepid recently added a feature to allow AI-powered judging of search results. This currently uses OpenAI - so to try it out you're going to need an API key and some credits (luckily Quepid itself is entirely free).
Cases, Books and Teams in Quepid
There's a few concepts in Quepid I should probably explain before going any further.
Case
A case is a set of test queries which might be aimed at testing a particular area of search. Let's say we're trying to figure out why two-word queries about bread don't work very well - we might have a Case with sour dough, french baguette, sliced loaf etc. as our queries:
Cases are one of the earliest concepts in Quepid, from back in the day when it only worked for one user. Each user can have lots of Cases. Each Case can be tied to a different search endpoint if necessary, with different search parameters, but you can also easily clone them to make multiple Cases work with the same backend configuration. 
Teams
A team is a group of judges, who collaborate on a particular task. Teams are how you share Cases (and some other things like search endpoints). Here's my new Team (where I'm a little lonely at present, but notice I can add an AI judge to keep me company!):
Books
A book is a collection of judgements, which can be carried out by a group of different people. Books can be shared between Teams and can be linked to one or more Cases, from which they import query/document pairs - these are the things that requires judging (basically 'is this document relevant to this query'). Books also have some great statistics to show how complete the judging process is - great for working with a Team. I'm going to create a Book, and then show how to populate it from a Case:
Creating an AI for relevance judgements
First we're going to spin up an AI Judge, which will join our Team in Quepid. This is a pretty simple process: go to our Team and click Create AI Judge. I've given it a name (because Homer's favourite saying is.....) and entered an OpenAI key. There's a sample prompt for OpenAI which we'll leave alone for now:
We now need to populate our Book with some query/document pairs from a Case: first we select the Case 'Two Word Baking' and click the Share menu option at the top, and share this case with our 'Master Bakers' Team. Now click the Judgements menu option at the top, and we can populate the Book 'Let's Make More Dough': select this Book, check the Populate Book option and then click the green button at bottom right: 
 Selecting our Book, we can watch it being populated with the query/document pairs (this takes a little while):
Next we need to click Settings and allow our AI judge to work with this Book:
Click 'Update Book' and we're ready to run our AI judge!
Go, Homer, go!
To create the AI-powered relevance judgments we select the Judgement Stats option in our Book. The Prepare to Judge button lets us start the process:
 On the dialog box that pops up we select Judge All Pairs and then click Judge Documents:
...and we're off! Once the process is finished we can click Judgements and see what happened - click the number in the ID column to see more information, including an explanation of why OpenAI made the rating (don't click anything on this page unless you want to change the rating!):
Returning to our Case we can see the scores, rolled up into overall metrics - it seems we need to do some work to improve our bread-related searches:
Can we trust Homer - and can we make him better?
Although it's exciting to see an AI automatically creating relevance judgments, we can't assume these judgements are perfect. It's worth reviewing the judgements against some kind of ground truth - in this case, someone who knows about food - to make sure we can rely on the data.
In general, there is some evidence that AI-powered judgements are reliable, as my ex-colleague Scott Stults writes in Using GPT for Relevancy Judgements.
Adding more context
Another issue is that we may not be providing enough information to OpenAI to make a judgement. We may have to work on our data extraction (more Javascript!). Some things that (in my limited experience) may help are:
 Retrieving pictures - useful for datasets where visual cues are important, for example books or fashion - OpenAI is able to interpret the picture data, for example it may spot that a dress has a flower pattern even if the dress isn't called 'floral'
Retrieving data pointed to by a URL. If one of the returned search fields is a URL - for example, a link to the entire document when only a snippet is provided in the search results, or a product page with many more details - you can ask OpenAI to follow this URL and consider this when judging. For example, edit the AI Judge settings in the Team page and add something like this to the prompt:"If there is a field in the document called 'refurl', follow this link and also evaluate this page."
Watch your spend!
Also remember to watch your API credits - even judging a few cases with an AI to write this blog cost around $0.30 to use OpenAI.
Conclusion
AI-powered relevance judgements give us a way to quickly ramp up our ability to test search and help us get past the 'cold start' problem often encountered, where human judges are unavailable or unwilling. It's exciting to see this feature available in Quepid - and it's very easy to get started.
In the future, we hope to see other AI providers become available in the tool - perhaps a locally-hosted LLM would reduce the overall cost, and if this was fine-tuned for the domain we're working with it might also improve the overall judgement quality. 
If you need help setting up your relevance judgement process & adding AI to the mix then please get in touch.
Ai Generated Stock photos by Vecteezy

### Hacking Quepid to test search relevance on any website
If you know your search is broken (often because lots of users and colleagues are complaining about it) the first step is to come up with some kind of measurement of just how bad it is. Only then should you start to fix it - perhaps by improving the source data, or tweaking the search engine configuration - as otherwise it's hard to tell if you're making things better or worse. You need to start to actively test search relevance, which hopefully will become something you do on a regular basis.
Quepid is a free, open source tool that I first became involved with in 2017 while leading Flax, a UK search consulting company. We had a client who was testing Apache Solr as a potential upgrade from the Endeca search engine, and I worked with the team at OpenSource Connections (OSC) to move Quepid from a tool used by a single developer to a tool used by whole teams. Later on after I joined OSC, Quepid became the centerpiece of the relevance testing process and I continued to contribute to its development and use it in client projects.
Quepid lets you connect to a search engine, create test cases including sets of queries and lets you manually rate how relevant the results from these queries are. Roll all these ratings up and you have a search metric - a number that shows you how good (or bad) search is on your site.
As Quepid was first built to work with the Apache Solr search engine, it usually relies on a direct connection to the search backend - which can take a while to set up. You might need permissions, tunnels, proxies and some developer time. The tool now works with many other search engines including OpenSearch, Elasticsearch, Vectara and Algolia, but you’ll still need help and the right access to get it working. 
Mis-using Search Endpoints
However, if you don't have access to the search engine backend, want to get started quickly, or you don't even know who runs the search function for a particular company, what can you do? Donning my black hoodie, I've come up with a hacky way to connect Quepid to potentially any website.
Quepid lets you create search endpoints that rely on a particular search engine or a HTTP Search API. Send a search request to this API and back should come some tidy JSON containing a set of search results. A lot of sites have a Search API just behind the front end - but again, you may need help to gain access to it.
What I'm going to demonstrate is how to utilise this Search API feature of Quepid to work directly with a website front end. It's not pretty, it's not a complete solution but it's enough to get you started. It will involve some inspection of website traffic, some Javascript coding and a little guessing.
There are some other options, which would involve writing a lot more code and/or configuration:
Write a web scraper or use a commercial scraping platform to send queries, grab the search results and turn them into a CSV file, then import this into Quepid manually (CSV import is another Quepid search endpoint option, but this is an offline process and thus not so interactive). 
Create or extend a web server that sends the search requests to the website, reformats the results then presents these on a Search API that can be easily hooked up to Quepid. Here’s a blog on how to do this with .NET.
What’s going on when we search?
I'll use a UK website as my testbed - Brakes, a major food distributor. Let's start by taking a look at how search works on their site - my test query is loaf:
As you might expect, we get a set of results, in this case presented in a table. Somehow we need to get these results out of the page and into Quepid, ready to rate them. 
Let’s first use the browser’s developer tools to inspect what’s going on when we make this search. Bringing up the tools (on Firefox for Windows it’s Ctrl-Shift-I) and selecting the Network tab, then refreshing the page, shows lots of traffic. We’re looking for some kind of search query - here it is, a HTTP GET request with the query word loaf:
If we click this line, select the Response tab on the right, and slide on the Raw switch, we can see the actual HTML sent back to the browser when the search happens. Somewhere in this should be our search results - let’s click into this HTML, Ctrl-F to find and look for ‘sidoli’ as that’s a pretty unique word from the first search result:
In this case, it looks like the search results are in a block of Javascript. This won’t always be the case of course - they could be formatted as HTML - but let’s see what we can do with this data.
Creating our endpoint
Let’s fire up Quepid and create a new search endpoint for this website. We’re going to pretend that Brakes’ website URL is actually the URL of a Search API - we’ll need to look back at the test search we just did and grab anything before the ‘?’. We’re using HTTP GET requests here: 
Make a note of what comes after the ‘?’ - in this case text=loaf - we’ll need this later.
Quepid uses Javascript to translate between what a Search API returns and how Quepid represents search results - there are two template functions provided, one to return a count of the number of search results and the other to return an object with the actual results. You can see this in the template code provided, where data is the response from the API:
numberOfResultsMapper = function(data){
   return data.length
};
docsMapper = function(data){
  let docs = [];
  for (let doc of data) {
     docs.push ({
       id: doc.publication_id,
       title: doc.title,
     });
  }
  return docs;
};
Note that this code assumes that data is a nice tidy bit of JSON, containing a set of result documents - this is certainly not what we’re getting back from the Brakes website! We’re going to have to add a Javascript function to extract our search results from the raw HTML we saw above and turn them into a format Quepid understands.
Extracting search results - the fiddly bit
In this case I’ve used some simple string manipulation to chop the search results out of the returned HTML and then Javascript’s eval() function to turn them into a Javascript object. I can’t pretend to be a Javascript expert, so my code may not look pretty but it does work, and you can see the various steps.
When you try to do the same for another, different website, you’ll need to write your own code to reformat whatever HTML is returned - perhaps you’ll have to strip out certain characters, traverse the DOM or whatever. I leave that as an exercise for the reader! 
For debugging, Quepid provides a simple syntax check with a red indicator at top left of the Javascript box when something is wrong. If you need to trace what you’re doing when trying to use your endpoint in Quepid, use Javascript’s console.log() function and Developer Tools to view the output. Here’s my Javascript with an extractor function:
myExtractor = function(data){
	const starttoken = "window.productListObject = [";  // the start of the Javascript block containing search results
	const endtoken = "]"; // the end of this block
	obj = [];
	str = data; // get all our HTML
	str2 = str.substring( str.indexOf(starttoken)); // chop off the start
	str3 = str2.substring(0,str2.indexOf(endtoken)+endtoken.length); // chop off the end
	obj = eval(str3); // turn this into an object
//	console.log(obj) // << use this to check the structure!
	return obj;
};
numberOfResultsMapper = function(data){
  myExtractor(data).length;
};
docsMapper = function(data){
  let docs = [];
  for (let doc of myExtractor(data)) {
	docs.push ({
 	id: doc.id,
 	title: doc.name, // map the object names into Quepid fields
 	price: doc.price,
 	product_id: doc.id
   });
  }
  return docs;
};
To make sure you’ve created an object that Quepid expects you can even print the object to the console (see the commented out line above marked with << , just before the end of my extractor function):
Creating a Quepid Case
We can now create a test case using our new endpoint to talk to the Brakes’ website. From the Quepid top menu, select ‘Relevancy Cases’ and ‘Create a case’ to start the wizard. Give your case a name, select the endpoint we just created. Select fields to match the ones we used above:
Add loaf as our first query. You’ll now get a stern warning:
To fix this, click ‘Finish’ and now the button. We need to tell Quepid how to send queries to the Brakes website in the correct format: remember we noted above that this is text=loaf - we replace ‘loaf’ with a special token that Quepid will use for each query:
Click ‘Rerun my Searches!’ at the bottom and we should now see some search results ready for rating - if this doesn’t work and you get an error, the Search Endpoint isn’t working correctly and you’ll need to go back and fiddle with Javascript - remember to pop open the developer tools to see more details of what happened. Here’s what you want to see:
Note that we’re only showing the information we could easily extract from the HTML - no pictures or links to actual products - and there’s no Explain data on the right to show us why this result matched, as would be provided if we were talking directly to some search engines like Solr. There are also a few odd characters in the titles! However, we can now rate these results and get a metric for how good our ‘loaf’ search is (here we’re using nDCG) and also add some more queries to this Case which will be run automatically on the Brakes’ site:
We can now try some different search configurations - and see if we can achieve better scores! Quepid can re-run all the tests automatically and re-uses any previous ratings.
Enhancements
Those with superior Javascript skills to mine should be able to come up with much better ways to extract the data Quepid needs - pictures are very useful when rating, for example, but I couldn’t figure out where they were being returned on the Brakes site. Many sites will return search results in HTML blocks which will require a lot more parsing - perhaps with the native DOMParser interface. Again, I suggest you use the developer console for debugging, you’re going to need it!
Do bear in mind that the documentation for Search Endpoints is rather thin. You may be able to get support in the #quepid channel in Relevance Slack where the Quepid developers hang out (thanks Eric Pugh for reviewing this blog and for stewarding Quepid).
Figuring out which queries to test is another issue - if you have access to query logs, or a list of problematic queries for a particular site, you should be able to build a test set. Sampling is a good approach.
Now we can hook up Quepid to any website, even AI-powered search is testable - we really don’t care if search is lexical, semantic or a hybrid of both. We could even use Quepid to rate the results of a Retrieval Augmented Generation (RAG) system, where just a single answer is generated.
Remember that sending a large number of automated searches to a website may make you look like a bot, and risk you getting your or Quepid’s IP blocked - so let’s be careful out there!
Test search relevance on any website
I’ve shown how we can use Quepid’s Search Endpoints feature and a scrap of Javascript to test search relevance on potentially any website. If you’d like to start measuring how good, or bad, your search is - without bothering your developers - get in touch and I can show you how to develop effective processes and tools to iteratively improve search quality.
Hacker Vectors by Vecteezy

### Bridging the moat - why open source AI is a disruptive business strategy
I've resisted writing much about this week's frenzy over DeepSeek, the relatively small Chinese company that has put the cat amongst the frantically multiplying large language model pigeons. The main reason I've avoided the discussion is that I'm pretty sure that this is only one step further along the path of LLM evolution - next week or next month, something else will be the new shiny object and we'll all have forgotten DeepSeek's big splash. Instead, I'd like to discuss how DeepSeek's labelling as open source AI helped them make such a big impact - and whether this labelling is accurate.
Inspiration and perspiration
It's frequently said that in software, we stand on the shoulders of giants. Everything depends on what came before: C# builds on C++ builds on C builds on B (you've been around for a while if you remember that last one!). Every new database, platform, content management system or search engine builds on the lessons learned from and is inspired by previous generations. 
The fuzzy part of this is what we mean by 'inspired' and practically how this translates into actual written code. Back in the early days, programmers would happily give away their code to others who needed it, to adapt and improve, in the expectation and hope that they in return would get access to the new version. Once people realised they could charge for software, this model changed, with some software becoming available only in compiled form under a written legal license. Those that still cared about sharing invented free software and slightly later, open source, using licenses to permit pretty much any use of the software as long as you also maintained the freedom to re-use, modify and distribute the derivatives.
Of course, not everybody played by the rules. There are plenty of well-documented instances of proprietary, closed source software being built on the backs of open source contributors and equally, open source code that looks suspiciously similar to a commercial alternative. In some cases people have genuinely re-created functionality from scratch, in many others just switching some variable names and functions around was enough. It's hard to replicate functionality - you can't create code in a vacuum and you can't un-see something you might base your program on, even unconsciously. If someone objects and brings in the lawyers we get into the nitty gritty detail and if people believe that program X looks way too similar to program Y, people get sued and settlements are made. If commercial strategies change, sometimes code that was previously open becomes more closed, which annoys previous contributors and may lead to a fork, creating a new open alternative. 
However, open source has become an established, and often disruptive force for good that underpins our technological world. Since you're giving away your IP (under certain conditions) it's challenging to make anywhere near as much money in open source than in commercial software, but it's also a great and quick way to build a large user base and active community. Repeatedly we see software sectors that were previously dominated by commercial vendors disrupted by open source alternatives - for example, the rise of Lucene-powered search engines in the early 2000s, or MySQL, now one of the most popular databases in the world.
Not really open AI
The evolution of LLMs has considerably muddied this picture, as they depend on training data, model code, weights and associated software. One principle of open source is that for something to be truly open, one must have unrestricted access to everything needed to build the system, but in the world of LLMs people seem perfectly happy to release a model and call it open source without also releasing the training data, weights etc. Even worse, some very well known companies have invented their own licenses (complete with restrictions) and decided to call these open source when they're clearly not.  What we see here is open source used for marketing and disruption, without truly signing up to the principles behind it. Sadly, journalists and commentators often fall for it.
No matter what the license, it's clear those building LLMs are still standing on each others' shoulders. Engineers and data scientists are trying to advance the state of the art, racing towards AGI, and some are using whatever method they can to achieve that - be it ingesting millions of copyrighted works or using the output of one model to train another. Playing fast and loose with naming, licenses and laws isn't an accident but a deliberate strategy - when you're creating the golden goose of AI and potentially attracting billions of dollars of investment, none of that seems to matter.
Unfortunately the open source world is still struggling to define what we mean by open AI, wish makes it even easier for AI companies to open-wash their offerings. The OSI have created a definition which isn't universally popular. We'll no doubt hear a lot more on this at the State of Open conference in London next week, hosted by OpenUK for which I'm proud to be an ambassador. 
DeepSeek's models have been widely reported as open source - but actually not everything you need to replicate them, such as the training code, is provided (although to their credit they've discussed how did this in an associated paper). Interestingly there is already an effort underway to provide a truly open model based on DeepSeek.
An open bridge
Time will tell whether whatever flavour of 'open' we decide on will be the best bridge over the moat being frantically dug by those large, well-funded (and mainly American) companies trying to land-grab & monetize AI. If we look back, it's clear that commercial software can eventually lose out to open source as this commoditizes the market. Enterprises in particular are nervous about sending their data over the wire to an API and may decide it's safer to run models in-house, which open source would permit. However, those with deep vendor relationships may choose to trust the offerings from Microsoft or Google rather than spending the time and money to build in-house AI expertise and capability. Most of the AI companies are now targeting enterprise adoption, although it won't be as easy as they may think. Open models, software and training data also help address the significant and justified concerns many people have about bias.
However if there's one thing that the DeepSeek release can show us is that saying your cutting-edge LLM is open source (no matter how inaccurately) can have a significant effect on the AI market. No matter what huge AI funding governments announce or grand commercial strategy you have, there's always the chance someone inspired by your work will try to replicate and improve it - and if they then release this to the world with fewer restrictions, your moat is no longer an effective defence. This applies to DeepSeek as much as anyone else.
Perhaps the larger players should bear this in mind, and we can only hope that some of them realise that releasing truly open source AI may be a winning strategy.
Interested in how open source AI can work for your business? Contact me.
Drawbridge Stock photos by Vecteezy

### Don't look for one ring to rule them all in enterprise search
Sequoia Capital recently released a report suggesting we "imagine a world where every profession has its own specialized AI search engine". This is a fallacy I call 'one ring to rule them all', the hope that if we could build the ultimate search & AI product it would automatically ingest any data and solve the enterprise use case for a particular sector. I'm confident in saying this is an impossible dream. So what am I Tolkein about? (sorry, couldn't resist!)
Why can't it just...
Anyone working in search will be familiar with the complaint "why can't it just work like Google". Users assume that the consumer search engine they're familiar with (most often, Google web search) is a perfect fit for the enterprise. They're unhappy with the performance and accuracy of search at work and just want it to be as easy to use and helpful as what they use at home.
By Barabas - Own work, CC BY-SA 3.0, Link
Google did of course try to take advantage of this with their Google Search Appliance, a rack-mounted 'search engine in a box' with the familiar branding. Not many will remember the GSA now, but for a few years it was a popular solution, likely to receive management approval and budget signoff due to the familiar Google brand. Just point it at your data and off you go! Except, of course, your data wasn't always that easy to get to, and would often need quite a lot of work to extract, condition and massage before it could be put into the magic box - great news for system integrators, many of whom made a good living from installing and supporting the GSA.
What people often mean however, is "why can't we just have one easy-to-use search engine at work". So why can't this be done, especially with all the power of search & AI we now have at our disposal?
In any business there are multiple use cases, user tasks and information needs. A lawyer, for example, needs to consult past cases for precendents, find all the letters and emails they've sent, organising all this by case and matter, plus figure out how to book holiday time, search company records, keep up with legal industry news etc. Some of this information lives within the company intranet, some is in public repositories, some is buried in a WordPerfect document from 1993, a database entry with bad metadata or a dodgily scanned PDF. For each type of data different ranking models may be needed - in some cases recency is most important, in others relevance, or perhaps a blend of the two. The information may need to be presented in different ways, with or without summaries or previews, or the search function itself may be buried inside another application, such a document management system or legal information system.
Enterprise search projects often promise, but never quite manage to reach all these silos. "Search over all your corporate data from one place" is a common theme. However, very simple things can often have the biggest impact on user experience - for example, why even search at all, when a redirect to a single page can give you the answer to "how do I book holiday time". If your user wants better search in their case management system, fix it there, don't force them to use another new system. Perhaps we decide that we shouldn't even allow people to find some of our less reliable data.
So why bother with AI for enterprise search?
I'm certainly not implying that AI techniques won't (and in many cases already have) revolutionise the world of enterprise search. With the ColPali model (now available in the leading Vespa search platform) we have a way of better searching PDFs (a print format that should never have become a content format in my humble opinion) on a visual basis rather than having to painfully extract text, tables and images. Swirl have cleverly turned the federated search model on its head, relying on the retrieval capabilities of the underlying databases, content management systems etc. but using a LLM to combine and summarise this heterogenous data into a coherent whole. Vector embeddings help boost retrieval by matching meaning rather than just keywords and Retrieval Augmented Generation helps reduce the chance of LLMs hallucinating an answer to your users' questions.
However none of these techniques completely solve the very real problems of enterprise search - many of which are people problems, not technology problems.
Learning the world
The challenge of building a search system for a particular business sector is that first, you have to learn how people in that sector think - and it's different for each sector. Building an enterprise search system for lawyers won't necessarily help you build one for medics, or telecom engineers. You'll need to employ ex-lawyers to do this effectively; create deep relationships with your legal clients; understand common standards, taxonomies, the history of the profession. 
A good strategy for developing an effective 'Google for my sector' is to focus on solving actual user problems, not trying to 'boil the ocean'. Do we really need to search legal cases from 20 years ago, or would it be better to help our new lawyers find an experienced colleague to ask? Sometimes the best strategy is to figure what not to index, rather than throwing every single piece of data into the mix. Data quality is the killer here - an item with bad metadata, tagged in the wrong way, can easily damage search result quality no matter if you are using traditional or AI techniques. Creating effective data curation processes is vital, but seldom a high priority.
Another challenge is that there is a huge amount of inertia and politics within large businesses (e.g. those potentially willing and able to pay for enterprise search projects and with sufficient data volume to make it worth it). Calculating the financial benefit of better search (so you can get sufficient budget for your project) has always been difficult, usually ending up with some vague promise of saving users' time and thus improving efficiency. Different departments may compete and actively resist the introduction of new features that they perceive may put them at a disadvantage. Some people may not even want to be found by a people search!
Your search team - assuming you have one at all - will probably be under-resourced and reactive, rather than proactive. They may not have time to keep up with the firehose of AI-related news, research and use these new techniques, spending most of their days fixing data problems, responding to user complaints prioritised by how important that user is in the organisation. It's very hard to say no when a senior manager insists that being able to find that opinion piece they wrote for the company intranet is the single most important search problem to solve today.
Are these people problems insuloble? Of course not, but they require patience, understanding and diplomacy, not just a blanket introduction of new technology.
The giants are coming
As Jo Kristian Bergum wrote recently, the AI giants are now focused on enterprise search, but may be unaware of the "soul-crushing challenges that will drain your team's energy long before you can sprinkle AI magic on top."  We're likely to see some grand announcements from OpenAI, Google and others during 2025 (I don't think we'll see a new yellow box though). Let's hope that they will take note of the decades of work that's already been done to understand enterprise search - a good starting point would be the many books written by Martin White (start with Enterprise Search and A History of Enterprise Search) or my own small contribution with Professor Udo Kruschwitz shown on the right (click for the free PDF).
Enterprise search has always, and will remain a hard problem. We can combine the exciting possibilities of AI with a deep understanding of the people and organisations we're trying to help, focusing on pragmatic solutions. Let's begin our quest!
Want to build enterprise search that truly serves your users? Interested in taking advantage of both AI & traditional techniques? Let's talk.
Broken Ring Stock photos by Vecteezy

### Speak your users' language with search & AI
 Let’s pan out from technologies, models, the firehose of news for a few minutes and consider the wider problem of connecting users to information using search & AI.
AI systems are often presented as a panacea, able to solve all business problems if only we throw enough computing power at them. But businesses serve people - external and internal users - and it's the people we need to help.
Solving user problems
Search and information retrieval in general are ways to solve user problems. I want to buy a washing machine for my family, with a limited budget and an eco-conscious mindset. I’m a lawyer looking for a past case, but I can’t quite remember the details. I work in a factory and I need to buy some specific parts for a machine - I miss the old paper catalogue from our supplier and don’t really want to use a computer, I’d rather talk to an expert. I need a holiday, but I’m not sure where to go, I need inspiration!
Over the years we’ve constructed complex information systems to gather, categorise and structure information in ways that (we hope) users can access effectively. We hope, because there’s a huge issue - our users don’t know what information we hold, or how we’ve organised and described it. To them, it’s often a black box. In the past they’d engage the services of an information expert deeply familiar with this box and what’s in it - a librarian or shop assistant for example - and through the medium of a conversation, they’d work together to find the information they need to carry out their task. As information volume grew, this became an unsustainable approach and we now rely on information retrieval systems to find what we need.
A question of translation
Many of these retrieval systems force the user to describe their information needs in very short sequences of words. These words probably don’t fully describe their needs and the words themselves may not match the contents of the black box. This is the central problem of search, a problem of mis-matched language: my user is not an expert in my field, writes their question in a different way to what we expect, or is using text to attempt to find something that isn’t textual, like an image.
For more complex or sector-specific information, some retrieval systems introduced complex query syntaxes that the user would have to learn to describe their information need, or a plethora of interface elements (filters, checkboxes etc.) they would have to choose from. Again, this complex query may not match the content, but at least with some training and practice the user could learn how to formulate an effective question. This type of search is common in areas such as law, patent examination, medical and recruitment, often described as ‘professional search’.
As search evolved, we developed many other ways to help our users: enhancing the findability of our content with metadata, structure and entity identification; replacing filters we might have to choose (without knowing exactly how they would restrict results) with facets allowing the user to cut down a long list of results; adding auto-suggest and auto-complete features to the search box to shortcut the query process; suggesting ways to better structure the query; using curated synonyms as a translation aid; adding spelling suggestions based on the words we knew from our content; developing complex ranking algorithms to better order the results. 
All of these can help of course, but the central problem still remains: our users and our content are speaking different languages.
Search can be regarded as a translation problem (watch this great talk from Doug Turnbull & Tommaso Teofili at Berlin Buzzwords) - turning the language of the user into the language of our content for better matching. We may need to reformulate the user’s query into something more useful, without necessarily knowing the context of this query - if we have only been given two or three words to work with, we lack context and the user’s intent is unclear. 
Modelling language for search
The rise of AI and specifically the rise of language models over the last few years promises us new tools for solving this issue of mis-matched language. Vector embeddings, derived from language models, give us a way to encode meaning, not just index specific words or terms. Language models can also help us reformulate and expand user queries into something more likely to match our content. Fine-tuned models can learn about our own content with all its specific terms, acronyms and shortforms.  
Embeddings perform better than traditional keyword search in many cases where there is a language mismatch, but keyword search is still useful when the query is exactly specified, for example when searching for a part number, or for items which must exactly match a particular quality. A hybrid of keyword and vector-based retrieval is often best, though how to mix the results from these two very different worlds remains a hard problem. Fine-tuning is expensive and time consuming, so we often have to rely on open source or commercial language models that don't know about our business or area of interest. 
Multi-modal models give us an opportunity to help those users searching for an image that isn't labelled with the words we type. Models trained on images of documents help reduce errors due to difficult and potentially faulty extraction of text from certain file formats such as PDFs. 
Changing the way we search
We can also consider how AI can help us to move away from traditional models of information retrieval - a typed query and ranked list of results. Conversational search (via a chatbot) lets us replicate that interaction with a librarian or subject matter expert, drawing out more context from the user during a chat session as we zero in on what they actually need. LLMs can also be used to summarise a set of results into a single answer - but due to the well-known hallucination problem, they cannot always be relied on: if they don’t contain the information we need, they may make up a plausible but incorrect answer. This is hugely risky for businesses that depend on reliable information, for example in the legal or medical sectors.
Retrieval Augmented Generation (RAG) is a technological solution to the need for a single answer to our user’s question, grounding a LLM in truth, backed by some kind of search engine. This engine - the R part of RAG - doesn’t necessarily need to be AI-driven, it can be a traditional search. The LLM is prompted to summarize an answer based on what the search engine retrieves - and to admit when it doesn't have enough information to do so. Not everyone is convinced RAG is the best way forward and there are some other approaches using automation to combine search and LLMs.
Interestingly, a lot of the activity in this area comes from outside the world of information retrieval - companies want to "do AI", build RAG or a chatbot, but they don't always consider the vital search component, or that traditional search techniques may be a simpler way to solve the user problem.
And so we come full circle - search needs AI to succeed, but often AI also needs great search. We're not going to replace every search engine with a chatbot, or use only vector embeddings for every search. Our toolbox has gained some more tools, we've not thrown it out and bought a robot that will make everything for us.
Choosing the best tool for the job with search & AI 
I'll finish with a very basic example of how to improve user experience (thanks Eric Pugh for suggesting this). In the Firefox web browser I use, if you type into the address bar something that looks like a URL (e.g. bbc.co.uk) and hit Enter, it will take you directly to that site. If you type something more like a multi-word query (e.g. bbc website) it will trigger a web search. This deceptively simple behaviour shortens the user journey.
Let's expand on this concept a little using our enhanced toolbox, remembering that we're here to solve a user problem. Our information systems should take the path of least resistance whenever possible, and adapt to the user's language. If I type in 'big washing machine' then let's do a hybrid search, using embeddings to cope with the multiple meanings and synonyms of 'big'; if I type in 'hotpoint nswr 846' then let's be more specific, searching using keywords and a brand filter for that model as I'm probably not interested in Indesit or Samsung machines. If I type in just 'washing machine' let's trigger a chatbot to ask for more detail, rather than giving the user a confusing list of thousands of items. If I type 'what is the best washing machine with large capacity under £300 for a family of 4 that doesn't want to use much electricity' then we can use RAG to provide a single answer. Vision models can support our user if they take a picture with their phone of their neighbour's washing machine they hear good things about!
Forcing our users to learn our language causes frustration - remember they don't know what we have or how we've described it. We have an arsenal of tools at our disposal that can help solve their problems, speaking their language with search & AI - so let's be creative!
Are you struggling to combine AI and traditional search techniques? Need to figure our your search & AI strategy? Let me help.
Language Stock photos by Vecteezy


## Pages
### Blog


### About Me
I'm an expert search consultant. My journey in search began in 1999 when I joined Muscat Ltd., co-founded by Martin Porter (author of the famous stemming algorithm) and helped lead a team that built a half-billion-page web search engine in 18 months. Leaving in 2001, I formed with some colleagues a consulting company, Flax, which I ran for the next 17 years before merging with our US partner OpenSource Connections. During this time we worked for clients including the UK government, leading recruitment companies, law firms and e-commerce companies, focusing on solutions based on Apache Solr and Elasticsearch, and I ran all non-technical aspects of the business including sales, recruiting, marketing, accounts and project management. I helped run and spoke at many leading industry events across the world. Flax then merged with our US partner, OpenSource Connections (OSC).As part of the OSC leadership team, I helped navigate the business through challenging times between 2019-2024. I ran search projects for clients across the world in multiple business sectors, some of them with revenues in the billions, acting as a senior Managing Consultant within small expert teams of relevance engineers. As Sales & Marketing Director I created, marketed and sold consulting and training engagements, often carrying the project from initial enquiry all the way through to successful delivery. Case studies on some of these projects are available. I created efficient and fully documented processes for sales and marketing, designed & developed collateral, maintained the company website, wrote many blogs and appeared on several industry podcasts and panels. I helped plan and manage a shift to mostly online engagements and the growth of the company beyond its original base of Charlottesville, USA to a mostly remote workforce distributed across the USA and Europe. In recent years I have also helped clients take advantage of recent innovations in AI-powered search, as AI has revolutionised our sector.
After attending OSC’s first Haystack conference in 2018, I took the idea to London to host the first Haystack Europe event and subsequently ran two of these conferences every year until 2024, handling promotion, ticket sales, scheduling, recording & editing videos and hosting the events in person – one attendee wrote ‘a spectacularly well run conference - Haystack is my favorite’. The most recent Haystack events were sold out and attracted nearly 250 attendees from across the world. I have also organised many smaller Meetup events including the London Lucene/Solr Meetup, Cambridge Enterprise Search Meetup and the online Haystack LIVE! Meetup which has over 1400 members. During my time at OSC the pandemic caused a sudden shift to online attendance and I worked with other event organisers to create highly-regarded online events, eventually transitioning these to hybrid events in the following years.
I am regularly asked to speak and host panels at industry events such as MICES, Berlin Buzzwords, Algolia’s DevCon and Search Solutions run by the British Computer Society, where in 2023 I was awarded Best Presentation. 
Embodying OSC’s mission statement to ‘Empower Search Teams’ I focused on building our brand and making OSC a leading and influential voice in the area of search relevance – developing educational materials such as a 200+ item video archive, glossaries, collections of useful tools, lists of worldwide Search Meetups, and administering the free Relevance Slack, which has now grown to a 5500+ person strong community covering every aspect of search. I forged links with many other companies in the sector, helping develop a shared understanding of search relevance principles such as the importance of measurement, leading to a recent role promoting User Behaviour Insights, a joint project between OSC and AWS OpenSearch which will change how all search engineers understand and utilise user behaviour data for improving result quality – my rich personal network has allowed me to introduce this concept to many search engine vendors and several are now working on adopting it. 
While at OSC I was also asked to join OpenUK, a relatively new organisation promoting open standards, open data and open source, particularly for government and regulators, and became an Ambassador for the organisation and a judge of the annual OpenUK Awards. I spoke at OpenUK’s State of Open Con in London in 2024 on ‘How to Build Open Source AI’. 
I co-authored the book ‘Searching the Enterprise’ for NOW Publications (get the free PDF by clicking on the image on the right) and as part of The Search Network, an international group of search experts, contributed to several annual ‘Search Insights’ reports. I am often featured on industry podcasts and interviewed by those writing about search & AI.Although I work with many different search engines, I was proud to have been asked to become an OpenSearch ambassador.
I love talking about, teaching and implementing great search & AI! But why call myself The Search Juggler? Well, I've also been a circus skills performer and teacher for over 30 years, and I've occasionally used these skills in my working life. There's a lot to keep in the air when you're building great search & AI - business requirements, people, data - and many requirements to juggle!
Want to know more? Contact me today!

### Home
Search isn't working? Let me help.
If you're struggling to provide relevant search results for your users, would like to take advantage of new AI-powered search techniques, have a search service or product that you would like to bring to market or promote more effectively, want to run a great event focused on search & AI or just need expert search consulting, talk to me.
About ME
HOW I CAN HELP
EVENTS & Talks
TESTIMONIALS
BLOG
CONTACT ME
SEARCH CONFERENCE CAlENDAR
JOIN THE LONDON SEARCH & AI MEETUP
About me
With my unique approach, I can transform your search strategy.I am a leading figure in the search industry, known for an honest, neutral and pragmatic viewpoint; I have held multiple roles including senior consultant, strategic advisor, project manager, sales & marketing director, conference organiser & speaker, trainer, writer & mentor. I am deeply connected with the business & technology of website and enterprise search engines. My past experience in software engineering gives me a highly informed perspective on search technology with a particular focus on open source platforms such as Lucene, Apache Solr, Elasticsearch and OpenSearch.  More recently I have helped several companies use traditional & modern AI techniques to supercharge search - here's some recommendations from the team at Moonpig who I helped in 2025.Read my story and contact me today to find out how I can help.
How I can help
With 25 years of experience in search, working with organisations across the world from startups to members of the Fortune 500, I am uniquely positioned to advise you. If I can't help on my own, I can use my deep and wide personal network to find the people you need. If you can't see what you need below, get in touch to discuss your requirements - I'm always happy to chat about search & AI!
Search Strategy Audit
Struggling to provide great results to your users?Working with your team either remotely or in person, I will analyze your strategic approach to search & AI and provide specific and targeted recommendations to drive your business forward. I'll identify blindspots, suggest better processes and patterns, show you how to build a great search team and discuss how you can measure search relevance.
Search Consulting
Need help from an experienced search expert?I am also available for hourly or daily consulting on all aspects of search & AI. I can train you on search fundamentals, help you develop processes to measure and continually improve search relevance, advise on technology choice, assist you as you create, build and upskill your search team and work with all levels of your business from engineering to executive. 
Market Positioning
Want to promote your search & AI product or service?Using my deep and long experience of the search sector, I will help you understand the world of search & AI, review your products, position your offerings for best traction, communicate your value and raise your profile. I'll show you which events to attend and how to actively participate in the search community.
Event Expert
Planning a conference or event about search & AI?As a well-known figure in the search community, an experienced and popular conference speaker and the organiser and host of many events including Haystack, I can help you create and run events to inform, educate and inspire. I'm also available as a keynote or track speaker - watch some of my past talks and check out my search conference calendar.
Watch my conference talks & podcasts
I'm an experienced conference speaker and host and I'm regularly interviewed on industry podcasts. Watch some of my past talks below and if you'd like me to speak at your event, join a panel or feature on a podcast, get in touch. In 2025 I hope to attend events including Elasticon London, OpenSearchCon, Berlin Buzzwords and Haystack - let me know if you'd like to meet up! 
VIew the Search conference calendar
05
Testimonials
"Charlie knows search and relevance...a unique perspective on search in all kinds of business domains and contexts.""...one of the most skilled professionals I’ve encountered in the field of information retrieval.""a knack for understanding the real requirements (both technical and organizational) and finding solutions for them.""..able to explain the parts I had been missing about techniques like RAG, and why to concentrate at least as much on the search part as the LLM, if not more so.""I highly recommend Charlie to help with your search capability even if you think it’s in a good place he will know how to improve it."From LinkedIn
Blog
Read my blogs from 2019-2024 here , from pre-2019 here - get the latest and greatest below!
More...
Contact
Need Help with Search & AI?
Book a MEETING with Calendly
+44 7767 825828
charlie@thesearchjuggler.comCambridge, UKThe Search Juggler Limited ○ UK Company number 16200797
OpenSearch Consultant • Elasticsearch Consultant • Solr Consultant • Enterprise Search Consultant • E-commerce Search Consultant • AI Powered Search Consultant • RAG Consultant • Site Search Expert • Relevance Engineer • Vespa Consultant
Solr Consulting, Elasticsearch Consulting, OpenSearch Consulting, Vespa Consulting, Vector Search Consulting, RAG Consulting, AI-powered search, e-commerce search, ecommerce search, Lucene Consulting, enterprise search consulting, website search consulting, relevance engineering, search relevance consulting, retrieval augmented generation consulting


## Events
### CHIIR
ACM SIGIR Conference on Human Information Interaction and RetrievalCHIIR is a multi-disciplinary research meeting. In addition to studies of interactive systems, information interaction, and retrieval, we encourage submissions on related topics, including human-human information interaction, novel interaction paradigms, new evaluation methods, and related research in a range of communities such as sociology, education, ethnography, psychology, human-computer interaction, and other relevant disciplines.

### OpenSearchCon China
OpenSearchCon is the annual conference designed to bring the OpenSearch community together – right to you. Whether you are a user, administrator, or developer, this event offers a focused opportunity to learn from real-world use cases, connect with local peers, and collaborate with experts. Join us to explore the latest in search, observability, and security applications. CFP open until January 11th 2026.

### OpenSearchCon Japan
OpenSearchCon Japan is a one-day, in-person event designed to bring the OpenSearch community together—right to you. Whether you’re a user, administrator, or developer, the roadshow offers a focused opportunity to learn from real-world use cases, connect with local peers, and collaborate with experts. Join us to explore the latest in search, observability, and security applications, all in a single, impactful day.
OpenSearchCon Japan will be co-located with Open Source Summit Japan (December 8-10.)

### Haystack EU 2025
Haystack is the conference for improving search relevance. If you're like us, you work to understand the shiny new tools or dense academic papers out there that promise the moon. Then you puzzle how to apply those insights to your search problem, in your search stack. But the path isn't always easy, and the promised gains don't always materialize.
Haystack is the conference for organizations where search, matching, and relevance really matters to the bottom line. For search managers, developers, relevance engineers & data scientists finding ways to innovate, see past the silver bullets, and share what actually has worked well for their unique problems. Please come share and learn!

### SISAP
*CFP Open until May 11th* 18th International Conference onSimilarity Search and Applications, SISAP 2025
The International Conference on Similarity Search and Applications (SISAP) is an annual forum for researchers and application developers in the area of similarity data management. It aims at the technological problems shared by numerous application domains, such as data mining, information retrieval, multimedia retrieval, computer vision, pattern recognition, computational biology, geography, biometrics, machine learning, and many others that need similarity searching as a necessary supporting service.

### CLEF
Conference and Labs of the Evaluation ForumInformation Access Evaluation meets Multilinguality, Multimodality, and Visualization*CFP Open until 6th May* CLEF 2025 is the 16th CLEF conference continuing the popular CLEF campaigns which have run since 2000 contributing to the systematic evaluation of information access systems, primarily through experimentation on shared tasks.
Building on the format first introduced in 2010, CLEF 2025 consists of an independent peer-reviewed conference on a broad range of issues in the fields of multilingual and multimodal information access evaluation, and a set of labs and workshops designed to test different aspects of mono and cross-language Information retrieval systems. Together, the conference and the lab series will maintain and expand upon the CLEF tradition of community-based evaluation and discussion on evaluation issues.

### NLPA
2026 8th International Conference on Natural Language Processing (ICNLP 2026) will be held in Xi'an, China from March 20 to 22, 2026. ICNLP 2026 is mainly Co-Sponsored by Xi'an University of Posts and Telecommunications and IEEE, hosted by School of Communications and Information Engineering, Xi'an University of Posts and Telecommunications, and jointly supported by many universities, industries, and academic institutions.Since its inception, the ICNLP conference series has been dedicated to advancing academic research and technological applications in NLP, providing a vital platform for researchers, students, developers, and practitioners worldwide to exchange innovative ideas and showcase cutting-edge achievements.ICNLP 2026 will cover a wide range of topics, including but not limited to natural language understanding, speech recognition, text generation, knowledge graphs, sentiment analysis, and cross-lingual information processing. The conference will feature keynote speeches by renowned experts from around the world, multiple academic discussions, and technical sharing sessions, providing participants with opportunities for in-depth exploration and collaboration. We warmly invite experts and scholars from the NLP community worldwide to join us in Xi'an to share their latest research achievements and collaboratively discuss the future directions of natural language processing.

### Community over Code North America
Community Over Code will feature four in-person days of sessions at the Hyatt Regency Minneapolis for ASF members, committers, and open source developers from around the world, focusing on Search, Big Data, Internet of Things, Community, Geospatial, Financial Tech, and many other topics. Each evening also features Birds of a Feather (BoF) sessions, where communities will have an opportunity for free-form discussion and planning around our various projects.

### Community over Code Asia
* CFP Closes 21st April * Community Over Code (formly known as ApacheCon) is the official global conference series of The Apache Software Foundation (ASF). Since 1998 – before the ASF’s incorporation – ApacheCon has been drawing participants at all levels to explore ”Tomorrow’s Technology Today” across 350+ Apache projects and their diverse communities. Community Over Code showcases the latest developments in Apache projects and emerging innovations through hands-on sessions, keynotes, real-world case studies, trainings, hackathons, and more.

### NTCIR
Since 1997, the NTCIR project has promoted research efforts for enhancing Information Access (IA) technologies such as Information Retrieval (IR), Text Summarization, Information Extraction (IE), and Question Answering (QA) techniques. Its general purposes are to:
Offer research infrastructure that allows researchers to conduct a large-scale evaluation of IA technologies
Form a research community in which findings based on comparable experimental results are shared and exchanged, and
Develop evaluation methodologies and performance measures of IA technologies.
Collaborative works in the NTCIR allow us to create large-scale test collections that are indispensable for checking the effectiveness of novel IA techniques. In addition, in the process of the collaboration, it is expected that deep insight into research problems is successfully shared among researchers.
In particular, the NTCIR-19 focuses on three topics on IA technology mainly;
Deep NLP tasks in specialized domains such as e-commerce, agricultural information, human values, finance, medicine, regulatory compliance, etc. (CAMEO, DAGRI, FEHU, FinArg-3, HIDDEN-RAD2, MedNLP-CALL, and RegCom)
Modern IR tasks such as instruction design for agentic search (AgenticInstruction), lifelog content retrieval (Lifelog-7), pre-trained model retrieval (ModelRetrieval), and known-item retrieval under uncertainty (Tip-of-the-Tongue).
Response evaluation tasks such as automatic evaluation of LLMs' responses (AEOLLM-2), confidence-aware RAG (R2C2), and claim verification in scientific literature (SciClaimEval).

### The ACM Web Conference
Since its inception in 1989, the World Wide Web has transformed the way we live, work, and connect. The Web Conference (formerly known as the International World Wide Web Conference, abbreviated WWW) has been the premier annual forum for presenting and discussing advances in research, development, standards, and applications shaping the future of the Web.
The 2026 ACM Web Conference will feature a high-quality program including research sessions, posters and demonstrations, a PhD symposium for junior scholars, workshops, tutorials, an industry track for practitioners, as well as thought-provoking keynote speakers, panels, special tracks, and a co-located conference (Web4All).

### Enterprise Search & Discovery
Join us in Washington, DC this November 17 - 20 for KMWorld 2025 where you’ll get practical advice, hear inspiring thought leadership, and have access to in-depth training and workshops on how KM and related disciplines can provide enormous value for your organization.
This year's conference theme — Enterprise Intelligence: Collaboration & Knowledge Sharing for Success — focuses on the innovative breakthroughs and learning experiences from KM practitioners as they guide their organizations into the future. Showcasing new and creative knowledge-sharing tools and techniques as well as human strategies that have an impact on all types of organizations and communities, KMWorld 2025 highlights knowledge sharing in action for exceptional innovation and bottom-line success.

### TREC
CFP Closes late May * An evaluation workshop series for measuring the effectiveness of search algorithms and other technologies that help us find information.

### Haystack US
Haystack is the conference for improving search relevance. If you're like us, you work to understand the shiny new tools or dense academic papers out there that promise the moon. Then you puzzle how to apply those insights to your search problem, in your search stack. But the path isn't always easy, and the promised gains don't always materialize.
Haystack is the conference for organizations where search, matching, and relevance really matters to the bottom line. For search managers, developers, relevance engineers & data scientists finding ways to innovate, see past the silver bullets, and share what actually has worked well for their unique problems. Please come share and learn!

### Search Solutions
Search Solutions is the BCS Information Retrieval Specialist Group’s (BCS IRSG) annual event focused on practitioner issues in the arena of search and information retrieval (IR). We bring together practitioners, researchers, analysts and end users to discuss the latest developments in the IR community and to share insights between research and practice.

### CIKM
*CFP closes May 16th*    The Conference on Information and Knowledge Management (CIKM) provides an international forum for presentation and discussion of research on information and knowledge management, as well as recent advances on data and knowledge bases. The purpose of the conference is to identify challenging problems facing the development of future knowledge and information systems, and to shape future directions of research by soliciting and reviewing high quality, applied and theoretical research findings. An important part of the conference is the Workshops program which focuses on timely research challenges and initiatives. CIKM has a strong tradition of workshops devoted to emerging areas of database management, IR, and related fields.

### SIGIR
"We are delighted to invite you to the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, to be held from 20–24 July 2026 in Melbourne | Naarm, Australia (and with ICTIR the day after, on July 25).
Now, we could repeat all the things you already know: that Melbourne | Naarm is an extraordinary city, renowned for its laneways, coffee culture, and liveability. But let’s also be honest: in July it will be winter, it may well rain and you’ll need to bring a jacket.
Even so, we know that you’ll be willing to travel halfway around the world to attend SIGIR 2026, to an event that will deliver a program that is as sharp and relevant as ever. Expect world-leading research in information retrieval, dynamic workshops, tutorials that will shift the way you think and opportunities to connect with members of our global community of researchers and practitioners.
The only thing more stimulating than the debates you’ll hear in the sessions will be the conversations you’ll have afterwards with peers and friends, both old and new. We anticipate a week of discovery, collaboration and inspiration—set in a city that has a way of winning over even the most sceptical visitor."

### ECIR
The 48th European Conference on Information Retrieval (ECIR 2026) is Europe's premier forum for cutting-edge research in Information Retrieval (IR). Bringing together researchers, practitioners, and industry experts, ECIR provides a platform for sharing new, unpublished, and innovative research shaping the future of IR.

### OpenSearchCon North America
*CFP Closes May 11*     OpenSearchCon is the annual conference that brings the OpenSearch community together to learn, connect, and collaborate. Users, administrators, and developers attend to explore solutions to real-world problems, network with peers, and explore the future of search, observability, and security applications.

### OpenSearchCon India
OpenSearchCon is the annual conference that brings the OpenSearch community together to learn, connect, and collaborate. Users, administrators, and developers attend to explore solutions to real-world problems, network with peers, and explore the future of search, observability, and security applications.

### MICES
Date TBC - but it's usually the day after Berlin Buzzwords!
MICES brings together participants from a variety of backgrounds, all sharing a common interest in e-commerce search. In order to stimulate and facilitate discussion, we will start with scheduled talks in the morning. This will be followed by self-organising sessions: participants are encouraged to initiate discussions about their topics of interest or to give an ad hoc presentation.

### Berlin Buzzwords
Berlin Buzzwords is Europe’s leading conference for modern data infrastructure, search and machine learning and is focused on open source software projects.
CFP open until 15th February 2026.

### OpenSearchCon Europe
OpenSearchCon is the annual conference that brings the OpenSearch community together to learn, connect, and collaborate. Users, administrators, and developers attend to explore solutions to real-world problems, network with peers, and explore the future of search, observability, and security applications.
CFP closes 18th January 2026