OpenSearch and Elasticsearch share a common heritage, which might imply they deal with third party plugins in the same way – but the reality is very different. For those using or thinking of adopting these platforms it’s important to understand how each engine works with these community-built plugins – or doesn’t.
Learning to Love Learning to Rank
Let’s start with a little history. The Elasticsearch Learning to Rank (LTR) plugin, which uses a machine learning model to re-rank search results, was first released in 2018 by my former employers OpenSource Connections after a successful collaboration with the Wikimedia Foundation. It’s a mature, well understood piece of open source technology with regular releases and a healthy contributor base. When AWS forked Elasticsearch and created OpenSearch, someone also forked this plugin and again this has become a healthy project.
However from Elasticsearch v8.12, Elastic created their own LTR plugin. The reasons I’ve heard range from ‘we need it to work properly with our hosted service’ to ‘security, security, security’ but whatever the actual reason, this is a paid feature for Elasticsearch. Sadly I can’t tell you exactly what subscription levels give you access to this LTR plugin as the documentation says:
and the general subscriptions page doesn’t really tell you anything about LTR (I suspect it’s a case of ‘Call Sales For More Details’). In any case, if you’re using Elastic’s hosted Elasticsearch, this is your only choice. Of course, if you’re self-hosting Elasticsearch either on the cloud or on-premise you can do whatever you like.
Let’s compare this to how OpenSearch deals with third party plugins. Although I know they have exactly the same (entirely reasonable) concerns about security, they chose to embrace the open source LTR plugin and this is available on the AWS hosted version of OpenSearch. They have worked with various contributors to the plugin to get this all to work, and very kindly they’ve also listed its heritage:
Rewriting the Rules with Querqy
Let’s consider another example. Querqy is a plugin that lets you rewrite queries based on a series of query-specific rules – it’s a powerful way to give some control of search results to commercial teams who understand things like ‘when the user searches for ‘trousers’ we should boost results from this category and/or brand’ or ‘this word is actually a synonym for this other word in our particular product language’. It’s used by giant ecommerce companies including Walmart, Asda and Otto and works with Apache Solr, Elasticsearch and OpenSearch, and replicates some of the functionality commonly available in specialised, commercial ecommerce search platforms. There’s another great project, SMUI, that gives you a simple user interface for creating rules, usable by business teams:
Sadly it also hasn’t been adopted by Elasticsearch, who instead chose to write their own query rules feature from version 9.0 – it replicates some, but not all of the functionality of Querqy. I’ve also heard from some clients that Elastic have told them that Querqy isn’t a good plugin to use for reasons including ‘because it doesn’t work on the hosted service’ – even if the client in question isn’t using that and self-hosting Elasticsearch instead.
OpenSearch does at least mention Querqy, although it’s only supported for certain versions of OpenSearch and there’s currently no availability on AWS’ hosted OpenSearch Service. There have also been various collaborative talks given by Querqy & OpenSearch maintainers, and discussions about how to make Querqy an officially supported plugin.
Why the difference is all about business
It’s understandable that not all third-party plugins should be adopted and made available by either Elasticsearch or OpenSearch – there’s a cost to integration and subsequent maintenance. However when a plugin is well-established with an existing user base and there are ways available to work with those maintaining it, why not try? AWS OpenSearch has to my knowledge done this with several organisations involved with plugin development, working collaboratively and helping fund development. It’s also helped OpenSearch provide a wider range of options for users.
Elastic’s approach however is more ‘not invented here’, and also influenced by their business strategy – of course, they’d much rather you bought hosting from them than self-hosted, so offering the functionality of these plugins only to subscription users makes sense. However it does mean that you have only one choice – use what Elastic builds, which isn’t necessarily as useful or full-featured as what is out there in the wider market.
Plugins like LTR or Querqy are built to fill gaps between what search platforms provide and what real users need – and if they’ve survived as long as these two projects have done, I think that implies that the gaps are still there. Working together with the community to fill these gaps would benefit everyone.
If you want to know more about how to use these plugins for your Elasticsearch or OpenSearch project, get in touch.