How is relevance determined?
Cludo Search is not a black box – the search relevance is based on a unique blend of machine learning and human customization on top of an algorithm.
Before customization or machine learning is applied, the Cludo search engine bases relevance using the Okapi BM25 algorithm. A rule of thumb for good search relevance – and SEO in general – is that a good content structure is key. While this also applies to Cludo, the relevance in a Cludo search engine is highly dependent on how the crawler is configured, i.e. what fields and content are indexed for the pages.
How relevance is determined in Cludo
Relevance in Cludo is determined by a combination of the core search algorithm, automatic text processing, manual tools you configure, and—when enabled—AI Search. The following sections summarize each of these; linked articles provide more detail.
Core algorithm
The search engine ranks results based on the presence and concentration of the search terms on each page. By default, multi-word searches use OR logic: a result can match one or more of the words, and results that match more words (or match them more prominently) tend to rank higher. Engine behavior can also be set to AND so that all words must match. For more on operators, see What are the default search operators?
Automatic text processing
Several features apply automatically to improve matching and relevance:
- Stemming — Word forms are normalized (e.g. “running” and “run” are treated as related), so results can match different forms of the same word without extra configuration. The engine uses language-specific analyzers that include stemming.
- Stop words — Common fill words (e.g. “the”, “and”) are ignored in the query so they do not dominate the ranking. Stop word lists are language-specific. See What stop words does Cludo use?
- Fuzzy matching — The engine can tolerate small spelling mistakes (e.g. skipped or swapped letters) and still return relevant results. For example, “recieve” may match “receive” automatically. You can add custom corrections via Misspellings.
- Bigram matching — When enabled for your account, documents where the query terms appear as adjacent words (e.g. “fantastic football” matching “fantastic football”) receive a relevance boost, so phrase-like matches rank higher.
- Compound words — In languages that use them, compound words are treated as single units; the parts are not matched separately. See How are compound words treated in searches?
Manual relevance tools
You can fine-tune relevance using tools in MyCludo:
- Synonyms — Link terms that mean the same thing (e.g. “waste”, “garbage”, “trash”) so that a search for any of them returns results for all. Synonyms affect which results are shown; they do not change which terms trigger page rankings.
- Misspellings — Define correct spellings for commonly misspelled queries so the engine replaces the misspelling and returns the right results.
- Boostings — Raise or lower the ranking of results that match certain rules: URL path (By Path), fixed field value (Field value), field match to the search term (Field search), numeric field, date freshness (Decay), or file type. Boostings add to the core algorithm; they do not replace it.
- Page rankings — Pin specific results to the top (or bottom) for chosen search terms.
- Intelligent Re-ranking — Uses past search and click activity to adjust rankings over time so that results with more engagement rank higher.
Crawler and index
What gets crawled and indexed directly affects relevance: only content that is in the index can appear in results and be ranked. Crawler configuration (which URLs and content are included, which fields are extracted, and how often the crawler runs) therefore influences which pages are available for the algorithm and tools above to rank. Ensuring important content is indexed and that key fields (e.g. title & description) are configured helps the engine surface the right results.
Impact on AI features
AI features such as AI Summary and AI Chat use the same search index and ranking. So the same factors—algorithm, automatic processing, manual tools, and crawler setup—determine which results are considered “top” or most relevant when those AI features run. Enabling AI Search further improves relevance by combining keyword and semantic (vector) matching.
How crawler fields and boosting affect the search relevance
The crawler only indexes the content that it is configured to read. When searching for results, only the content indexed by the crawler is searchable.
The title and description of a page are always required for a crawler to pick it up. Additional fields such as meta description, subtitles, or intro text can also be set up. These fields will then be indexed separately, making sure these are not only searchable but also allowing for later adjustment to the relevance using boosting.
How to measure relevance
Relevance for search can be measured using the Mean Reciprocal Rank (MRR). Mean Reciprocal Rank is a statistical measure which takes a list of possible search page rankings and defines an order by the position of the relevance page ranking and click-through rates. For example, if someone searches a term, clicks on the first-page result, that would be a perfect MRR score of 1. The reciprocal rank is calculated using:
| Search Query | Page Rankings | Clicked on Ranking | Rank in Rankings | Reciprocal Rank |
|---|---|---|---|---|
| dog | 1. doggy 2. doghouse 3. dogs | dogs | 3 | 1/3 = 0.33 |
| monkey | 1. monkey 2. monkey bars 3. monkey pets | monkey bars | 2 | 1/2 = 0.5 |
| cat | 1. cat 2. catholic 3. category | cat | 1 | 1 = 1 |
A score of 0.5 is considered a good standard MRR score. This means the visitors are on clicking on the 2nd result or higher on average. It is possible to get the average MRR score of a search engine by reaching out to support.
How to impact the search algorithm
With Cludo, there are multiple ways to impact or customize the search algorithm, from determining which page results should show up for specific queries, to prioritize or de-prioritize certain areas of a website, or even using machine learning to dynamically adjust the order of results based on user activity.
The following Cludo tools can be used to customize the search algorithm:
- Page rankings
- Intelligent Re-ranking
- Boostings
These tools all act on top of the Okapi BM25 algorithm in the search application.
