Skip to content
Return to: MkDocs-MaterialX Start Page

Header

Search: Better, Faster, Smaller

Search: Better, Faster, Smaller

This is the story of how we managed to completely rebuild client-side search, delivering a significantly better user experience while making it faster and smaller at the same time.

The search of Material for MkDocs is by far one of its best and most-loved assets: multilingual, offline-capable, and most importantly: all client-side. It provides a solution to empower the users of your documentation to find what they're searching for instantly without the headache of managing additional servers. However, even though several iterations have been made, there's still some room for improvement, which is why we rebuilt the search plugin and integration from the ground up. This article shines some light on the internals of the new search, why it's much more powerful than the previous version, and what's about to come.

The next section discusses the architecture and issues of the current search implementation. If you immediately want to learn what's new, skip to the section just after that.

Architecture⚓︎

Architecture

Material for MkDocs uses lunr together with lunr-languages to implement its client-side search capabilities. When a documentation page is loaded and JavaScript is available, the search index as generated by the built-in search plugin during the build process is requested from the server:

const index$ = document.forms.namedItem("search")
  ? __search?.index || requestJSON<SearchIndex>(
    new URL("search/search_index.json", config.base)
  )
  : NEVER

Search Index⚓︎

Search Index

The search index includes a stripped-down version of all pages. Let's take a look at an example to understand precisely what the search index contains from the original Markdown file:

Expand to Inspect Example 
# Example

## Text

It's very easy to make some words **bold** and other words *italic*
with Markdown. You can even add [links](#), or even `code`:

```
if (isAwesome) {
  return true
}
```

## Lists

Sometimes you want numbered lists:

1. One
2. Two
3. Three

Sometimes you want bullet points:

* Start a line with a star
* Profit!

{
  "config": {
    "indexing": "full",
    "lang": [
      "en"
    ],
    "min_search_length": 3,
    "prebuild_index": false,
    "separator": "[\\s\\-]+"
  },
  "docs": [
    {
      "location": "page/",
      "title": "Example",
      "text": "Example Text It's very easy to make some words bold and other words italic with Markdown. You can even add links , or even code : if (isAwesome) { return true } Lists Sometimes you want numbered lists: One Two Three Sometimes you want bullet points: Start a line with a star Profit!"
    },
    {
      "location": "page/#example",
      "title": "Example",
      "text": ""
    },
    {
      "location": "page/#text",
      "title": "Text",
      "text": "It's very easy to make some words bold and other words italic with Markdown. You can even add links , or even code : if (isAwesome) { return true }"
    },
    {
      "location": "page/#lists",
      "title": "Lists",
      "text": "Sometimes you want numbered lists: One Two Three Sometimes you want bullet points: Start a line with a star Profit!"
    }
  ]
}

Seach Index Inspection

If we inspect the search index, we immediately see several problems:

  1. All content is included twice: the search index contains one entry with the entire contents of the page, and one entry for each section of the page, i.e., each block preceded by a headline or subheadline. This significantly contributes to the size of the search index.

  2. All structure is lost: when the search index is built, all structural information like HTML tags and attributes are stripped from the content. While this approach works well for paragraphs and inline formatting, it might be problematic for lists and code blocks. An excerpt:

… links , or even code : if (isAwesome) { … } Lists Sometimes you want …

  • Context: for an untrained eye, the result can look like gibberish, as it's not immediately apparent what classifies as text and what as code. Furthermore, it's not clear that Lists is a headline as it's merged with the code block before and the paragraph after it.

  • Punctuation: inline elements like links that are immediately followed by punctuation are separated by whitespace (see , and : in the excerpt). This is because all extracted text is joined with a whitespace character during the construction of the search index.

It's not difficult to see that it can be quite challenging to implement a good search experience for theme authors, which is why Material for MkDocs (up to now) did some monkey patching to be able to render slightly more meaningful search previews.

Search Worker⚓︎

Search Worker

The actual search functionality is implemented as part of a web worker1, which creates and manages the lunr search index. When search is initialized, the following steps are taken:

  1. Linking sections with pages: The search index is parsed, and each section is linked to its parent page. The parent page itself is not indexed, as it would lead to duplicate results, so only the sections remain. Linking is necessary, as search results are grouped by page.

  2. Tokenization: The title and text values of each section are split into tokens by using the separator as configured in mkdocs.yml. Tokenization itself is carried out by lunr's default tokenizer, which doesn't allow for lookahead or separators spanning multiple characters.

    Important

    Why is this important and a big deal? We will see later how much more we can achieve with a tokenizer that is capable of separating strings with lookahead.

  3. Indexing: As a final step, each section is indexed. When querying the index, if a search query includes one of the tokens as returned by step 2., the section is considered to be part of the search result and passed to the main thread.

Now, that's basically how the search worker operates. Sure, there's a little more magic involved, e.g., search results are post-processed and rescored to account for some shortcomings of lunr, but in general, this is how data gets into and out of the index.

Search Previews⚓︎

Deep Dive: The Evolution of Search Previews

Users should be able to quickly scan and evaluate the relevance of a search result in the given context. A concise summary with highlighted occurrences of search terms is essential for a great search experience.

The Current Shortcoming

Search preview generation currently falls short; previews often appear without search terms because they are truncated after 320 characters.




The Problem: The first two results look irrelevant as they don't visually include the query string, even though they are matches.

The Roadmap to Relevance⚓︎

To solve this once and for all, we are evaluating three critical factors for the next generation of the search engine:

  1. Word boundaries: Some themes2 generate previews by expanding text around an occurrence until a whitespace is hit. While this works for Western languages, it fails for languages like Japanese or Chinese3 which require dedicated segmenters.

    Example of a whitespace-broken preview:
    … channels, e.g., or which can be configured via mkdocs.yml …

  2. Context-awareness: Whitespace isn't a "universal" fix. In code blocks, removing or collapsing whitespace can fundamentally change the meaning of the snippet, making the preview misleading.

  3. Structure: Preserving structural information (like whether a word appears in a heading vs. a code block) allows for a much higher "information density" in the results, helping users evaluate relevance at a glance.

What's New?⚓︎

What's New?

After we built a solid understanding of the problem space and before we dive into the internals of our new search implementation to see which of the problems it already solves, a quick overview of what features and improvements it brings:

  • Better: support for rich search previews, preserving the structural formation of code blocks, inline code, and lists, so they are rendered as-is, as well as lookahead tokenization, more accurate highlighting, and improved stability of typeahead. Also, a slightly better UX.

  • Faster and smaller: significant decrease in search index size of up to 48% due to improved extraction and construction techniques, resulting in a search experience that is up to 95% faster, which is particularly helpful for large documentation projects.

Rich Search Previews⚓︎

Feature: Rich Search Previews

We’ve completely rebuilt the search plugin from scratch to preserve structural information like code blocks, inline code, and lists.

Visual Comparison⚓︎

Outcome: Search results now respect formatting, making them significantly more readable.

Outcome: Results were stripped of all context and formatting.

The Data Shift⚓︎

Code blocks are now first-class citizens of search previews. By moving from a flat-text index to a structured HTML index, we can render complex elements directly in the search results.

Expand to inspect search index 
{
  ...
  "docs":
}

{
  ...
  "docs":
}

Optimization: Efficiency & Structure

By refining the search index, we’ve achieved significant gains in both performance and utility. Here is how the situation has improved:

  1. Content Deduplication
    The search index no longer stores page content twice. By indexing only sections and linking them to parents, we’ve achieved a significant reduction in size, leading to faster transfers and a smaller memory footprint.

  2. Structural Preservation
    Sections now include a subset of HTML. This solves the "punctuation issue" where extra whitespace used to be inserted, and it allows for much more effective scanning.

Markup Comparison⚓︎

Notice how the new structure handles code snippets compared to the old flat-text method:

… links, or even <code>code</code>:</p> <pre><code>if (isAwesome){ … }\n</code></pre>

Result: Clean formatting with no artificial whitespace.

… links , or even code : if (isAwesome) { … }

Result: Messy punctuation and lost context.

Next Step: Let's look at how this data is broken down during tokenization.

Tokenizer Lookahead⚓︎

Tokenizer Lookahead

The default tokenizer of lunr uses a regular expression to split a given string by matching each character against the separator as defined in mkdocs.yml. This doesn't allow for more complex separators based on lookahead or multiple characters.

Fortunately, our new search implementation provides an advanced tokenizer that doesn't have these shortcomings and supports more complex regular expressions. As a result, Material for MkDocs just changed its own separator configuration to the following value:

[\s\-,:!=\[\]()"/]+|(?!\b)(?=[A-Z][a-z])|\.(?!\d)|&[lg]t;

While the first part up to the first | contains a list of single control characters at which the string should be split, the following three sections explain the remainder of the regular expression.4

Case Changes⚓︎

Case Changes

Many programming languages use PascalCase or camelCase naming conventions. When a user searches for the term case, it's expected that PascalCase and camelCase will appear. By adding the following match group to the separator, this is achieved with ease:

(?!\b)(?=[A-Z][a-z])

This regex combines a negative lookahead ((?!\b), not a word boundary) with a positive lookahead ((?=[A-Z][a-z]), an uppercase followed by a lowercase character).

Behavior:
- PascalCase Pascal, Case
- camelCase camel, Case
- UPPERCASE UPPERCASE (no split)

Searching for searchHighlight now correctly brings up sections like search.highlight, proving this works even within search queries.5

Version Numbers⚓︎

Version Numbers⚓︎

Indexing version numbers is another problem that can be solved with a small lookahead. Usually, . should be considered a separator to split words like search.highlight. However, splitting version numbers at . will make them undiscoverable. Thus, the following expression:

\.(?!\d)

This regex matches a . only if it is not immediately followed by a digit (\d). This ensures version numbers stay intact and searchable. For example, searching for 9.7.6 will now correctly bring up the release notes.

HTML/XML tags⚓︎

HTML/XML Tags

If your documentation includes HTML/XML code examples, you may want to allow users to find specific tag names. Unfortunately, the < and > control characters are encoded in code blocks as &lt; and &gt;.

By adding the following expression to the separator, you can ensure these tags remain searchable:

&[lg]t;

We've only just begun to scratch the surface of the new possibilities tokenizer lookahead brings. If you found other useful expressions, you're invited to share them in the comment section.

Accurate Highlighting⚓︎

Accurate Highlighting

Highlighting is the last step in the search process and involves highlighting all search term occurrences in a given search result. For a long time, highlighting was implemented through dynamically generated regular expressions.6

This approach has some problems with non-whitespace languages like Japanese or Chinese3 since it only works if the highlighted term is at a word boundary. However, Asian languages are tokenized using a dedicated segmenter, which cannot be modeled with regular expressions.

Now, as a direct result of the new tokenization approach, our new search implementation uses token positions for highlighting, making it exactly as powerful as tokenization:

  1. Word boundaries: As the new highlighter uses token positions, word boundaries are equal to token boundaries. This means that more complex cases of tokenization (e.g., case changes, version numbers, HTML/XML tags) are now all highlighted accurately.

  2. Context-awareness: As the new search index preserves some of the structural information of the original document, the content of a section is now divided into separate content blocks—paragraphs, code blocks, and lists.

    Only the content blocks that actually contain occurrences of one of the search terms are considered for inclusion into the search preview. If a term only occurs in a code block, it's the code block that gets rendered, see, for example, the results of twitter.

Benchmarks⚓︎

Benchmarks

We conducted two benchmarks—one with the documentation of Material for MkDocs itself, and one with a massive corpus of Markdown files totaling over 800,000 words—a scale most projects will likely never reach:

Project Before Now Relative
Material for MkDocs
Index size 573 kB 335 kB –42%
Index size (gzip) 105 kB 78 kB –27%
Indexing time7 265 ms 177 ms –34%
KJV Markdown8
Index size 8.2 MB 4.4 MB –47%
Index size (gzip) 2.3 MB 1.2 MB –48%
Indexing time 2,700 ms 1,390 ms –48%

Benchmark results

  • The results show that indexing time—the time taken to set up search upon page load—has dropped by up to 48%, meaning the new search is up to 95% faster. This is a significant improvement for large documentation projects.

  • While 1.3s may still sound long, using the new client-side search with instant loading only creates the search index on the initial load. When navigating, the index is preserved across pages, so this cost is only paid once.

User Interface⚓︎

User Interface

Additionally, some small improvements have been made, most prominently the more results on this page button, which now sticks to the top of the search result list when open. This enables the user to jump out of the list more quickly.

What's Next?⚓︎

What's Next?

Our new search implementation is a big improvement to Material for MkDocs. It solves some long-standing issues which needed to be tackled for years. Yet, it's only the start of a search experience that is going to get better and better.

Next up:

  • Context-aware search summarization: Currently, the first two matching content blocks are rendered as a search preview. With the new tokenization technique, we laid the groundwork for more sophisticated shortening and summarization methods, which we're tackling next.

  • User interface improvements: Since we now have full control over the search plugin, we can add meaningful metadata to provide more context and a better experience. We'll explore some of those paths in the future.

If you've made it this far, thank you for your time and interest in Material for MkDocs! This is the first blog article that I decided to write after a short Twitter survey prompted me to. You're invited to leave a comment to share your experiences with the new search implementation.

  1. Prior to , search was carried out in the main thread which
    locked up the browser, rendering it unusable. This problem
    was first reported in (#)904 and, after some back and forth,
    fixed and released in . ↩

  2. At the time of writing, Just the Docs and Docusaurus use this method. ↩

  3. China and Japan are both within the top 5 countries of origin for Material for MkDocs users. ↩↩

  4. As a fun fact: the separator default value of the search plugin being [\s\-]+ always has been kind of irritating, as it suggests that multiple characters can be considered being a separator. However, the +is completely irrelevant, as regular expression groups involving multiple characters were never supported by lunr's default tokenizer. ↩

  5. Previously, search queries weren't correctly tokenized because lunr disables the pipeline for terms containing wildcards. To ensure a smooth typeahead experience, Material for MkDocs adds wildcards to the end of search terms (unless prefixed with + or -), which effectively disabled standard tokenization until now. ↩

  6. Using the separator as defined in mkdocs.yml, a regular expression was constructed that was trying to mimic the tokenizer. As an example, the search query search highlight was transformed into the rather cumbersome regular expression (^|<separator>)(search|highlight), which only matches at word boundaries. ↩

  7. Smallest value of ten distinct runs. ↩

  8. We agnostically use KJV Markdown as a tool for testing to learn how Material for MkDocs behaves on large corpora, as it's a very large set of Markdown files with over 800k words. ↩