MAIS AMOR POR FAVOR

Fast JS Search on Gatsby: Roll Your Own Site Search

Fast JS Search on Gatsby

Fast JS Search on Gatsby: Roll Your Own Site Search

SHARE:

πŸ”₯ Fast JS Search on Gatsby

In this post we look at how to add fast JS search on Gatsby. We are talking about site search here. That is allowing users to search for content on your site from within your site. Of course you could use one of the popular search services to provide the search functionality. However if you are the kind of developer who likes to get their hands dirty and roll their own functionality, this post is for you. We will even see how you can keep your site static while providing site search using JS Search.

JS Search was developed by Brian Vaughan from Facebook's core React team . Based on Lunr, it is so lightweight that we can build a search index just in time. Despite being lightweight, JS Search is highly configurable. We take advantage of the configurability to add stemming so that we return similar results. As an example, if the user searches for β€œlike”, we return results which match likes, liked, likely and liking. If all of this sounds like your cup of tea, let's get going.

πŸ“ Plan of Action

We will make search available to the user via the main home page on the example.com/ route. Search will be triggered when an s search parameter is included in the URL. So if the use goes to https://example.com/, the regular home page is loaded. However, if instead they go to https://example.com/?s=cheese, we will render the search component and show search results for β€œcheese”. Of course, the user will not need to know this to trigger a search β€” when the use clicks the search button, we will direct them to the search page (by navigating to https://example.com/?s= programatically). Let's look at how we will implement this.

  1. We will use the Climate Gatsby Starter, so we can hit the ground running and save ourselves pasting in boiler plate code. If you are adding search to an existing site, it should still be easy to follow along.

  2. Our first step will be to make the search documents available to the home page component, so it can build an index when needed. JS Search is so fast that there is no real advantage to building the search index ahead of time and serialising it. We will make the documents available using Gatsby's onCreatePage API to add the documents to the home page's context.

  3. Next we will create a Search component with placeholder content and add a search button to our nav bar.

  4. With all the preliminaries out of the way we can update our home page to render the Search component instead of its regular content whenever needed.

  5. Finally we can get into the JS Search part.

Bonus

If you have followed one of these tutorials before, you know we like to add a little polish! How about some first class user experience (UX)? Wouldn't it be great if when the user opens up a search result the search term is highlighted in the text? On top, we will scroll the first occurrence of the search term into view.

Does that sound like a plan? Let's get to it then!

πŸ—³οΈ Poll

How do you currently implement site search on new projects?
Voting reveals latest results.

🧱 Fast JS Search on Gatsby

Getting going

First of all let's clone the Gatsby Starter Climate repo. This is an MDX blog starter with some dummy content. Essentially we just need to add site search. To clone the repo and fire it up type these commands in the terminal:

gatsby new gatsby-site-search https://github.com/rodneylab/gatsby-starter-climate.git
cd gatsby-site-search
cp .env.EXAMPLE .env.development
cp .env.EXAMPLE .env.production
gatsby develop

We will need a few packages to add search functionality, let's add them all now:

npm install js-search remark stemmer strip-markdown

Your new MDX blog with demo content should now be running at localhost:8000/ . The site is already fully functional and you would start customising it now with your own content if you weren't already doing this tutorial. Take a few moments to familiarise yourself with the project files and once you've seen what's what, let's carry on.

Gatsby Node

To tinker with the onCreatePage API, we need to create a gatsby-node.js file in the project's root folder:

gatsby-node.js
javascript
1const remark = require('remark');
2const strip = require('strip-markdown');
3const { createFilePath } = require('gatsby-source-filesystem');
4
5exports.onCreateNode = async ({ node, getNode, actions }) => {
6 const { createNodeField } = actions;
7 if (node.internal.type === 'Mdx') {
8 const slug = createFilePath({ node, getNode, basePath: 'content/blog' });
9 createNodeField({
10 name: 'slug',
11 node,
12 value: slug,
13 });
14 }
15};

What are we doing here? Before we get on to the onCreatePage API we need to make the post slug available, using the onCreateNode API . We will use it in onCreatePage so that later, when a user clicks a search result, they can be directed to the right post. Speaking of onCreatePage, lets add this code to the end of gatsby-node.js:

gatsby-node.js
javascript
17exports.onCreatePage = async ({ page, actions, getNodes }) => {
18 const { createPage, deletePage } = actions;
19
20 if (page.path.match(/^/$/)) {
21 const postsRemark = await getNodes().filter((value) => value.internal.type === 'Mdx');
22 const allPosts = [];
23 postsRemark.forEach((element) => {
24 let body;
25 const { rawBody } = element;
26 const bodyStartIndex = rawBody.indexOf('---
27', 4);
28 const markdownBody = rawBody.slice(bodyStartIndex);
29 remark()
30 .use(strip)
31 .process(markdownBody, (err, file) => {
32 if (err) throw err;
33 body = String(file);
34 });
35 const { categories, featuredImageAlt, postTitle, seoMetaDescription, tags } =
36 element.frontmatter;
37 const { slug } = element.fields;
38 allPosts.push({
39 id: element.id,
40 body,
41 categories,
42 featuredImageAlt,
43 postTitle,
44 seoMetaDescription,
45 slug,
46 tags,
47 });
48 });
49
50 deletePage(page);
51 createPage({
52 ...page,
53 context: {
54 ...page.context,
55 postData: {
56 allPosts,
57 },
58 },
59 });
60 }
61};

onCreatePage as the name suggests, runs for every page, during the build process. We just need to add our search documents to the home page. We use a regular expression in line 20 only to run the code block for the home page. The following line (line 21) gets all of the site's MDX blog posts.

One by one we will pull off data from the blog posts which we want users to be able to search for. In lines 35–36, you can see we are pullling data from the post frontmatter. The preceding code extracts the post body from its original MDX format (which includes the frontmatter as well as the post body itself). We place this data for each post into an array, along with an id which will we useful later. Finally in lines 49–58, we replace the page object, adding a context. The context, like Gatsby's GraphQL API makes data available to site pages. We will see shortly how we access this data in the page's React component.

That's all the backend stuff complete. Next, let's add some UI.

Adding a Search Component and Button

Let's create a placeholder Search component. We will fill in content and logic later. Create a Search.jsx file in the src/components directory:

src/components/Search.jsx
jsx
import React from 'react';
const Search = () => <>Search</>;
export { Search as default };

Then we need to add a search button to the layout header, so let's edit src/components/Layout.jsx. First we'll update imports:

src/components/Layout.jsx
jsx
1import dayjs from 'dayjs';
2import { graphql, Link, navigate } from 'gatsby';
3import { StaticImage } from 'gatsby-plugin-image';
4import PropTypes from 'prop-types';
5import React from 'react';
6import { COPYRIGHT_ENTITY } from '../constants/entities';
7import {
8 CameraIcon,
9 FacebookIcon,
10 GithubIcon,
11 LinkedinIcon,
12 SearchIcon,
13 TwitterIcon,
14} from './Icons';

Next we want to show a search button on every page, except when the Search component is rendered. We will use a new prop, hideSearch to help:

src/components/Layout.jsx
jsx
119export const PureLayout = ({ children, data: { site }, hideSearch }) => {

Then let's add the search button itself to the existing nav in the header:

src/components/Layout.jsx
jsx
131<nav className={nav}>
132 <ul>
133 <li>
134 <Link aria-label="Jump to home page" to="/">
135 Home
136 </Link>
137 </li>
138 <li>
139 <Link aria-label="Jump to contact page" to="/contact/">
140 Contact
141 </Link>
142 </li>
143 {!hideSearch ? (
144 <li>
145 <button
146 type="button"
147 className={hoverJump}
148 onClick={(event) => {
149 event.preventDefault();
150 navigate('/?s=');
151 }}
152 >
153 <SearchIcon />
154 </button>
155 </li>
156 ) : null}
157 </ul>
158 </nav>

Here we use Gatsby's navigate function to navigate programatically to the search page.

To finish off in this file, let's update the prop types:

src/components/Layout.jsx
jsx
176PureLayout.defaultProps = {
177 hideSearch: false,
178};
179
180PureLayout.propTypes = {
181 children: PropTypes.oneOfType([PropTypes.arrayOf(PropTypes.node), PropTypes.node]).isRequired,
182 data: PropTypes.shape({
183 site: PropTypes.shape({
184 buildTime: PropTypes.string,
185 siteMetadata: PropTypes.shape({
186 facebookPage: PropTypes.string,
187 githubPage: PropTypes.string,
188 linkedinProfile: PropTypes.string,
189 twitterUsername: PropTypes.string,
190 }),
191 }),
192 }).isRequired,
193 hideSearch: PropTypes.bool,
194};

This won't work yet because the SearchIcon we import doesn't yet exist. Let's create a search icon using Feather icons in the src/components/Icons.jsx file:

src/components/Layout.jsx
jsx
176export const SearchIcon = () => (
177 <span role="img" aria-label="search">
178 <FeatherIcon icon="search" />
179 </span>
180);

OK, we just need to make the Search button look nice, before we move on:

src/components/Layout.module.scss
scss
21.nav {
22 display: flex;
23 margin-left: auto;
24 list-style-type: none;
25
26 ul {
27 display: flex;
28 align-items: flex-end;
29 padding-bottom: 0;
30 margin-bottom: $spacing-0;
31
32 li {
33 display: flex;
34 font-size: $font-size-4;
35 margin-left: $spacing-6;
36 margin-bottom: $spacing-1;
37 }
38 }
39
40 button {
41 background-color: $color-accent;
42 border-style: none;
43 color: $color-theme-4;
44 cursor: pointer;
45 }
46}

If you click the search button, the URL changes, but nothing interesting happens. Let's change that in the following section.

Update Home Page with Search

Earlier we added a little data to our home page's context. Let's take a look at that data now. Edit src/pages/index.jsx:

src/pages/index.jsx
scss
21import { graphql } from 'gatsby';
22import PropTypes from 'prop-types';
23import React from 'react';
24import BlogRoll from '../components/BlogRoll';
25import Card from '../components/Card';
26import { PureLayout as Layout } from '../components/Layout';
27import Search from '../components/Search';
28import { PureSEO as SEO } from '../components/SEO';
29import { isBrowser } from '../utilities/utilities';
30
31export default function Home({ data, pageContext }) {
32 const { allPosts } = pageContext.postData;
33 if (isBrowser) {
34 const searchParam = new URLSearchParams(window.location.search.substring(1)).get('s');
35 if (searchParam !== null) {
36 return (
37 <>
38 <Layout data={data} hideSearch>
39 <Search data={data} posts={allPosts} />
40 <pre>{JSON.stringify(allPosts, null, 2)}</pre>
41 </Layout>
42 </>
43 );
44 }
45 }
46 return (
47 <>
48 <SEO
49 data={data}
50 title="Home"
51 metadescription="Climate - Gatsby v3 MDX Blog Starter - starter code by Rodney Lab to help you get going on your next blog site"
52 />
53 ...

Now in your browser go to localhost:8000/?s=anything . You should see the regular home page content replaced. Instead you will see the Search component with it's placeholder content. Below that, we are rendering the context data we made available in the onCreatePage API. Isn't that awesome? If you're happy with how the context API works now, delete line 40.

A couple of notes. In line 34 we are use the URLSearchParams interface to access the search parameters from the browser address bar. The check in line 33 just makes sure this code does not execute on the server while the page is building, as it could break the build process (there is no browser address available during server build).

Before we move on let's update the prop types:

src/pages/index.jsx
scss
54Home.propTypes = {
55 data: PropTypes.shape({
56 site: PropTypes.shape({
57 buildTime: PropTypes.string,
58 }),
59 }).isRequired,
60 pageContext: PropTypes.shape({
61 postData: PropTypes.shape({
62 allPosts: PropTypes.arrayOf(
63 PropTypes.shape({
64 categories: PropTypes.arrayOf(PropTypes.string),
65 featuredImageAlt: PropTypes.string,
66 postTitle: PropTypes.string,
67 seoMetaDescription: PropTypes.string,
68 tags: PropTypes.arrayOf(PropTypes.string),
69 }),
70 ),
71 }),
72 }).isRequired,
73};

The final step in this post is to wire up the search component. Why don't we limber up for the home straight?

JS Search

Let's define a search options file before we crack open the Search component:

src/config/search.js
scss
1const searchOptions = {
2 indexStrategy: 'Prefix match',
3 searchSanitiser: 'Lower Case',
4 indexBy: ['body', 'categories', 'featuredImageAlt', 'postTitle', 'seoMetaDescription', 'tags'],
5 termFrequency: true,
6 removeStopWords: true,
7 stemWords: true,
8};
9
10export { searchOptions as default };

These are options we will pass to JS Search, but what do they do?

  • indexStrategy: Prefix match, the default, returns a match for β€œcheese” with β€œc”, β€œch”, β€œche” etc. Other alternatives allow substring matches from any part of the word, not just the start (so the algorithm returns a match for β€œcheese” with β€œeese”). Finally you can request an exact match, so only β€œcheese” returns a match for β€œcheese”.
  • searchSanitiser: whether we want to have case sensitive search or not. Lower case corresponds to case insentitive (β€œChEeSe” matches β€œcheese”)
  • indexBy: This tells JS Search which of the document data to index documents by.
  • termFrequency: true switches on inverse document frequency which basically means very common words do not artificially push results higher up the results list.
  • stopWords: The algorithm ignores stop words like and, that and the which are unlikely to be important for the search query when we set this option to true.
  • stemWords: We mentioned this earlier. Going back to β€œcheese”, the algorithm would consider, β€œcheesy” a match for β€œcheese”.

Search Component

With that out of the way, let's replace the Search component placeholder code:

src/config/search.js
scss
1import { navigate } from 'gatsby';
2import {
3 AllSubstringsIndexStrategy,
4 CaseSensitiveSanitizer,
5 ExactWordIndexStrategy,
6 LowerCaseSanitizer,
7 PrefixIndexStrategy,
8 Search as JSSearch,
9 StemmingTokenizer,
10 StopWordsTokenizer,
11 TfIdfSearchIndex,
12 UnorderedSearchIndex,
13} from 'js-search';
14import PropTypes from 'prop-types';
15import React, { useEffect, useRef, useState } from 'react';
16import { stemmer } from 'stemmer';
17import searchOptions from '../../config/search';
18import { H_ELLIPSIS_ENTITY } from '../constants/entities';
19import { isBrowser } from '../utilities/utilities';
20import BlogPostSummary from './BlogPostSummary';
21import { SearchIcon } from './Icons';
22import {
23 searchButton,
24 searchInput,
25 searchInputContainer,
26 searchLabel,
27 searchNoResultsText,
28 searchTextInput,
29} from './Search.module.scss';
30
31const Search = ({ data, posts }) => {
32 const {
33 termFrequency = true,
34 removeStopWords = true,
35 stemWords = true,
36 indexStrategy = 'Prefix match',
37 searchSanitiser = 'Lower case',
38 indexBy = ['body', 'postTitle'],
39 } = searchOptions;
40
41 let searchParam;
42 if (isBrowser) {
43 const params = new URLSearchParams(window.location.search.substring(1));
44 searchParam = params.get('s') || '';
45 }
46
47 const [isLoading, setLoading] = useState(true);
48 const [search, setSearch] = useState(null);
49 const [searchQuery, setSearchQuery] = useState(searchParam);
50 const [searchResults, setSearchResults] = useState([]);
51 const searchInputNode = useRef();
52
53 const addSearchIndices = (dataToSearch) => {
54 indexBy.forEach((element) => {
55 dataToSearch.addIndex(element);
56 });
57 };
58
59 const getPostNode = (postID) => {
60 const { edges } = data.allMdx;
61 return edges.find((value) => value.node.id === postID).node;
62 };
63
64 const getPostFrontmatter = (postData) => {
65 const { datePublished, featuredImageAlt, tags, seoMetaDescription, thumbnail, wideThumbnail } =
66 getPostNode(postData.id).frontmatter;
67 const { postTitle } = postData;
68 return {
69 datePublished,
70 featuredImageAlt,
71 thumbnail,
72 wideThumbnail,
73 postTitle,
74 seoMetaDescription,
75 tags,
76 };
77 };
78
79 const rebuildIndex = () => {
80 const dataToSearch = new JSSearch('id');
81
82 if (removeStopWords) {
83 dataToSearch.tokenizer = new StopWordsTokenizer(dataToSearch.tokenizer);
84 }
85 if (stemWords) {
86 dataToSearch.tokenizer = new StemmingTokenizer(stemmer, dataToSearch.tokenizer);
87 }
88 if (indexStrategy === 'All') {
89 dataToSearch.indexStrategy = new AllSubstringsIndexStrategy();
90 } else if (indexStrategy === 'Exact match') {
91 dataToSearch.indexStrategy = new ExactWordIndexStrategy();
92 } else if (indexStrategy === 'Prefix match') {
93 dataToSearch.indexStrategy = new PrefixIndexStrategy();
94 }
95
96 dataToSearch.sanitizer =
97 searchSanitiser === 'Case sensitive'
98 ? new CaseSensitiveSanitizer()
99 : new LowerCaseSanitizer();
100 dataToSearch.searchIndex =
101 termFrequency === true ? new TfIdfSearchIndex('id') : new UnorderedSearchIndex();
102
103 addSearchIndices(dataToSearch);
104 dataToSearch.addDocuments(posts);
105 setSearch(dataToSearch);
106 setLoading(false);
107 };
108
109 // build the search index when the component mounts
110 useEffect(() => {
111 rebuildIndex();
112 }, []);
113
114 // once the index is built, if we are already waiting for a search result, search and update UI
115 useEffect(() => {
116 if (searchInputNode.current) {
117 searchInputNode.current.focus();
118 }
119 if (search !== null && searchQuery !== '') {
120 const queryResult = search.search(searchQuery);
121 setSearchResults(queryResult);
122 }
123 }, [search]);
124
125 const handleChange = (event) => {
126 const queryResult = search.search(event.target.value);
127 setSearchQuery(event.target.value);
128 setSearchResults(queryResult);
129 navigate(`/?s=${event.target.value}`);
130 };
131
132 const handleSubmit = (event) => {
133 event.preventDefault();
134 };
135
136 const queryResults = searchQuery === '' ? posts : searchResults;
137
138 if (isLoading || search === null) {
139 return <p>Searching{H_ELLIPSIS_ENTITY}</p>;
140 }
141
142 return (
143 <>
144 <form onSubmit={handleSubmit}>
145 <label htmlFor="Search">
146 <div className={searchInputContainer}>
147 <div className={searchInput}>
148 <span className={searchLabel}>Search for</span>
149 <input
150 aria-label="Search blog posts"
151 ref={searchInputNode}
152 className={searchTextInput}
153 autoComplete="off"
154 spellCheck={false}
155 id="Search"
156 value={searchQuery}
157 onChange={handleChange}
158 placeholder="Search"
159 type="search"
160 />
161 <button aria-labelledby="Search" type="submit" className={searchButton}>
162 <SearchIcon />
163 </button>
164 </div>
165 </div>
166 </label>
167 </form>
168 {searchQuery === '' ? null : (
169 <>
170 {searchResults.length ? (
171 <>
172 <h1>{`Search results (${searchResults.length})`}:</h1>
173 <section role="feed">
174 {queryResults.map((value, index) => {
175 const { id } = value;
176 const frontmatter = getPostFrontmatter(value);
177 return (
178 <article aria-posinset={index} aria-setsize="-1" key={id}>
179 <BlogPostSummary frontmatter={frontmatter} slug={value.slug.slice(1)} />
180 </article>
181 );
182 })}
183 </section>
184 </>
185 ) : (
186 <p className={searchNoResultsText}>
187 Nothing like that here! Why don&apos;t you try another search term?
188 </p>
189 )}
190 </>
191 )}
192 </>
193 );
194};
195
196Search.propTypes = {
197 data: PropTypes.shape({
198 allMdx: PropTypes.shape({
199 edges: PropTypes.arrayOf(
200 PropTypes.shape({
201 node: PropTypes.shape({
202 id: PropTypes.string.isRequired,
203 frontmatter: PropTypes.shape({
204 datePublished: PropTypes.string.isRequired,
205 }),
206 }),
207 }),
208 ),
209 }),
210 }).isRequired,
211
212 posts: PropTypes.arrayOf(
213 PropTypes.shape({
214 catergories: PropTypes.arrayOf(PropTypes.string),
215 featuredImageAlt: PropTypes.string,
216 postTitle: PropTypes.string,
217 seoMetaDescription: PropTypes.string,
218 tags: PropTypes.arrayOf(PropTypes.string),
219 }),
220 ).isRequired,
221};
222
223export { Search as default };

Let's look at this a bit at a time.

In lines 32–39, we import the options we just defined. Then in lines 41–45 we get the search parameter, this is similar to what we did on the home page. We need to do this so we know what the user intends to search for. Later we will set the URL based on the search term the user enters into the search box. The search parameter is the single source of truth, when it come to the user's search term.

Lines 53–57 & 79–107 configure the search using the options we just set. We use React's useEffect API with an empty array as the second argument, so that the search indices are only built once, when the component mounts in lines 108–110. When JS Search builds the search index, it generates the search object and stores this in the component's state (line 104). Now whenever we have a search query, we just use the search object (which has the index embedded) to get our search results.

What happens when the user types a search query in the search box? Let's look at that step, by step. With each key press the handleChange method is called. We could add a throttle function here but JS Search is pretty fast so there's probably no need, unless we have a large number of documents. Anyway, the handleChange method (lines 125–130), gets the search results from our search object, sets the UI to display the search results and changes the browser URL to match the search query.

JS Search on Gatsby: Search Results

If there are some search results, we display a BlogPostSummary component for each result. This is the same component we use on the regular home page, just instead of feeding it all posts in date order, we display search matches, most relevant first.

That was a huge chunk of code, and we only looked at the most important parts, so do let me know, if some aspects need a touch more clarification.

πŸ’― Try itΒ out

Fast JS Search on Gatsby: Try it Out

Try searching for β€œcamera”. This is in every post so all three dummy posts appear in results. Now if we start over, this time searching for β€œfold”, you see the two posts where folding cameras are mentioned appear. Click a result and the browser opens it up for you. Next search for β€œElon Musk” β€” no results! Looks like it's all working.

πŸ™ŒπŸ½ It's not Right but it'sΒ OK

In this post we learned:

  • about JS Search configuration options,
  • how we can pass data to a component's context using Gatsby's onCreatePage API,
  • using the above to add search to your Gatsby site.

We are still missing the UX polish. No need to worry though, we will add the shine in this follow-up post on highlighting search results using Web APIs. You can see the full code for this JS Search on Gatsby on the Rodney Lab Git Hub repo . You can visit the live demo site at gatsby-site-search.rodneylab.com .

Keen to hear how you plan to use this! Is there something from this post you can leverage for a side project or even client project? I do hope so! Let me know if there is anything in the post that I can improve on, for anyone else creating this project. You can leave a comment below, @ me on twitter or try one of the other contact methods listed below.

πŸ™πŸ½ Fast JS Search on Gatsby: Feedback

If you have found this post useful, even though you can only afford even a tiny contribution, please consider supporting me through Buy me a Coffee.

Finally, feel free to share the post on your social media accounts for all your followers who will find it useful. As well as leaving a comment below, you can get in touch via @askRodney on Twitter and also askRodney on Telegram . Also, see further ways to get in touch with Rodney Lab. I post regularly on SvelteKit as well as Gatsby JS among other topics. Also subscribe to the newsletter to keep up-to-date with our latest projects.