Using ElasticSearch with Next.js

In this article, we share how to use ElasticSearch with Next.js to index your Firestore documents and make them searchable.

ElasticSearch is a popular search engine that sits next to an application's primary data store. This is particularly interesting to Makerkit's users as Firestore does not have excellent search capabilities; therefore, we need a search engine that sits next to it.

In this article, we will learn how to use ElasticSearch and Next.js together, so you can deploy them for your SaaS.

To demonstrate ElasticSearch's capabilities, we will be indexing documents written to our Firestore database and implementing a search input in our UI that allows us to pull the results from ElasticSearch.

This tutorial assumes we have a working Next.js application running. If you need any help with setting that up, you can check the following articles:

  1. Setting up the Firestore Emulators with Next.js
  2. Setting up Firebase Functions

Why use ElasticSearch?

ElasticSearch is a capable search engine that works very well for various use cases, such as:

  • full-text search
  • custom weighted search
  • data analytics

In a typical SaaS, ElasticSearch could be deployed for various reasons, for example:

  • storing and retrieving structured analytical data (clicks, views, sessions, or any sort of data) for creating interactive dashboards
  • storing unstructured data (such as free text), and then allowing users to search through it and return the most relevant documents

Given that databases such as Firebase Firestore do not support full-text search and complex analytical aggregations, ElasticSearch could be a great option to place beside the primary datastore.

Running ElasticSearch with Docker

The easiest way to install ElasticSearch is using the Docker Image.

If you have Docker installed and running, add the Docker configuration below to your project:

docker-compose.yml
version: '2.2'
services:
 es01:
   image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
   container_name: es01
   environment:
     - node.name=es01
     - cluster.name=es-docker-cluster
     - discovery.seed_hosts=es02
     - cluster.initial_master_nodes=es01,es02
     - bootstrap.memory_lock=true
     - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   ulimits:
     memlock:
       soft: -1
       hard: -1
   volumes:
     - data01:/usr/share/elasticsearch/data
   ports:
     - 9200:9200
   networks:
     - elastic
 es02:
   image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
   container_name: es02
   environment:
     - node.name=es02
     - cluster.name=es-docker-cluster
     - discovery.seed_hosts=es01,es03
     - cluster.initial_master_nodes=es01,es02
     - bootstrap.memory_lock=true
     - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
   ulimits:
     memlock:
       soft: -1
       hard: -1
   volumes:
     - data02:/usr/share/elasticsearch/data
   networks:
     - elastic
 kib01:
   image: docker.elastic.co/kibana/kibana:7.17.0
   container_name: kib01
   ports:
     - 5601:5601
   environment:
     ELASTICSEARCH_URL: http://es01:9200
     ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200"]'
   networks:
     - elastic
volumes:
 data01:
   driver: local
 data02:
   driver: local
 data03:
   driver: local
networks:
 elastic:
   driver: bridge

You can now run the ELK stack from your IDE or from your terminal. If the latter, use the command below:

docker compose up

To test that the stack is working, check if Kibana is running at http://localhost:5601.

Running ElasticSearch in a Next.js Application

Creating the ElasticSearch Client

ElasticSearch comes with an official Node.js client that makes it relatively easy to work with, typically versioned with the same version as its server counterpart (Elastic.co version).

To install it, run the following command:

npm i @elasticsearch/client --save

Now we can create a function to initialize the ElasticSearch client:

elastic.ts
async function createElasticSearchClient(options: ClientOptions) {
 const { Client } = await import('@elastic/elasticsearch');
 return new Client(options);
}

The above is pretty simple, but as you can see, we have to provide the options that include the host and the credentials to the ElasticSearch server. Additionally, we provide the local environment host we will use during development.

Let's assume the function getOptions can retrieve the configuration from the environment variables. These options have the following interface:

{
  node?: string;
  cloud?: string;
  password?: string;
  username?: string;
}

The cloud property is used by elastic.co for identifying the node to connect to, so we add it to the example if you choose to build a deployment using the service.

Otherwise, you can use the property node to point to an Elastic server. Instead, we point to our Docker instance if cloud is not provided.

const DEFAULTS: ClientOptions = {
  node: 'http://localhost:9200',
};
function getElasticOptions(): ClientOptions {
  // remember to define "getOptions" to get your credentials!
  const { cloud, password, username } = getOptions();
  // this branch is for prod (i.e. cloud is provided)
  if (cloud) {
    return {
      cloud: {
        id: cloud,
      },
      auth: {
        username,
        password,
      },
    };
  }
  // this branch is for dev
  return {
    node: DEFAULTS.node,
  };
}

We dynamically import the Elastic client so that it will be loaded only if used. Now, we can create a factory to get an elastic client:

export function getElasticClient() {
  return createElasticSearchClient(
    getElasticOptions()
  );
}

For the sake of this blog post, I export the function from a file named elastic.ts:

src/core/elastic.ts
import { ClientOptions } from '@elastic/elasticsearch';
async function createElasticSearchClient(options: ClientOptions) {
 const { Client } = await import('@elastic/elasticsearch');
 return new Client(options);
}
const DEFAULTS: ClientOptions = {
 node: 'http://localhost:9200',
};
export function getElasticClient() {
 return createElasticSearchClient(getElasticOptions());
}
function getElasticOptions(): ClientOptions {
 // this branch is for dev
 return {
   node: DEFAULTS.node,
 };
}

Indexing documents with the Node.js ElasticSearch client

The process of indexing documents means we're adding data to our ElasticSearch database, which will, in turn, analyze and ingest it. After getting indexed by ElasticSearch, the document will become searchable.

ElasticSearch can automatically understand your data's shape when the index is created by indexing a document. With that said, if the data type of the documents is a bit complex (for example, it has nested arrays), you will need to pre-define the schema before it is created.

Indexing a document in ElasticSearch is fairly simple. For example, let's index a simple JSON document using the index method:

async function indexBlogPost(post: BlogPost) {
  const elastic = await getElasticClient();
  return elastic.index({
    index: `blog-posts`,
    body: post,
  });
}

Using the index method above, we have:

  1. Created an index named blog-posts (if this did not exist yet)
  2. Added a document containing the properties of post

A practical example: automatically indexing Firestore documents in ElasticSearch

As Firebase's limited search functionality requires a different search engine, we need to work it around by indexing each document written to the database.

To automate the process, we can use Firestore Triggers: these triggers allow us to fire a function every time a document is written or deleted.

Listening to Firestore document updates

Let's assume we are writing to a Firestore path tasks/{taskId}. To receive updates for the path's documents, we can use the Cloud Function below:

functions/src/index.ts
import { firestore } from 'firebase-functions';
export const onTaskCreated = firestore
 .document('tasks/{taskId}')
 .onWrite((change, context) => {
   // do something here
 });

Writing Firestore documents to ElasticSearch

Now, let's write the logic to push our data to ElasticSearch:

functions/src/index.ts
import { firestore } from 'firebase-functions';
export const onTaskCreated = firestore
 .document('tasks/{taskId}')
 .onWrite(async (change) => {
   const task = change.after.data();
   if (!task) {
     return;
   }
   // decorate Task data with its ID
   task.id = change.after.id;
   const elastic = await getElasticClient();
   await elastic.index({
     body: task,
     index: `tasks`,
   });
   console.log(`Task successfully indexed`);
 });

After creating a document, we can check if it gets created by navigating to Kibana. For example, if you have created an index pattern, you can check it out in the Discover Tab in Kibana.

You should suffix your index names for splitting indexes; alternatively, consider the Lifecycle API that will do it for you based on index size or document age.

Then, we can target the indexes we search using an index pattern: let's assume we name our indexes tasks-0001 and tasks-0002, we can search our index using the pattern tasks-*.

Creating a Next.js API endpoint to query ElasticSearch

Now that we can automatically sync up Elastic and Firestore, we want to be able to query Elastic's documents using an API function.

To do so, we will write a Next.js endpoint that will fetch the documents from ElasticSearch and return them to the client.

Before doing that, though, let's see how we would write an ElasticSearch query to fetch tasks by their name field:

POST tasks*/_search
{
  "query": {
    "match": {
      "name": "<QUERY>"
    }
  }
}

Now, let's write the same query using the ElasticSearch Node.js client. We create the API function at api/tasks/search.ts with the following content:

api/tasks/search.ts
async function tasksSearchHandler(
 req: NextApiRequest, 
 res: NextApiResponse
) {
 const { q: query } = req.query;
 // validate the "q" query parameter exists and is a string
 if (!query || typeof query !== 'string') {
   return res.status(400).end();
 }
 const elastic = await getElasticClient();
 const result = await elastic.search({
   index: 'tasks*',
   query: {
     match: {
       name: query,
     },
   },
 });
 // we send back the list of documents found
 const tasks = result.hits.hits.map((item) => item._source);
 return res.json(tasks);
}

Fetching ElasticSearch documents from the Next.js client

Let's create a hook to fetch the data after the user runs a search.

NB: this is a highly simplified version for fetching data. Best to use libraries such as SWT, React-query, or Makerkit's own useApiRequest if you use it.

function useSearchTasks() {
  return useCallback((query: string) => {
    return fetch(`/api/tasks/search?q=${query}`)
      .then(r => r.json());
  }, []);
}

Displaying results in a Next.js application

Now, we can query the data from our React components.

  1. First, we want to create a search bar:
function SearchInput(props: React.PropsWithChildren<{
  onSearch: (query: string) => void;
}>) {
  return (
    <form onSubmit={e => {
      e.preventDefault();
      
      const query = new FormData(e.currentTarget).get('query');
        props.onSearch(query as string);
    }}>
      <input name='query' placeholder='Search task...' />
      <button>Search Tasks</button>
    </form>
  );
}
  1. Finally, we can add the SearchInput to the TasksContainer component, which is responsible for fetching and displaying the tasks if any was found.
function TasksContainer() {
  const [tasks, setTasks] = useState<Task[]>();
  const searchTasks = useSearchTasks();
  const onSearch = useCallback(
    async (query: string) => {
      const tasks = await searchTasks(query);
      setTasks(tasks);
    },
    [searchTasks]
  );
  return (
    <>
      <div>Type below to search for your tasks:</div>
      <SearchInput onSearch={onSearch} />
      {tasks && tasks.length ? <TasksList tasks={tasks} /> : <NoTaskFound />}
    </>
  );
}

🥳 And here is the final result!