# Using ElasticSearch with Next.js

> In this article, we share how to use ElasticSearch with Next.js to index your Firestore documents and make them searchable.

*Published: 2022-10-05*
*Canonical: https://makerkit.dev/blog/tutorials/nextjs-elasticsearch*

---

ElasticSearch is a popular search engine that sits next to an application's primary data store. This is particularly interesting to Makerkit's users as Firestore does not have excellent search capabilities; therefore, we need a search engine that sits next to it.

In this article, we will learn how to use ElasticSearch and Next.js together, so you can deploy them for your SaaS. 

To demonstrate ElasticSearch's capabilities, we will be indexing documents written to our Firestore database and implementing a search input in our UI that allows us to pull the results from ElasticSearch.

This tutorial assumes we have a working Next.js application running. If you need any help with setting that up, you can check the following articles:

1. [Setting up the Firestore Emulators with Next.js](/blog/tutorials/setting-up-firebase-emulators-nextjs)
2. [Setting up Firebase Functions](/docs/next-fire/firebase-functions)

### Why use ElasticSearch?

ElasticSearch is a capable search engine that works very well for various use cases, such as:

- full-text search
- custom weighted search
- data analytics

In a typical SaaS, ElasticSearch could be deployed for various reasons, for example:

- storing and retrieving structured analytical data (clicks, views, sessions, or any sort of data) for **creating interactive dashboards**
- storing unstructured data (such as free text), and then allowing users to search through it and **return the most relevant documents**

Given that databases such as Firebase Firestore do not support full-text search and complex analytical aggregations, ElasticSearch could be a great option to place beside the primary datastore.

## Running ElasticSearch with Docker

The easiest way to install ElasticSearch is using the Docker Image.

If you have Docker installed and running, add the Docker configuration below to your project:

 ```yaml {% title="docker-compose.yml" %}
version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.17.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200"]'
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge
```

You can now run the ELK stack from your IDE or from your terminal. If the latter, use the command below:

```
docker compose up
```

To test that the stack is working, check if Kibana is running at [http://localhost:5601](http://localhost:5601).

{% alert type="info" %}
It may take a while before it's up and running, so don't worry if it doesn't work right away
{% /alert %}

## Running ElasticSearch in a Next.js Application

### Creating the ElasticSearch Client

ElasticSearch comes with an official Node.js client that makes it relatively easy to work with, typically versioned with the same version as its server counterpart ([Elastic.co version](https://elastic.co)).

To install it, run the following command:

```
npm i @elasticsearch/client --save
```

Now we can create a function to initialize the ElasticSearch client:

 ```tsx {% title="elastic.ts" %}
async function createElasticSearchClient(options: ClientOptions) {
  const { Client } = await import('@elastic/elasticsearch');

  return new Client(options);
}
```

The above is pretty simple, but as you can see, we have to provide the options that include the host and the credentials to the ElasticSearch server. Additionally, we provide the local environment host we will use during development.

Let's assume the function `getOptions` can retrieve the configuration from the environment variables. These options have the following interface:

```tsx
{
  node?: string;
  cloud?: string;
  password?: string;
  username?: string;
}
```

The `cloud` property is used by [elastic.co](https://elastic.co) for identifying the node to connect to, so we add it to the example if you choose to build a deployment using the service.

Otherwise, you can use the property `node` to point to an Elastic server. Instead, we point to our Docker instance if `cloud` is not provided.

```tsx
const DEFAULTS: ClientOptions = {
  node: 'http://localhost:9200',
};

function getElasticOptions(): ClientOptions {
  // remember to define "getOptions" to get your credentials!
  const { cloud, password, username } = getOptions();

  // this branch is for prod (i.e. cloud is provided)
  if (cloud) {
    return {
      cloud: {
        id: cloud,
      },
      auth: {
        username,
        password,
      },
    };
  }

  // this branch is for dev
  return {
    node: DEFAULTS.node,
  };
}
```

We dynamically import the Elastic client so that it will be loaded only if used. Now, we can create a factory to get an elastic client:

```tsx
export function getElasticClient() {
  return createElasticSearchClient(
    getElasticOptions()
  );
}
```

For the sake of this blog post, I export the function from a file named `elastic.ts`:

 ```ts {% title="src/core/elastic.ts" %}
import { ClientOptions } from '@elastic/elasticsearch';

async function createElasticSearchClient(options: ClientOptions) {
  const { Client } = await import('@elastic/elasticsearch');

  return new Client(options);
}

const DEFAULTS: ClientOptions = {
  node: 'http://localhost:9200',
};

export function getElasticClient() {
  return createElasticSearchClient(getElasticOptions());
}

function getElasticOptions(): ClientOptions {
  // this branch is for dev
  return {
    node: DEFAULTS.node,
  };
}
```

### Indexing documents with the Node.js ElasticSearch client

The process of indexing documents means we're adding data to our ElasticSearch database, which will, in turn, analyze and ingest it. After getting indexed by ElasticSearch, the document will become searchable. 

ElasticSearch can automatically understand your data's shape when the index is created by indexing a document. With that said, if the data type of the documents is a bit complex (for example, it has nested arrays), you will need to pre-define the schema before it is created.

Indexing a document in ElasticSearch is fairly simple. For example, let's index a simple JSON document using the `index` method:

```tsx
async function indexBlogPost(post: BlogPost) {
  const elastic = await getElasticClient();

  return elastic.index({
    index: `blog-posts`,
    body: post,
  });
}
```

Using the `index` method above, we have:

1. Created an index named `blog-posts` (if this did not exist yet)
2. Added a document containing the properties of `post`

## A practical example: automatically indexing Firestore documents in ElasticSearch

As Firebase's limited search functionality requires a different search engine, we need to work it around by indexing each document written to the database. 

To automate the process, we can use `Firestore Triggers`: these triggers allow us to fire a function every time a document is written or deleted.

### Listening to Firestore document updates

Let's assume we are writing to a Firestore path `tasks/{taskId}`. To receive updates for the path's documents, we can use the Cloud Function below:

 ```tsx {% title="functions/src/index.ts" %}
import { firestore } from 'firebase-functions';

export const onTaskCreated = firestore
  .document('tasks/{taskId}')
  .onWrite((change, context) => {
    // do something here
  });
```

### Writing Firestore documents to ElasticSearch

Now, let's write the logic to push our data to ElasticSearch:

 ```tsx {% title="functions/src/index.ts" %}
import { firestore } from 'firebase-functions';

export const onTaskCreated = firestore
  .document('tasks/{taskId}')
  .onWrite(async (change) => {
    const task = change.after.data();

    if (!task) {
      return;
    }

    // decorate Task data with its ID
    task.id = change.after.id;

    const elastic = await getElasticClient();

    await elastic.index({
      body: task,
      index: `tasks`,
    });

    console.log(`Task successfully indexed`);
  });
```

After creating a document, we can check if it gets created by navigating to Kibana. For example, if you have created an index pattern, you can [check it out in the Discover Tab in Kibana](http://localhost:5601/app/discover). 

{% img src="/assets/images/posts/kibana-discover-tab.webp" width="2544" height="1626" /%}

You should suffix your index names for splitting indexes; alternatively, consider the Lifecycle API that will do it for you based on index size or document age.

Then, we can target the indexes we search using an index pattern: let's assume we name our indexes `tasks-0001` and `tasks-0002`, we can search our index using the pattern `tasks-*`.

### Creating a Next.js API endpoint to query ElasticSearch

Now that we can automatically sync up Elastic and Firestore, we want to be able to query Elastic's documents using an API function. 

To do so, we will write a Next.js endpoint that will fetch the documents from ElasticSearch and return them to the client.

Before doing that, though, let's see how we would write an ElasticSearch query to fetch tasks by their `name` field:

```json
POST tasks*/_search
{
  "query": {
    "match": {
      "name": "<QUERY>"
    }
  }
}
```

Now, let's write the same query using the ElasticSearch Node.js client. We create the API function at `api/tasks/search.ts` with the following content:

 ```tsx {% title="api/tasks/search.ts" %}
async function tasksSearchHandler(
  req: NextApiRequest, 
  res: NextApiResponse
) {
  const { q: query } = req.query;

  // validate the "q" query parameter exists and is a string
  if (!query || typeof query !== 'string') {
    return res.status(400).end();
  }

  const elastic = await getElasticClient();

  const result = await elastic.search({
    index: 'tasks*',
    query: {
      match: {
        name: query,
      },
    },
  });

  // we send back the list of documents found
  const tasks = result.hits.hits.map((item) => item._source);

  return res.json(tasks);
}
```

### Fetching ElasticSearch documents from the Next.js client

Let's create a hook to fetch the data after the user runs a search. 

NB: this is a highly simplified version for fetching data. Best to use libraries such as SWT, React-query, or Makerkit's own `useApiRequest` if you use it.

```tsx
function useSearchTasks() {
  return useCallback((query: string) => {
    return fetch(`/api/tasks/search?q=${query}`)
      .then(r => r.json());
  }, []);
}
```

### Displaying results in a Next.js application

Now, we can query the data from our React components. 

1) First, we want to create a search bar:

```tsx
function SearchInput(props: React.PropsWithChildren<{
  onSearch: (query: string) => void;
}>) {
  return (
    <form onSubmit={e => {
      e.preventDefault();
      
      const query = new FormData(e.currentTarget).get('query');
        props.onSearch(query as string);
    }}>
      <input name='query' placeholder='Search task...' />

      <button>Search Tasks</button>
    </form>
  );
}
```

2) Finally, we can add the `SearchInput` to the `TasksContainer` component, which is responsible for fetching and displaying the tasks if any was found.

```tsx
function TasksContainer() {
  const [tasks, setTasks] = useState<Task[]>();
  const searchTasks = useSearchTasks();

  const onSearch = useCallback(
    async (query: string) => {
      const tasks = await searchTasks(query);

      setTasks(tasks);
    },
    [searchTasks]
  );

  return (
    <>
      <div>Type below to search for your tasks:</div>

      <SearchInput onSearch={onSearch} />

      {tasks && tasks.length ? <TasksList tasks={tasks} /> : <NoTaskFound />}
    </>
  );
}
```

🥳 And here is the final result!

{% Video src="/assets/images/posts/elastic-tasks-demo.mp4" /%}