Log all the searches going through Elasticsearch

You are looking for a way to retrieve the full Query DSL sent by an application to Elasticsearch in order to debug or simply see what’s going on. This article got you covered.

Sometimes we cannot inspect the HTTP query before it gets to Elasticsearch – maybe because we don’t control the application, or because there is no logger, or because we cannot edit production running code, etc.

Thankfully there is a simple way to log all searches and indexation requests sent to Elasticsearch in the engine log file.

Enable logging of all searches

We are going to leverage the slowlog functionality. It’s already enabled by default but with a 5 seconds threshold on the info level.

We lower the threshold of the trace level and we switch to it:

PUT foobar/_settings
{
  "index.search.slowlog.threshold.query.trace": "0s",
  "index.search.slowlog.level": "trace"
}

Inspecting the logs

Now every search request sent to Elasticsearch will appear in the logs like this:

{"type": "index_search_slowlog", "timestamp": "2021-03-17T22:25:45,839Z", "level": "DEBUG", "component": "i.s.s.query", "cluster.name": "docker-cluster", "node.name": "es01", "message": "[foobar][0]", "took": "95.8micros", "took_millis": "0", "total_hits": "0 hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"query\":{\"multi_match\":{\"query\":\"Covfefe\",\"fields\":[\"description^1.0\",\"title^10.0\"],\"type\":\"best_fields\",\"operator\":\"OR\",\"slop\":0,\"prefix_length\":0,\"max_expansions\":50,\"zero_terms_query\":\"NONE\",\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}}", "cluster.uuid": "-YsHSUgLTbG20EmG0GRAbg", "node.id": "DwoE3nCWQvGGPk-M-7rKhQ"  }

The part we want is inside the source field. As it is JSON inside a JSON it’s escaped, we just need to replace \" by " and we are good to go.

{"query":{"multi_match":{"query":"Covfefe","fields":["description^1.0","title^10.0"],"type":"best_fields","operator":"OR","slop":0,"prefix_length":0,"max_expansions":50,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1}}}

We can run this request in Kibana or any tool we like.

Note that the query we see here also contains the default options of search queries, so it’s not exactly the same Query DSL produced by the application.

Disable logging of all searches

To go back to the default configuration:

PUT foobar/_settings
{
  "index.search.slowlog.threshold.query.trace": "500ms",
  "index.search.slowlog.level": "info"
}

It also works for indexing requests

This is less common but we can do the same with the indexing options.

Enabling slowlog with full source (warning this can be heavy):

PUT foobar/_settings
{
  "index.indexing.slowlog.threshold.index.trace": "0s",
  "index.indexing.slowlog.level": "trace",
  "index.indexing.slowlog.source": true
}

Getting back to the defaults:

PUT foobar/_settings
{
  "index.indexing.slowlog.threshold.index.trace": "500ms",
  "index.indexing.slowlog.level": "info",
  "index.indexing.slowlog.source": "1000"
}

These tips and a lot of other nice “copy and paste” snippets are part of our Elasticsearch Cheatsheet.

Happy debug!