Log all the searches going through Elasticsearch
You are looking for a way to retrieve the full Query DSL sent by an application to Elasticsearch in order to debug or simply see what’s going on. This article got you covered.
Sometimes we cannot inspect the HTTP query before it gets to Elasticsearch – maybe because we don’t control the application, or because there is no logger, or because we cannot edit production running code, etc.
Thankfully there is a simple way to log all searches and indexation requests sent to Elasticsearch in the engine log file.
Enable logging of all searches
We are going to leverage the slowlog functionality. It’s already enabled by default but with a 5 seconds threshold on the info
level.
We lower the threshold of the trace
level and we switch to it:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "0s",
"index.search.slowlog.level": "trace"
}
Inspecting the logs
Now every search request sent to Elasticsearch will appear in the logs like this:
{"type": "index_search_slowlog", "timestamp": "2021-03-17T22:25:45,839Z", "level": "DEBUG", "component": "i.s.s.query", "cluster.name": "docker-cluster", "node.name": "es01", "message": "[foobar][0]", "took": "95.8micros", "took_millis": "0", "total_hits": "0 hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"query\":{\"multi_match\":{\"query\":\"Covfefe\",\"fields\":[\"description^1.0\",\"title^10.0\"],\"type\":\"best_fields\",\"operator\":\"OR\",\"slop\":0,\"prefix_length\":0,\"max_expansions\":50,\"zero_terms_query\":\"NONE\",\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}}", "cluster.uuid": "-YsHSUgLTbG20EmG0GRAbg", "node.id": "DwoE3nCWQvGGPk-M-7rKhQ" }
The part we want is inside the source
field. As it is JSON inside a JSON it’s escaped, we just need to replace \"
by "
and we are good to go.
{"query":{"multi_match":{"query":"Covfefe","fields":["description^1.0","title^10.0"],"type":"best_fields","operator":"OR","slop":0,"prefix_length":0,"max_expansions":50,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1}}}
We can run this request in Kibana or any tool we like.
Note that the query we see here also contains the default options of search queries, so it’s not exactly the same Query DSL produced by the application.
Disable logging of all searches
To go back to the default configuration:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.level": "info"
}
It also works for indexing requests
This is less common but we can do the same with the indexing
options.
Enabling slowlog with full source (warning this can be heavy):
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "0s",
"index.indexing.slowlog.level": "trace",
"index.indexing.slowlog.source": true
}
Getting back to the defaults:
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "500ms",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}
These tips and a lot of other nice “copy and paste” snippets are part of our Elasticsearch Cheatsheet.
Happy debug!