Log all the searches going through Elasticsearch
You are looking for a way to retrieve the full Query DSL sent by an application to Elasticsearch in order to debug or simply see what’s going on. This article got you covered.
Sometimes we cannot inspect the HTTP query before it gets to Elasticsearch – maybe because we don’t control the application, or because there is no logger, or because we cannot edit production running code, etc.
Thankfully there is a simple way to log all searches and indexation requests sent to Elasticsearch in the engine log file.
Section intitulée enable-logging-of-all-searchesEnable logging of all searches
We are going to leverage the slowlog functionality. It’s already enabled by default but with a 5 seconds threshold on the info
level.
We lower the threshold of the trace
level and we switch to it:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "0s",
"index.search.slowlog.level": "trace"
}
Section intitulée inspecting-the-logsInspecting the logs
Now every search request sent to Elasticsearch will appear in the logs like this:
{"type": "index_search_slowlog", "timestamp": "2021-03-17T22:25:45,839Z", "level": "DEBUG", "component": "i.s.s.query", "cluster.name": "docker-cluster", "node.name": "es01", "message": "[foobar][0]", "took": "95.8micros", "took_millis": "0", "total_hits": "0 hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"query\":{\"multi_match\":{\"query\":\"Covfefe\",\"fields\":[\"description^1.0\",\"title^10.0\"],\"type\":\"best_fields\",\"operator\":\"OR\",\"slop\":0,\"prefix_length\":0,\"max_expansions\":50,\"zero_terms_query\":\"NONE\",\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}}", "cluster.uuid": "-YsHSUgLTbG20EmG0GRAbg", "node.id": "DwoE3nCWQvGGPk-M-7rKhQ" }
The part we want is inside the source
field. As it is JSON inside a JSON it’s escaped, we just need to replace \"
by "
and we are good to go.
{"query":{"multi_match":{"query":"Covfefe","fields":["description^1.0","title^10.0"],"type":"best_fields","operator":"OR","slop":0,"prefix_length":0,"max_expansions":50,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1}}}
We can run this request in Kibana or any tool we like.
Note that the query we see here also contains the default options of search queries, so it’s not exactly the same Query DSL produced by the application.
Section intitulée disable-logging-of-all-searchesDisable logging of all searches
To go back to the default configuration:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.level": "info"
}
Section intitulée it-also-works-for-indexing-requestsIt also works for indexing requests
This is less common but we can do the same with the indexing
options.
Enabling slowlog with full source (warning this can be heavy):
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "0s",
"index.indexing.slowlog.level": "trace",
"index.indexing.slowlog.source": true
}
Getting back to the defaults:
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "500ms",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}
These tips and a lot of other nice “copy and paste” snippets are part of our Elasticsearch Cheatsheet.
Happy debug!
Commentaires et discussions
Elasticsearch the right way in Symfony
You are building an application with Symfony – good choice 😜 – but now you need some full-text search capabilities? This article is for you. Multiple options are available: going full RDMS and using FULLTEXT indexes – yes it works; using a third party SaaS like Algolia or Elastic…
Lire la suite de l’article Elasticsearch the right way in Symfony
Nos articles sur le même sujet
Nos formations sur ce sujet
Notre expertise est aussi disponible sous forme de formations professionnelles !
Elasticsearch
Indexation et recherche avancée, scalable et rapide avec Elasticsearch
Ces clients ont profité de notre expertise
Dans le cadre du renouveau de sa stratégie digitale, Orpi France a fait appel à JoliCode afin de diriger la refonte du site Web orpi.com et l’intégration de nombreux nouveaux services. Pour effectuer cette migration, nous nous sommes appuyés sur une architecture en microservices à l’aide de PHP, Symfony, RabbitMQ, Elasticsearch et Docker.
Qobuz nous a également sollicité pour la refonte de son moteur de recherche pour employer Elasticsearch, dont JoliCode a une très forte expertise. L’indexation en temps réel et les problématique de droits sur les contenus musicaux ont été les principales difficultés rencontrées. Au final, l’emploi d’ElasticSearch et notre approche technique ont permis…
Dans le cadre d’une refonte complète de son architecture Web, le journal en ligne Mediapart a sollicité l’expertise de JoliCode afin d’accompagner ses équipes. Mediapart.fr est un des rares journaux 100% en ligne qui n’appartient qu’à ses lecteurs qui amène un fort traffic authentifiés et donc difficilement cachable. Pour effectuer cette migration, …