Log all the searches going through Elasticsearch
You are looking for a way to retrieve the full Query DSL sent by an application to Elasticsearch in order to debug or simply see what’s going on. This article got you covered.
Sometimes we cannot inspect the HTTP query before it gets to Elasticsearch – maybe because we don’t control the application, or because there is no logger, or because we cannot edit production running code, etc.
Thankfully there is a simple way to log all searches and indexation requests sent to Elasticsearch in the engine log file.
Section intitulée enable-logging-of-all-searchesEnable logging of all searches
We are going to leverage the slowlog functionality. It’s already enabled by default but with a 5 seconds threshold on the info
level.
We lower the threshold of the trace
level and we switch to it:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "0s",
"index.search.slowlog.level": "trace"
}
Section intitulée inspecting-the-logsInspecting the logs
Now every search request sent to Elasticsearch will appear in the logs like this:
{"type": "index_search_slowlog", "timestamp": "2021-03-17T22:25:45,839Z", "level": "DEBUG", "component": "i.s.s.query", "cluster.name": "docker-cluster", "node.name": "es01", "message": "[foobar][0]", "took": "95.8micros", "took_millis": "0", "total_hits": "0 hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"query\":{\"multi_match\":{\"query\":\"Covfefe\",\"fields\":[\"description^1.0\",\"title^10.0\"],\"type\":\"best_fields\",\"operator\":\"OR\",\"slop\":0,\"prefix_length\":0,\"max_expansions\":50,\"zero_terms_query\":\"NONE\",\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}}", "cluster.uuid": "-YsHSUgLTbG20EmG0GRAbg", "node.id": "DwoE3nCWQvGGPk-M-7rKhQ" }
The part we want is inside the source
field. As it is JSON inside a JSON it’s escaped, we just need to replace \"
by "
and we are good to go.
{"query":{"multi_match":{"query":"Covfefe","fields":["description^1.0","title^10.0"],"type":"best_fields","operator":"OR","slop":0,"prefix_length":0,"max_expansions":50,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1}}}
We can run this request in Kibana or any tool we like.
Note that the query we see here also contains the default options of search queries, so it’s not exactly the same Query DSL produced by the application.
Section intitulée disable-logging-of-all-searchesDisable logging of all searches
To go back to the default configuration:
PUT foobar/_settings
{
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.level": "info"
}
Section intitulée it-also-works-for-indexing-requestsIt also works for indexing requests
This is less common but we can do the same with the indexing
options.
Enabling slowlog with full source (warning this can be heavy):
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "0s",
"index.indexing.slowlog.level": "trace",
"index.indexing.slowlog.source": true
}
Getting back to the defaults:
PUT foobar/_settings
{
"index.indexing.slowlog.threshold.index.trace": "500ms",
"index.indexing.slowlog.level": "info",
"index.indexing.slowlog.source": "1000"
}
These tips and a lot of other nice “copy and paste” snippets are part of our Elasticsearch Cheatsheet.
Happy debug!
Commentaires et discussions
Elasticsearch the right way in Symfony
You are building an application with Symfony – good choice 😜 – but now you need some full-text search capabilities? This article is for you. Multiple options are available: going full RDMS and using FULLTEXT indexes – yes it works; using a third party SaaS like Algolia or Elastic…
Lire la suite de l’article Elasticsearch the right way in Symfony
Nos articles sur le même sujet
Nos formations sur ce sujet
Notre expertise est aussi disponible sous forme de formations professionnelles !
Elasticsearch
Indexation et recherche avancée, scalable et rapide avec Elasticsearch
Ces clients ont profité de notre expertise
Après avoir monté une nouvelle équipe de développement, nous avons procédé à la migration de toute l’infrastructure technique sur une nouvelle architecture fortement dynamique à base de Symfony2, RabbitMQ, Elasticsearch et Chef. Les gains en performance, en stabilité et en capacité de développement permettent à l’entreprise d’engager de nouveaux marchés…
Qobuz nous a également sollicité pour la refonte de son moteur de recherche pour employer Elasticsearch, dont JoliCode a une très forte expertise. L’indexation en temps réel et les problématique de droits sur les contenus musicaux ont été les principales difficultés rencontrées. Au final, l’emploi d’ElasticSearch et notre approche technique ont permis…
Nous avons développé un outil statistique complet développé pour ORPI. Basé sur PHP, Symfony et Elasticsearch, cet outil offre à toutes les agences du réseau une visibilité accrue sur leurs annonces. Il garantit également une transparence totale envers les clients, en fournissant des statistiques détaillées sur les visualisations et les contacts de…