How to visualize Symfony logs in dev with Elasticsearch and Kibana

Symfony comes with Monolog and some extension like easy-log-handler that writes logs in a fancier format in var/log/dev.log.

But if you are building a complex application, many API endpoints can be hit at the same time, many workers can run at the same time, … and so, finding something can be very difficult.

To debug in production, we use the ELK stack: Elasticsearch, Logstash, and Kibana. These tools are really powerful and we already wrote about it 🇫🇷.

You may have tried but running Logstash and writing the good configuration is boring. That’s why I contributed to Symfony few months ago an ElasticsearchLogstashHandler.

This handler pushes logs directly to Elasticsearch and format them to something similar to Logstash specification. It dials directly with the HTTP interface of Elasticsearch. This means it will slow down your application if Elasticsearch takes time to answer. Even if all HTTP calls are done asynchronously. So use it only in a dev environment. Actually, it could be possible to use it in a production environment, but you will need to wrap this handler in a handler with buffering capabilities (like the FingersCrossedHandler, or BufferHandler) in order to call Elasticsearch only once with a bulk push. For even better performance and fault tolerance, a proper ELK stack is recommended.

Section intitulée how-to-use-itHow to use it?

Section intitulée 1-enable-the-handler1. Enable the handler:

# config/packages/dev/monolog.yaml
monolog:
    handlers:
        es:
            type: service
            id: Symfony\Bridge\Monolog\Handler\ElasticsearchLogstashHandler

# config/packages/services.yaml
services:
    Symfony\Bridge\Monolog\Handler\ElasticsearchLogstashHandler:
        autowire: false

Section intitulée 2-run-some-docker-containers2. Run some docker containers:

It’s possible to not use docker, but docker really eases this step. So here we go:

docker network create sf-es-k
docker run -it --rm --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" --network sf-es-k docker.elastic.co/elasticsearch/elasticsearch:7.4.2
docker run -it --rm -p 5601:5601 --network sf-es-k docker.elastic.co/kibana/kibana:7.4.2

Section intitulée 3-profit3. Profit:

1. Open http://127.0.0.1:5601/app/kibana#/management/kibana/index_pattern;
2. Hit you application at least one time to generate some logs;
3. Use monolog* as index pattern;
4. Click on Next step;
5. Use @timestamp as Time Filter field name;
6. Click on Create index pattern;
7. Return on the discover view the discover view and 🤩.

Section intitulée 4-bonus4. Bonus:

I also recommend you to enable more processors:

• TokenProcessor: Adds information from the current user’s token to the record namely username, roles and whether the user is authenticated;
• WebProcessor: Overrides data from the request using the data inside Symfony’s request object;
• RouteProcessor: Adds information about current route (controller, action, route parameters);
• ConsoleCommandProcessor: Adds information about current console command;
• UidProcessor: Adds a unique identifier into records.

The last processor UidProcessor is really important to me. It allows to group all logs for only one message (HTTP request or AMQP message). Without that, all logs will be melded together.

Note: If you are using some workers, don’t forget to reset monolog service after each (AMQP) message:

$this->logger->reset();