How to Fix Memory Leak in Doctrine Migrations
I had to write a Doctrine migration to fix some bad data. Unfortunately there was so much data (~600K records) that the migration leaked. The memory consumption reaches about 12Gb! That’s not really acceptable!
Usually, when there is a leak in Doctrine DBAL (not the ORM), it comes from the SQL logger. When you execute 600K SQL queries, there are 600K log records in the memory!
With Doctrine DBAL 2, we can write the following line of code to avoid the leak:
$this->connection->getConfiguration()->setSQLLogger(null);
However, this doesn’t work anymore with Doctrine DBAL 3! Now they have a middleware system to build the connection. All middlewares wrap the driver to add more capabilities.
So, to fix the migration, I decided to remove all middlewares and create a new connection:
final class Version20230206154337 extends AbstractMigration
{
public function up(Schema $schema): void
{
$this->connection->getConfiguration()->setMiddlewares([]);
$this->connection = DriverManager::getConnection(
$this->connection->getParams(),
$this->connection->getConfiguration(),
);
// regular doctrine migration code
}
Obviously, removing all middlewares can be a bad idea, but in my case only the followings were registered:
Doctrine\DBAL\Logging\Middleware
Doctrine\Bundle\DoctrineBundle\Middleware\DebugMiddleware
Sentry\SentryBundle\Tracing\Doctrine\DBAL\TracingDriverMiddleware
Each middleware brings debug, so it’s safe to remove them all. But some colleagues pushed me to find the culprit, and you may be surprised, but it’s the DebugMiddleware
!
If you prefer to remove only this middleware, you can use the following code:
$middlewares = array_filter(
$this->connection->getConfiguration()->getMiddlewares(),
fn ($middleware) => !$middleware instanceof \Doctrine\Bundle\DoctrineBundle\Middleware\DebugMiddleware
);
$this->connection->getConfiguration()->setMiddlewares($middlewares);
I hope this article can help you to reduce memory leak when dealing with a big amount of data.
Commentaires et discussions
Ces clients ont profité de notre expertise
Afin de soutenir le développement de son trafic, Qobuz a fait appel à JoliCode afin d’optimiser l’infrastructure technique du site et les échanges d’informations entre les composants de la plateforme. Suite à la mise en place de solution favorisant l’asynchronicité et la performance Web côté serveur, nous avons outillé la recherche de performance et…
Groupama confie à JoliCode la maintenance et les évolutions de son outil de souscription d’épargne salariale. Pierre angulaire de l’acquisition de nouveau client, l’outil permet aux apporteurs de saisir toutes les données de l’entreprise, faire signer numériquement le contrat au client et de conclure la souscription. Le tunnel est accompagné d’outil…
Pour améliorer les performances et la pertinence des recherches sur le site e-commerce, JoliCode a réalisé un audit approfondi du moteur Elasticsearch existant. Nous avons optimisé les processus d’indexation, réduisant considérablement les temps nécessaires tout en minimisant les requêtes inutiles. Nous avons également ajusté les analyses pour mieux…