How to Fix Memory Leak in Doctrine Migrations
I had to write a Doctrine migration to fix some bad data. Unfortunately there was so much data (~600K records) that the migration leaked. The memory consumption reaches about 12Gb! That’s not really acceptable!
Usually, when there is a leak in Doctrine DBAL (not the ORM), it comes from the SQL logger. When you execute 600K SQL queries, there are 600K log records in the memory!
With Doctrine DBAL 2, we can write the following line of code to avoid the leak:
$this->connection->getConfiguration()->setSQLLogger(null);
However, this doesn’t work anymore with Doctrine DBAL 3! Now they have a middleware system to build the connection. All middlewares wrap the driver to add more capabilities.
So, to fix the migration, I decided to remove all middlewares and create a new connection:
final class Version20230206154337 extends AbstractMigration
{
public function up(Schema $schema): void
{
$this->connection->getConfiguration()->setMiddlewares([]);
$this->connection = DriverManager::getConnection(
$this->connection->getParams(),
$this->connection->getConfiguration(),
);
// regular doctrine migration code
}
Obviously, removing all middlewares can be a bad idea, but in my case only the followings were registered:
Doctrine\DBAL\Logging\Middleware
Doctrine\Bundle\DoctrineBundle\Middleware\DebugMiddleware
Sentry\SentryBundle\Tracing\Doctrine\DBAL\TracingDriverMiddleware
Each middleware brings debug, so it’s safe to remove them all. But some colleagues pushed me to find the culprit, and you may be surprised, but it’s the DebugMiddleware
!
If you prefer to remove only this middleware, you can use the following code:
$middlewares = array_filter(
$this->connection->getConfiguration()->getMiddlewares(),
fn ($middleware) => !$middleware instanceof \Doctrine\Bundle\DoctrineBundle\Middleware\DebugMiddleware
);
$this->connection->getConfiguration()->setMiddlewares($middlewares);
I hope this article can help you to reduce memory leak when dealing with a big amount of data.
Commentaires et discussions
Ces clients ont profité de notre expertise
Nous avons développé une plateforme de site génériques autour de l’API Phraseanet. À l’aide de Silex de composants Symfony2, nous avons accompagné Alchemy dans la réalisation d’un site déclinable pour leurs clients. Le produit est intégralement configurable et supporte de nombreux systèmes d’authentification (Ldap, OAuth2, Doctrine ou anonyme).
Dans le cadre de notre partenariat avec Carb0n, nous avons réalisé un examen complet de leur infrastructure, allant de l’environnement de production, jusqu’à au code de l’application. Nous avons fourni des recommandations stratégiques pour son amélioration. Notre intervention a notamment couvert : Évaluation et renforcement de la sécurité sur AWS, …
LOOK Cycle bénéficie désormais d’une nouvelle plateforme eCommerce disponible sur 70 pays et 5 langues. La base technique modulaire de Sylius permet de répondre aux exigences de LOOK Cycle en terme de catalogue, produits, tunnel d’achat, commandes, expéditions, gestion des clients et des expéditions.