MongoDB | 2654731

Info

The _flushReshardingStateChange command uses the arbitraryExecutor in order to flush the metadata by calling onCollectionVersionPlacementMismatch. This method may block by either: Taking a collection lock Waiting on the critical section future This is a problem because the arbitrary executor does not expect any blocking methods to be called in there. This is because it is essentially always the same one in sharding, which is one that piggybacks on the Networking ASIO threads. This means that it will be a thread pool with a fixed size and prone to deadlocks if mishandled. In this situation there is a potential deadlock waiting to happen given this scenario: Operation A is in a multi-document transaction against collection C which uses an AsyncResultsMerger. Operation B starts a resharding operation on C and enqueues a MODE_S lock since it cannot acquire it. Operation B triggers an eventual execution (Operation C) of the _flushReshardingStateChange command that blocks until the critical section is signalled by resharding. Operation A now attempts to kill the AsyncResultsMerger, this waits until all callbacks are executed to avoid invalid memory accesses. These callbacks are executed on the arbitraryExecutor. At this point Operation A is waiting until Operation C finishes which waits on Operation B and in turn waits on Operation A again.

Top User Comments

xgen-internal-githook commented on Fri, 26 Apr 2024 17:37:33 +0000: Author: {'name': 'Jordi Olivares Provencio', 'email': 'jordi.olivares-provencio@mongodb.com', 'username': 'jordiolivares'} Message: SERVER-89893 Use fixed executor in _flushReshardingStateChange command (#21547) GitOrigin-RevId: 79ed4afa6014c5eb2aac041c08f6520af4e9b294 Branch: master https://github.com/mongodb/mongo/commit/937b2da6cc4fd753e908c47c7093d04f556f81f9 max.hirschhorn@10gen.com commented on Fri, 26 Apr 2024 13:06:41 +0000: Thanks jordi.olivares-provencio@mongodb.com for catching this. I had it wrong in SERVER-58081 and it is actually the fixed executor which is using ThreadPool::Options::kUnlimited to configure its maximum number of threads.

Steps to Reproduce