So nothing at all changes when you up the GI settings? Are you using a VRayLight with area shadows along with that HDRI? Maybe the MB distributes the lights Subdivs over the time frame of the dynamic geometry. Could also have something to do with reflected GI caustics.
I had a similar problem with an animation and it turned out that I had to calculate the irradiance map for each frame, rather than incremental calculation. Took forever to render…