It may work depending on the specific case… try computing the irradiance map with everything visible; then do one more pass with incremental add to the same map and with “Visible to camera” off for the objects that will have animated visibility - this is to add information about parts of the scene that may have been hidden by those objects; then you can render the final animation.