I used render mask for animation and I noticed something odd. When rendering the next frame the previous frame remains in the vfb and it saves out with the current frame on top of the previous frame. Instead of starting a fresh render.
The render mask doesn’t touch the parts of the frame that are not it the mask and for animation the frames remain. You could use the “Clear” option for the Previous render (in Settings - System Advanced).