“The VFB will mostly clear the image if render elements have been added or removed” - this is indeed the case, thank you for your answer.
I disable render elements during test renders because I assume it’ll speed up rendering. (I only noticed it now since I usually use Maya’s framebuffer).