VRayHDRI will load the texture only when something requests the color of that texture during rendering. For matte objects, it depends - if you have matte shadows enabled, then the underlying material might still be computed and in that case the texture will be loaded.
Best regards,
Vlado