How to make glass render black environment with VRay sky?

Cannot get it to work, I tried with and without checking not visible to refraction (have to leave reflection on, as it is needed)

(But thank you for the tip, I had completely forgotten the override material - it has given me some ideas to try.)