Not sure if this is a VRay problem, or Max, but when I assign a material with a VRayDirt material assigned, and attempt to render for AO pass, Max crashes as soon as the final bucket render starts (also happens with progressive renders). Thinking this is a VRay error since I can bake maps with both Standard and MentalRay.
Make sure you are using only V-Ray specific bake elements. If you use any standard Max ones, you will get a crash. It is somewhere on the “to do” list to fix this.
Best regards,
Vlado