Well, I use Z-Brush and Vray all the time. I actually used to run into this issue quite a lot until I finally discovered the way that works for me:
IN ZBRUSH>
1.) Build your base model in Z-brush with Z-Spheres or other modeling program (Silo, Modo, Max, whatever).
2.) Save the tool as a polymesh in ZBrush or import form other program. You should be sculpting on a Polymesh and not the original Z-spheres model.
3.) Choose GOV or AV before you start sculpting. Subdivide (to level 7 or so) and Sculpt your displacement… Don’t forget to save the tool.
4.) Go to the base level (Level 1) of your mesh and create a displacement map at a very high resolution. I usually just crank it up to 4k but you can make it smaller is memory is an issue. I use “Smooth UV” and “Adaptive” when creating.
5.) Go to the “Alpha” menu and choose the newly created Alpha Map, flip it vertically and export as a Tiff image.
6.) Export the base mesh as an OBJ.
IN MAX>
1.) Import the OBJ and put a VrayMtl on it.
2.) In the material editor, add the TIFF Image to the displacement slot. Sometimes I stick it in the Bump slot too if it still looks to soft in the end. Make sure to turn off filtering in the texture map. This is important because Max’s filtering really messes up 16 bit per pixel images.
3.) Add a VrayDisplacement modifier to the mesh.
4.) Drag and drop the displacement map from the material editor to the displacement slot of the VrayDisplacemnt modifier. Choose to make an “instance” of the original
5.) Choose “Smooth” Not 2d or 3d! This will smooth and displace the mesh dynamically and helps save memory. In my opinion, it’s better to do this then adding a mesh or turbo smooth because it saves memory. When you’re rendering huge amounts of data it really does make a difference. besides, if your using a displacement map, Vray is going to control the subdivision level anyway so why not just have it smooth it dynamically as well.
6.) Set the “use tight bounds”, “Keep continuity”, and “use material” check boxes in displacement modifier.
7.) Play with the “Shift” and “Amount” settings until you get it to look right. In my experience, Shift should be negative one half of the “Amount” value. So if you have an Amount of 4 your shift should be -2.
That should do it. The key to this is not using Max’s mesh smooth and letting Vray handle that and displacement at the same time.