i just found out thet the VrayDisplacementModifier is not compatible with the composit map from MAX.
the simple test is:
take a noise map and do a 3d displacement rendering on a simple sphere.
then do that exactly same noise map into a composit map (normal blend mode), and apply the map again to the displacement modifier and have a look what happens.
well, its not quite what you would expect 
or is there some sort of logic i miss?