That is just your alpha being premultiplied against black in your rendering, and then viewed over white in PS. Go to the menu => Layer => Matting => Remove black matte, and you should be good.
Unless I totally misunderstood what you were asking.
as trixian said, this has nothing to do with vray. it’s rather ps that doesn’t know how to deal with premultiplied images. removing black matte can help a wee bit, but for this kind of compositing with ps you’re probably better off saving your output in a format which allow you to choose not to premultiply by its alpha.
Turn your AA filter off entirely, knock up your max subdivisions and sharpen it in post, if you need to. If it doesnt fix it completley, it’ll go a long way to helping.