I am getting a white fringe around the edge of my renders. The images are saved as tifs with the alpha channel saved then removed in photoshop using the Load channel as selection and Add layer mask commands.
Be sure to render your image in front of a black background, this reduces those white fringes a lot.
Next, try to apply a smooth selection with 1px within PS. Dunno the correct english conversion, but in german version it’s called “Weiche Auswahlkante”
Render with a black background, make sure to set premultiplied alpha in your save as targa options, then use Jon Seagull’s Photoshop TGA to Transparency action: Jon Seagull
It’s not a vray thing, it is a pre-multiply thing, To antialias the edges it needs to blend with something, that something is your background color. Compositing packages handle this by replacing the black (best background color to use) or whatever color you choose with opacity on the fly. Photoshop does the same thing for white or black.