I got this scene where a sphere is a matte object with shadows checked.
here is image describes what happens whith it.
the strange shadow border on img 3 is produced by max omni light with vray shadows. The broken faces shadow on img 4 is produced by vray light. I have tried setting ray bias to different values but get same deal.
Scene is here
http://www3.telus.net/public/vinnik88/vrayBugs/vray\_bug\_.max