Tweet
Hi All,
I'll try to keep this brief and to the point! We have had a problem recently where individual tiles would sporadically render black whilst others rendered correctly - Max 2018, VRay 4.10.02 and Deadline to do the tiling. After some exploration, I'm confident in stating that the reason they're rendering black is because none of the proxies are being loaded (and since *all* geometry in the scene is proxied, this means it renders black). The proxies are located on the same server as the render output, which significantly reduces (though does not eliminate) the likelyhood of the problem being network or permission related, since the (black) output is being successfully written out.
We have a tool that localises proxies for the artists - it copies them all to a local C: or D: drive and stores their original network location as a user defined property. The tool also enables the user to remap these proxies back to the server (and also allows a different user who might open a max file saved whilst localised - but who does not themselves have the assets localised - to remap them back to the network successfully) - and our render pipeline runs a pre-render script that does all sorts of things including this proxy remapping process, so that all renders on the farm use the network location rather than the localised location, irrespective of whether the file was localised or not when submitting it. This whole system has worked very well for some time, without issues.
The problem with the tiles has only started to occur recently and, so far, only in one project, but since this one project is doing 95% of our rendering right now, it's possible that others are affected and we just haven't seen it. As far as I can tell, the following three things need to be true for a render node to erroneously render black:
1) ) The render submitted has localised VRProxies mapped to the C or D drive.
2) The machine rendering does *not* have a copy of these proxies stored on its C or D drive, and
3) The first tile rendered by a given machine is empty by design (ie it's at the edge of the frame and there's no geo there) - it is then the subsequent tiles that this same machine renders which render out black also when they shouldn't.
If any of the above is not true, the tile renders fine. I can't tell if it *always* renders black if these three are true, but I have yet to find an instance of it rendering erroneously if any of these are not true.
Could this be a bug with VRay's loading? My suspicion is that it is a problem relating to a combination of a) the scene having a bunch of missing proxies when it's opened, which only get re-linked during the pre-render script and b) the first tile having no geometry, leading to VRay not loading any proxies into RAM. Somehow this is then causing VRay to not load proxies on subsequent render tasks, even when the buckets should be hitting the geo. This doesn't really make any sense (and as I say, this system has been working fine for a long time), but given the combination of requirements it's the only thing that vaguely makes sense to me. I understand that proxies are only loaded if a bucket requires them - perhaps the fact they're missing prior to the remap is throwing out VRay's ability to determine if they're needed - but only if the first task actually doesn't require any loaded? A few final things:
- If I tell Deadline to restart Max between each frame, the renders work successfully, including the same tiles and same machines rendering incorrectly on the same job. If I switch this setting back (so it does not reload Max) and requeue the same tasks, the node will node fail. This leads me to believe that the relinking process is working correctly, but VRay is nontheless failing to load them.
- Deadline's log does not complain about missing maps or proxies.
- If the machine happens to have the localised proxies in the same location (ie it's another user's workstation that's been put on the farm who also localised the file and thus has the assets locally), they render fine. This leads me to believe that the act of opening the file without the proxies being found that's contributing to the problem.
- All proxy objects are situated at 0, 0, 0. If you open a file where they're localised and you don't have a local copy, you see what appears to be thousands of over-lapping boxes at the origin. Given my theory above, could this be relevant?
- The "solution" at the moment to ensure our renders work correctly is to return the proxies to the network location before rendering - this is a fine workaround, but it can be a lengthy process that shouldn't be required from the user.
If anyone has any insight, I'd be really happy to hear it as this is causing me to tear my hair out!
Thank you for reading,
Dan
I'll try to keep this brief and to the point! We have had a problem recently where individual tiles would sporadically render black whilst others rendered correctly - Max 2018, VRay 4.10.02 and Deadline to do the tiling. After some exploration, I'm confident in stating that the reason they're rendering black is because none of the proxies are being loaded (and since *all* geometry in the scene is proxied, this means it renders black). The proxies are located on the same server as the render output, which significantly reduces (though does not eliminate) the likelyhood of the problem being network or permission related, since the (black) output is being successfully written out.
We have a tool that localises proxies for the artists - it copies them all to a local C: or D: drive and stores their original network location as a user defined property. The tool also enables the user to remap these proxies back to the server (and also allows a different user who might open a max file saved whilst localised - but who does not themselves have the assets localised - to remap them back to the network successfully) - and our render pipeline runs a pre-render script that does all sorts of things including this proxy remapping process, so that all renders on the farm use the network location rather than the localised location, irrespective of whether the file was localised or not when submitting it. This whole system has worked very well for some time, without issues.
The problem with the tiles has only started to occur recently and, so far, only in one project, but since this one project is doing 95% of our rendering right now, it's possible that others are affected and we just haven't seen it. As far as I can tell, the following three things need to be true for a render node to erroneously render black:
1) ) The render submitted has localised VRProxies mapped to the C or D drive.
2) The machine rendering does *not* have a copy of these proxies stored on its C or D drive, and
3) The first tile rendered by a given machine is empty by design (ie it's at the edge of the frame and there's no geo there) - it is then the subsequent tiles that this same machine renders which render out black also when they shouldn't.
If any of the above is not true, the tile renders fine. I can't tell if it *always* renders black if these three are true, but I have yet to find an instance of it rendering erroneously if any of these are not true.
Could this be a bug with VRay's loading? My suspicion is that it is a problem relating to a combination of a) the scene having a bunch of missing proxies when it's opened, which only get re-linked during the pre-render script and b) the first tile having no geometry, leading to VRay not loading any proxies into RAM. Somehow this is then causing VRay to not load proxies on subsequent render tasks, even when the buckets should be hitting the geo. This doesn't really make any sense (and as I say, this system has been working fine for a long time), but given the combination of requirements it's the only thing that vaguely makes sense to me. I understand that proxies are only loaded if a bucket requires them - perhaps the fact they're missing prior to the remap is throwing out VRay's ability to determine if they're needed - but only if the first task actually doesn't require any loaded? A few final things:
- If I tell Deadline to restart Max between each frame, the renders work successfully, including the same tiles and same machines rendering incorrectly on the same job. If I switch this setting back (so it does not reload Max) and requeue the same tasks, the node will node fail. This leads me to believe that the relinking process is working correctly, but VRay is nontheless failing to load them.
- Deadline's log does not complain about missing maps or proxies.
- If the machine happens to have the localised proxies in the same location (ie it's another user's workstation that's been put on the farm who also localised the file and thus has the assets locally), they render fine. This leads me to believe that the act of opening the file without the proxies being found that's contributing to the problem.
- All proxy objects are situated at 0, 0, 0. If you open a file where they're localised and you don't have a local copy, you see what appears to be thousands of over-lapping boxes at the origin. Given my theory above, could this be relevant?
- The "solution" at the moment to ensure our renders work correctly is to return the proxies to the network location before rendering - this is a fine workaround, but it can be a lengthy process that shouldn't be required from the user.
If anyone has any insight, I'd be really happy to hear it as this is causing me to tear my hair out!
Thank you for reading,
Dan
