VRay 3.10.03 Distributed Problems

[electricart]

Hi,

We're having a particular problem with our render nodes with VRay 3.10.03. It happens on a number of different scenes for us and is proving to be extremely frustrating, as it comes and goes and appears to happen randomly, and we have yet to find a solution.

The problem is that certain buckets render much darker than others, as if lights or a map is absent.

We have 'Restart Servers on Render End' switched on.
We use UNC paths and have checked that all paths are resolved correctly.
Permissions on the server are valid for access for the nodes.
Upon reboot, the "%TEMP%" folder for the render user and "%LOCALAPPDATA%\Autodesk\3dsMax" folders are cleaned via script.
There is no GI in this scene.
We are using VRay Spawner as a service.
All nodes and workstations are on the same version of VRay.

Re-rendering the same scene will result in a different node displaying the problem, so it does not seem to be node specific.

Rebooting the render nodes completely does not solve the issue.

Here is an example of the problem.



Here are the most notable lines/errors from the log of one of the nodes.

[2015/May/7|14:45:08] Broadcasting TM_SERVER_STARTED after scene error
[2015/May/7|14:45:08] Received TM_STREAM_ERROR from 0.0.0.0
[2015/May/7|14:45:08] Receiving DR scene from 192.168.1.84
[2015/May/7|14:45:08] error: Error receiving DR scene (0), closing DR session
[B][2015/May/7|14:45:08] warning: Failed to delete server scene file "C:\Users\render\AppData\Local\Autodesk\3dsMax\2015 - 64bit\ENU\temp\vray_server_scene.max" (183)[/B]

...

[B][2015/May/7|14:45:35]   "default" preset not loaded: [loadPreset] Cannot open file "C:\Users\render\AppData\Local\Autodesk\3dsMax\2015 - 64bit\ENU\en-US\plugcfg\vray.cfg"[/B]

...

[2015/May/7|14:46:12] [DR] Calling beginSeqeunce()
[B][2015/May/7|14:46:12] Current project folder is "C:\Users\vlado\Documents\3dsMax"[/B]
[2015/May/7|14:46:12] Entering beginSequence().

This leads me to a few questions;

Why is the project folder set to Vlado's documents?

When is the "%LOCALAPPDATA%\Autodesk\3dsMax" folder populated with the vray.cfg file? I.e. when does

C:\Users\render\AppData\Local\Autodesk\3dsMax\2015 - 64bit\ENU\en-US\plugcfg\vray.cfg

Get copied in? It must not be just when Max is started as a slave by VRay Spawner, something else must happen, because if we remove the "C:\Users\render\AppData\Local\Autodesk\3dsMax " folder, the folder gets re-created automatically by Max when Max is started as slave by VRay Spawner, however the vray.cfg is missing.

Also what causes this;

warning: Failed to delete server scene file "C:\Users\render\AppData\Local\Autodesk\3dsMax\2015 - 64bit\ENU\temp\vray_server_scene.max"

And how can we fix it? Or is it not necessary to fix this error?

Any help on this matter would be greatly appreciated.

kind regards,
Tristan.

[tashko.zashev]

Initially please confirm that you have only one V-Ray spawner service running on the render nodes?
Also please confirm that issue with the different buckets occurs on different node each time you start a new render process? It is important for our further troubleshooting. You can check this with our DRbucket render element.

[2015/May/7|14:45:08] warning: Failed to delete server scene file "C:\Users\render\AppData\Local\Autodesk\3dsMax\201 5 - 64bit\ENU\temp\vray_server_scene.max" (183)

Usually this warning appears when the scene takes more memory and "Restart Servers on Render End" is enabled. V-Ray tries to delete the temp file during the memory release operations.

"default" preset not loaded: [loadPreset] Cannot open file "C:\Users\render\AppData\Local\Autodesk\3dsMax\201 5 - 64bit\ENU\en-US\plugcfg\vray.cfg"

This error should not be related to the issue you have.

Current project folder is "C:\Users\vlado\Documents\3dsMax"

V-Ray changes the project folder on all render nodes participating in the DR render. What is the project folder set on the Main machine?

[electricart]

Hi there, thank you so much for your quick reply.

Initially please confirm that you have only one V-Ray spawner service running on the render nodes?

Actually we have two Spawner services running - one for Max 2014, one for Max 2015.

Also please confirm that issue with the different buckets occurs on different node each time you start a new render process? It is important for our further troubleshooting.

We have checked this and yes, it is a different node each time. If we remove one node from the render (any node, not even the problematic node from the most recent render), it will tend to render ok a few times, then the problem will return.

Usually this warning appears when the scene takes more memory and "Restart Servers on Render End" is enabled.

Sorry I'm not sure I understand. More memory than what? None of our test scenes come particularly close to maxing out the memory of the nodes - each node has 24GB RAM, usually there is at least 8GB free memory on the node during render.

V-Ray tries to delete the temp file during the memory release operations.

Ok but why does it fail to then delete it? Could this be causing our issue?

"default" preset not loaded: [loadPreset] Cannot open file "C:\Users\render\AppData\Local\Autodesk\3dsMax \201 5 - 64bit\ENU\en-US\plugcfg\vray.cfg"
This error should not be related to the issue you have.

Ok, that's good to know, thanks. It would be good to know though when this file is usually created as we clear out that folder on every reboot?

V-Ray changes the project folder on all render nodes participating in the DR render. What is the project folder set on the Main machine?

The project folder is set to the folder of the job, which is on our server, and is a UNC path - e.g.

\\SERVER\Work\Active\AAAA0000_Job

We never would have set the project folder to "C:\Users\vlado\Documents\3dsMax" - I can only assume this is some kind of default in VRay?

We will try with only one Spawner service running and see if we can still reproduce the error.

kind regards,
Tristan.

[electricart]

Hi there,

So we just tested with only one VRay Spawner service running on the nodes (Max 2014).

The problem still persists unfortunately.

[electricart]

Hi there,

Just to follow up, we have tried the following;

• Created a new scene and merged the old scene into the new one (to try to eliminate the possibility of a corrupt scene)
• Re-created the materials (to try to eliminate the possibility of a corrupt material)
• Removed the following folders to reset the Max settings on the Workstation

"%LOCALAPPDATA%\Autodesk\3dsMax"
"%DOCUMENTS%\3dsMax"

None of these measures helped and the issue persists unfortunately.

We have noticed that the problem usually occurs with either the last one or two nodes to join the distributed render.

In main test scene we are using to diagnose the problem, we have noticed the following;

The test scene has two dome lights using VRayHDRI maps. One dome is used for refraction, the other dome is used for reflections and everything else. Each dome uses a different HDRI map.

We have noticed that on the nodes that fail to render correctly, the darker buckets are consistent with a render done if the reflection dome was switched off completely.

This isn't a path or permissions issue - the same nodes sometimes render correctly, sometimes don't. It just seems like VRay is randomly rendering without one of the domes on random nodes.

[tashko.zashev]

Thank you Tristan,

[2015/May/7|14:45:08] warning: Failed to delete server scene file "C:\Users\render\AppData\Local\Autodesk\3dsMax \201 5 - 64bit\ENU\temp\vray_server_scene.max" (183)

I need to check this more carefully, because the warning appears even if the scene was deleted right after the render has been completed.

The project folder is set to the folder of the job, which is on our server, and is a UNC path - e.g.

Actually this project path ("C:\Users\vlado\Documents\3dsMax" ) is used by the dummy scene when V-Ray Spawner is running on the machine. It will not cause any rendering problems, but we will change it to something more appropriate.

However, would it be possible to send us your problematic scene file to test it in our environment? If yes, please send us an email with request for FTP access or just send us directly a download link to the same mail.
Then we will continue troubleshooting the issue via our ticket system.


In a quick test here with second dome light only for reflections, the issue was not reproduced.