Possible problem with XREF scenes and distributed rendering

[tricky]

Are there any known issues with distributed rendering when the scene has XREF scenes within it?

Without the XREF scene (which has lots of models of chairs etc for a furniture showroom), all our renderfarm nodes kick in and render fine.

If we add the XREF (around 150MB compressed), all the nodes seem to lock up and their RAM usage for the max session locks at around 1.4GB, zero CPU usage.

The local workstation renders fine.

When we cancel the job, all the DR machines are still locked up and the RAM for the 3dsmax.exe task isn't flushed. They need to be manually restarted to work again.

WTF?

[tricky]

Any word on this? Do XREF scenes work on Max2016 with VRay 3.30.03 DR?
On our system, certainly with this scene, the DR nodes lock up until forced to restart.

[NicoC]

Hello Richard,

We regularly use XRef scenes with BB and DR, without problem.
However, have you tried not to use scene compression ? I don't know if this could be a reason for your problem, but I guess it's worth trying

[crazy homeless guy]

Not to high jack the thread but if we have Xref's set to anything besides fully visible they don't render. This is a relatively new error but I don't know if it is Max or Vray causing it since we upgraded to Max2016 and Vray3.3 at the same time.

[tashko.zashev]

Any word on this? Do XREF scenes work on Max2016 with VRay 3.30.03 DR?
On our system, certainly with this scene, the DR nodes lock up until forced to restart.

The issue should be related to this specific scene or setup because X-ref scenes are rendered fine in DR mode in V-Ray 3.3.
Is there any chance to try your Xref scene here ?
Have you checked the V-Ray log on one of the salves machines ? Is there any useful information - error or warning ?

Not to high jack the thread but if we have Xref's set to anything besides fully visible they don't render. This is a relatively new error but I don't know if it is Max or Vray causing it since we upgraded to Max2016 and Vray3.3 at the same time.

Just did a test with X-Ref scene with both "Visible" and "Box" Display Options ticked and that worked for me. Could you please provide more details or step by step instructions on how to replicate this behavior here?

[auke]

Hello, we have been running into the same of at least similar problem. All nodes render fine when Xrefs are turned off. But when loading an Xref scene the nodes will go into a perpetual crash loop trying to load the scene, crashing and then trying again. This is an excerpt out of one of the slave's log files:

2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] ================================================== ==========
2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] Console created, V-Ray A for x64 from Mar 23 2016, 03:28:56
2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] ================================================== ==========
2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] Compiled with Intel C++ compiler, version 14
2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] Host is 3ds Max, version 18
2016/04/14 07:10:33 INF: [12144] [13580] [V-Ray] V-Ray DLL version is 3.25.01
2016/04/14 07:10:48 INF: [12144] [13580] [V-Ray]
2016/04/14 07:10:48 INF: [12144] [13580] [V-Ray] Starting DR session from 192.168.0.39
2016/04/14 07:10:48 INF: [12144] [13580] [V-Ray] Receiving DR scene from 192.168.0.39
2016/04/14 07:10:51 DBG: [12144] [13580] Interface8::LoadFromFile(C:\Users\--xxx--\AppData\Local\Autodesk\3dsMax\2016 - 64bit\ENU\temp\vray_server_scene.max)
2016/04/14 07:10:51 INF: [12144] [13580] Starting to load file: C:\Users\--xxx--\AppData\Local\Autodesk\3dsMax\2016 - 64bit\ENU\temp\vray_server_scene.max
2016/04/14 07:10:51 WRN: [12144] [13580] MAXScript Callback script Exception: -- Runtime error: No method found which matched argument list
2016/04/14 07:14:32 INF: [12144] [13580] Done loading file: C:\Users\--xxx--\AppData\Local\Autodesk\3dsMax\2016 - 64bit\ENU\temp\vray_server_scene.max
2016/04/14 07:14:32 INF: [12144] [13580] SYSTEM: Production renderer is changed to V-Ray Adv 3.30.05. Previous messages are cleared.
2016/04/14 07:14:35 INF: [12144] [13580] [V-Ray] Tiled bitmap cache size set to 1000 MB
2016/04/14 07:14:35 INF: [12144] [13580] [V-Ray] Preparing renderer...
2016/04/14 07:14:35 INF: [12144] [13580] [V-Ray] Using Intel Embree ray server
2016/04/14 07:14:35 INF: [12144] [13580] [V-Ray] Using full light evaluation
2016/04/14 07:14:35 WRN: [12144] [13580] [V-Ray] Light cache is used for glossy rays; it is recommended to turn on light cache retracing.
2016/04/14 07:14:35 WRN: [12144] [13580] [V-Ray] Subpixel color mapping is on: rendered result may have incorrect brightness.
2016/04/14 07:14:35 ERR: [12144] [14588] An unexpected exception has occurred in the network renderer and it is terminating.
2016/04/14 07:14:35 ERR: [12144] [13580] [V-Ray] UNHANDLED EXCEPTION: Scanning scene Last marker is at ./src/vrenderlight.cpp, line 494: Exiting VRenderLightObj::init()
2016/04/14 07:15:19 DBG: [13532] [21948] Starting network
2016/04/14 07:15:19 DBG: [13532] [21948] Calling NetRenderPreLoad
2016/04/14 07:15:19 DBG: [13532] [21948] in NetWorkerPreLoad. jobFile: ; jobname: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max
2016/04/14 07:15:19 DBG: [13532] [21948] in Init. jobFile: ; jobname: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max
2016/04/14 07:15:19 DBG: [13532] [21948] in Init. calling nrGetIface. jobToSend: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max
2016/04/14 07:15:20 DBG: [13532] [21948] in NetWorkerPreLoad. curJobname: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max; init: 0
2016/04/14 07:15:20 DBG: [13532] [21948] in NetWorkerPreLoad. calling PostInitMessageSystem()
2016/04/14 07:15:20 DBG: [13532] [21948] in NetWorkerPreLoad. srv_pid: 13960
2016/04/14 07:15:20 DBG: [13532] [21948] leaving NetWorkerPreLoad. LoadLib()
2016/04/14 07:15:20 DBG: [13532] [21948] NetRenderPreLoad passed
2016/04/14 07:15:20 INF: [13532] [21948] [V-Ray] Using folder "C:\Users\--xxx--\AppData\Local\Temp\vray_assets_cache" for caching assets for DR.
2016/04/14 07:15:20 INF: [13532] [21948] [V-Ray] Using file "C:\Users\--xxx--\AppData\Local\Temp\vray_assets_cache/cache_info_20204.txt" for storing assets cache information.
2016/04/14 07:15:23 INF: [13532] [21948] SYSTEM: Production renderer is changed to NVIDIA mental ray. Previous messages are cleared.
2016/04/14 07:15:24 DBG: [13532] [21948] Interface8::LoadFromFile(C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max)
2016/04/14 07:15:24 INF: [13532] [21948] Starting to load file: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max
2016/04/14 07:15:24 WRN: [13532] [21948] MAXScript Callback script Exception: -- Runtime error: No method found which matched argument list
2016/04/14 07:15:24 INF: [13532] [21948] Done loading file: C:\Users\--xxx--\AppData\Local\Temp\vraydummy2016.max
2016/04/14 07:15:24 INF: [13532] [21948] SYSTEM: Production renderer is changed to Default Scanline Renderer. Previous messages are cleared.
2016/04/14 07:15:24 INF: [13532] [21948] Starting network rendering
2016/04/14 07:15:24 INF: [13532] [21948] Max is ready to begin render

This piece of logging repeats itself for as long as the host keeps sending this job to the slaves. We are trying to pin down the problem internally as well, will update this post when we find useful information.

[tashko.zashev]

Hi,

It would be great if you can send us any scene file that reproduces the issue so we can troubleshoot it here too.
We can give you access to our FTP if need, just send us request to suppor@chaosgroup.com

[auke]

I'm afraid I can't share it right now due to the scene's sensitive nature, it would to much time to clean it up far enough at the moment unfortunatly. Will try to find some time for that later on.

After testing some things though we were suprised the issue was resolved by restarting 3ds Max completely, clearing the Xref out of the list and then re-attaching it. This scene is now identical to the one which had been crashing nodes last night. In case this only happens when rendering multiple frames we will run into it later again. Will try to keep you updated.

[Svetlozar Draganov]

Glad to hear you've found a way to resolve the issue.
Feel free to contact us again when you have time to clean the sensitive information from the scene and the crash still occurs.

[tricky]

We will try and find the file that was causing the problem. Clearly there are some issues somewhere as it doesn't seem to be just us. Unfortunately, the scenes are massive with cars, plants, furniture etc so it will take a while to archive.