Vray Beta 2 build crashing consistently on Distributed rendering on farm

V-Ray for Maya V-Ray for Maya :: Issues
1

Odd behavior with the beta:

It seems to render the frame fine but at the end during the exporter cleanup all of the render nodes suddenly switch from rendering from the host machine ip to ip 0.0.0.0 and the maya scene on the user machine gets a fatal error and shuts down.

The host machine crashes at
[2011/Apr/3|18:25:23] V-Ray: ========================
[2011/Apr/3|18:25:23] V-Ray: Clearing exporter memory…

Below is the crash report on the render.

This happens constantly on scenes with shave and a haircut as well as with a shot with just procedurals and two 2k displacement maps on a standard vray material.

Machines are all 12 core xeon’s with 24 gig of ram on windows 7.

This didn’t happen on build 14535 but seems to be the case with 2.00.01

Here is the crash on one of the slaves (sometimes they go to 0.0.0.0 and other times they crash out…)

Batch rendering works fantastically.

error.PNG.jpg
error.PNG.jpg1124×780 207 KB

2

When are we going to see Nightly builds keep rolling with bugfixes that are being addressed in 2.0?

3

I’ve finally gotten it to write a crashdump file.
Here is the error before crash.
[2011/Apr/3|21:55:11] V-Ray: Clearing exporter memory…
Writing crash dump to “C:\Users\Brandon\AppData\Local\Temp\VRayMaya.dmp”
[2011/Apr/3|21:55:12] V-Ray error: Fatal error freeing exporter memory.
[2011/Apr/3|21:55:12] V-Ray error: Fatal error rendering image.

I can send you the file if you’d like?

4

Yes, if you can send me both the scene and the dump that produced the errors to vlado@chaosgroup.com it will be very helpful!

Best regards,
Vlado

5

The beta will be updated quite often before the release (our goal was about once a week); we want to take the time until the release to organize the nightly builds a little better.

Best regards,
Vlado

6

Sure thing!
Will upload first thing in the morning!
Thanks for the quick reply!

7

Not sure if the scene helped out at all but I’m getting consistent crashing in this build 2.00.01 at the freeing memory portion of the end frame all across the farm (no distributed obviously, just a pure render) across all nodes.

Here is the output message (same type of crashing as the scene I submitted earlier last week)

I’ve tried reverting a build back to the March 28th nightly but that doesn’t seem to save things…there are no rigs in the scene and everything is a converted mipmapped tiled Exr for all textures optimized for Vray (also there is no fur or any extraneous plugins.)

Error Message
=======================================================
Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Slave Log
=======================================================
y: GI rays: 21029281
0: STDOUT: [2011/Apr/11|21:37:14] V-Ray: Reflection rays: 21562282
0: STDOUT: [2011/Apr/11|21:37:14] V-Ray: Refraction rays: 0
0: STDOUT: [2011/Apr/11|21:37:14] V-Ray: Unshaded rays: 12387
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray error: Exception : Clearing ray server (freeing resource manager)
0: INFO: Deadline is ignoring error: “Error: Exception : Clearing ray server (freeing resource manager)” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Exception : Clearing ray server (freeing resource manager)
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Number of intersectable primitives: 964385
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: SD triangles: 2772
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: MB triangles: 0
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Static primitives: 961612
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Moving primitives: 0
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Infinite primitives: 1
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray error: There was a fatal error rendering the scene.
0: STDOUT: Error: There was a fatal error rendering the scene.
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Render interrupted
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: ========================
0: STDOUT: [2011/Apr/11|21:37:16] V-Ray: Clearing exporter memory…
0: STDOUT: Writing crash dump to “C:\Users\User\AppData\Local\Temp\VRayMaya.dmp”
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.

at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

8

We could not reproduce the error right away with your scene, but there are a few more things to try with it…

BTW is there any possibility that you are using the VRAY_TEXTURE_CACHE environment variable?

Best regards,
Vlado

9

Ok nevermind, I could get it to crash on one of the machines… it’s too late for Beta 2, but it will be fixed in Beta 3. Thanks a lot for the scene!

Best regards,
Vlado

10

Any idea what may be causing the crash? I’d love to find a way to circumvent it and get the renders to push through before beta 3…

11

The crash should go away if you disable the V-Ray frame buffer and use the Maya Render View instead.

Best regards,
Vlado

12

Great! That’s the easiest fix yet.
Thanks.

13

Looks like crash is still happening, even with the Vray frame buffer disabled and updated to Beta 2.
Same symptoms it seems.

=======================================================
Error Message
=======================================================
Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Slave Log
=======================================================
ing resource manager)
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Number of intersectable primitives: 256707
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: SD triangles: 2358
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: MB triangles: 2
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Static primitives: 57938
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Moving primitives: 196408
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Infinite primitives: 1
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray error: There was a fatal error rendering the scene.
0: STDOUT: Error: There was a fatal error rendering the scene.
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Render interrupted
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: ========================
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray: Clearing exporter memory…
0: STDOUT: Writing crash dump to “C:\Users\User\AppData\Local\Temp\VRayMaya.dmp”
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray error: Fatal error freeing exporter memory.
0: INFO: Deadline is ignoring error: “Error: Fatal error freeing exporter memory.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Fatal error freeing exporter memory.
0: STDOUT: [2011/Apr/12|09:11:37] V-Ray error: Fatal error rendering image.
0: INFO: Deadline is ignoring error: “Error: Fatal error rendering image.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Fatal error rendering image.
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.

at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Error Type
=======================================================
RenderPluginException

=======================================================
Error Stack Trace
=======================================================
at Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()

14

Ok; I’m looking for a fix now; I can get this to you when it is ready so please email me to vlado@chaosgroup.com

Best regards,
Vlado

15

Emailed you!
Thank you!
I had the beginnings of a panic attack slowly rising as the frames froze…
=)

16

Well, it seems to crash after the frames are rendered and saved, so you should be able to render the thing even with the crashing, no?

Best regards,
Vlado

17

I can but it stalls out and tries to rerender the frame after reporting that the crash was an error which causes the frame to immediately start over until a non crashed version of the frame goes through.
The other issue is that sometimes the frame doesn’t complete entirely on the vrimg write and one or two of the channels (usually the multimattes) come out partially corrupted.

18

Just tried another setup..stripping all of the Shave and a Haircut data out of the scene and limiting all the textures…I was hoping that maybe the error was on the shave side of things causing the crash…looks like that’s not the case.

The only other thing I think that may be crashing the render is the light cache.
I’m going to send off a job where it is disabled, see what happens.

Does the fact that each of the render nodes are dual 12 core hyperthreaded xeons have any potential issue (the idea being that the light cache threads may screw up past 16?)

For continuity sake I’ve attached the error from the shave free attempt
=======================================================
Error Message
=======================================================
Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Slave Log
=======================================================
2011/Apr/12|10:27:28] V-Ray: SD triangles: 2772
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: MB triangles: 0
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: Static primitives: 951878
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: Moving primitives: 0
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: Infinite primitives: 1
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray error: There was a fatal error rendering the scene.
0: STDOUT: Error: There was a fatal error rendering the scene.
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: Render interrupted
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: ========================
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray: Clearing exporter memory…
0: STDOUT: Writing crash dump to “C:\Users\User\AppData\Local\Temp\VRayMaya.dmp”
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray error: Fatal error freeing exporter memory.
0: INFO: Deadline is ignoring error: “Error: Fatal error freeing exporter memory.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Fatal error freeing exporter memory.
0: STDOUT: [2011/Apr/12|10:27:28] V-Ray error: Fatal error rendering image.
0: INFO: Deadline is ignoring error: “Error: Fatal error rendering image.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Fatal error rendering image.
0: STDOUT: Scene C:\Users\User\AppData\Local\Prime Focus\Deadline\slave\jobsData\SCN_01_06_C01_NOSHAVE.mb completed.
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Deadline caught error “Error: There was a fatal error rendering the scene.”. If this error message is unavoidable but not fatal to your render, please email deadline-support@primefocusworld.com with the error message, and disable the Maya job setting Strict Error Checking.

at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Error Type
=======================================================
RenderPluginException

=======================================================
Error Stack Trace
=======================================================
at Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()

19

I THINK that its been figured out.
Metzger and I went through things piece by piece and I managed to get a consistent render out of the box.
It looks like a bug with the scene referencing.

By creating the scene from scratch and redoing all referencing things seemed to render.

I won’t really know until tomorrow when the farm parses the scene but here’s hoping that maybe it will get through.

20

Well..looks like the crashing is still occurring.