Dynamic Memory Limit, Hyperthreading and farm machines crashing....

[MartinB]

We switched some of our farm machines over to hyperthreading a while back for testing and we're getting a nice 20-30% speed increase. The only problem is the hyperthreading machines seem to crash fairly regularly (only on vray jobs), while the non hyperthreaded machines never crash.

The frequency of crashes seems to have noticeably reduced since we started using a pre-render script to set the Dynamic Memory Limit to roughly 2gig less than what the render box has in (we have a mix of 24 and 64gig machines, all machines are dual quad core xeons, so 8 real threads, 16 when hp enabled)

Now one thing I noticed today was about 10 frames of a big job didn't go through over night, 7 of the frames/machines didn't have hyperthreading on, and they were just sat there stuck at about 90%, but the machines were still up and you could ssh into them, while the 3 that had hyperthreading on had actually fallen over and needed rebooting. These were all 24gig machines, I resubmitted those frames and again they failed at 90%, so I put them on the 64gig machines and they wen't through fine.

I was just trying to understand the relationship between thread count and memory usage and is there any other settings I could be trying to get rid of the crashes. Again this only seems to happen with vray, our renderman and nuke jobs go through fine on the hyperthreaded machines

Cheers
Martin

[t.petrov]

Hi,

What happens if you start the same frames on the same machines?
Do they crash again or it is some random process?

Keep in mind that the dynamic limit is not 100% exact and it is possible to make a scene where more memory is used.
A way to optimise this is to use smaller buckets, this will minimize the number of objects simultaneously visible in the
scene and will allow (probably) more objects to be collapsed (removed from memory), of course this will increase the
render time, but this is inevitable.

[vlado]

We switched some of our farm machines over to hyperthreading a while back for testing and we're getting a nice 20-30% speed increase. The only problem is the hyperthreading machines seem to crash fairly regularly (only on vray jobs), while the non hyperthreaded machines never crash.

Which V-Ray build is this? There were some issues in some of the recent builds, including the official 2.20 builds on our site, which have been fixed in the latest nightly/stable builds.

Best regards,
Vlado

[d.velop]

Hey,

is there any solution yet?
Same thing here: render is stuck at 90% and CPU utilization is down to almost 0.
After an undefined amount of time, where nothing seems to happen, it suddenly finishes.

0: INFO: Rendering with Maya version 2012.0
0: INFO: Rendering with VRay.
0: INFO: Render Argument: -r vray -s 490 -e 490 -b 1 -rd "folder" -proj "filename.mb"
0: STDOUT: [2012/Nov/12|14:22:30] V-Ray: V-Ray for Maya version 2.20.01 from Feb 1 2012, 17:09:48
0: STDOUT: [2012/Nov/12|14:22:30] V-Ray: V-Ray core version is 2.00.01
...

0: STDOUT: [2012/Nov/12|15:37:59] V-Ray: (Estimated time remaining: 0h 12m 45.9s): 80 % completed
0: STDOUT: [2012/Nov/12|15:44:05] V-Ray: (Estimated time remaining: 0h 6m 21.1s): 90 % completed
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Number of raycasts: 3837790745
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Camera rays: 34422232
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Shadow rays: 56660616
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: GI rays: 34360161
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Reflection rays: 3294633664
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Refraction rays: 3070117460
0: STDOUT: [2012/Nov/12|17:19:24] V-Ray: Unshaded rays: 344650406
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: Number of intersectable primitives: 12146133
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: SD triangles: 0
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: MB triangles: 0
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: Static primitives: 11990480
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: Moving primitives: 155653
0: STDOUT: [2012/Nov/12|17:19:28] V-Ray: Infinite primitives: 0
0: STDOUT: [2012/Nov/12|17:19:33] V-Ray: Successfully written image file

The rendernodes do have 24 or 32GB of Ram and 6Core (12threads) or 12Core (24 threads).
Here is a link to the full logfile: Deadline_Log.rtf.zip

Hope someone can help me out.
Thanks.

[vlado]

This seems a different issue; from what I see the image successfully completes (it's not really stuck at 90%, we just don't print the 100% mark) and the output file is written. Have you watched memory usage during that time? It is possible that V-Ray tries to clean up the memory used during rendering. Especially if you use lots of subdivision surfaces or tiled OpenEXR textures, this may take a (longish) while. We've done some improvements in the nightly builds, but I see that you are still using the official ones.

Best regards,
Vlado

[Dariusz Makowski (Dadal)]

If its cleaning maybe a time kill limit after 10 mins to kill process and move on if it saved image?

[d.velop]

I watched the CPU and Memory Monitor around those 90% I talked about earlier for quite a while (not the whole time tho) - it was constant around 8 of 32GB.
Where can I find these nightly builds?

And no, i cannot kill the process, because the file is written afterwards.

[vlado]

You can get the nightly builds if you email us to vraymaya@chaosgroup.com

Best regards,
Vlado

[d.velop]

Thanks. I tried the most current builds right now (V-Ray for Maya version 2.25.01, revision 22495 from Nov 14 2012).
I can't see any difference at all.

From what I saw is that the memory usage is absolute constant over time (untill the render really finishes).
Over time just one CPU core after another finishes.

Should I take that as granted and ignore it or is there anything else I could try?

[paulDryzen]

Your bucket size is too big.

[vlado]

Should I take that as granted and ignore it or is there anything else I could try?

Well, you can get us the scene to vraymaya@chaosgroup.com - we can look into it and see what's going on and maybe suggest workarounds, or see if there is a way to improve the code.

Best regards,
Vlado