Node CPU levels after Rhino Crash

[Travis Serio_cg]

Nodes using High CPU after Rhino crash.

I've noticed that when using swarm, if Rhino crashes due to V-Ray it will often leave the nodes with high cpu level's and requires a node computer restart to bring them back to normal levels. Is there a way that I can force the nodes to stop whatever they are doing with out restarting the network?

[frip51]

Hi Travis,
have you tried to disable (and re-enable) the respective nodes through Swarm UI on the host computer?

[Travis Serio_cg]

Yeah, unfortunately they still appear stuck. I almost need some network command line utility that says "Errrbody stop!" =)

I noticed 2 days ago that I had at least 3 nodes that had been sitting at 100% CPU usage for at least 3 days.... i was like "that's gotta be good for them".

[frip51]

A suitable command line routine for a remote reboot would be [shutdown /m \\computername /r /f], whereas computername needs to be changed to the respective computer name (or its IP address), obviously.

[Travis Serio_cg]

yes that would be ideal. I've also noticed that during interactive renders using swarm that the nodes will often go "Unreachable" if you make some material change / options change / geometry change. I haven't tracked it down to an exact thing yet, but it seems that interactive rendering using swarm is still a pretty fragile system. But anyway, you have to stop the render, sit and watch the local host swarm url and wait for them all to reset and come back, then start the render again.

I should note that both of these issues seem to be related to the interactive rendering only, and if you do a normal dr render everything works as intended with the swarm, at least for me anyway.

[Peter.Chaushev]

Nodes using High CPU after Rhino crash.

I've noticed that when using swarm, if Rhino crashes due to V-Ray it will often leave the nodes with high cpu level's and requires a node computer restart to bring them back to normal levels. Is there a way that I can force the nodes to stop whatever they are doing with out restarting the network?

Do you know under what circumstances Rhino crashes while you're using V-Ray?
If you are able to reproduce the render node abnormal cpu usage, please share (either by email or PM) the machine's swarm log?

Kind regards,
Peter

[Travis Serio_cg]

One of the areas where i've been able to force it to crash is setting the render output size to match the viewport. Then enable interactive rendering with Swarm enabled and render to the vfb (not the rhino viewport). Now while its rendering, resize the Rhino viewport and then press the button to update the viewport resolution in the V-Ray UI to try to get the VFB to update its dimensions while rendering (which i hate having to press that button btw). Once you press the button it will often cause Rhino to freeze and then a crash follows. This is fairly repeatable on my machine.

Crashes such as this can occasionally leave my nodes in some weird state where they show a high cpu load coming from v-ray on that machine. I've found a couple of nodes to hang at 100% cpu load for several days! I was wondering why it was warmer than it should have been in that room!

I just installed some Amazon Echo enabled power strips so I can just say "Alexa - Restart Render Box 18" which dumps and restarts power to the box. lol yeah... im that lazy.

I will fish out the swarm log. Whats the log's file path? And where should I email it?

[frip51]

Hahahaaa, that Alexa use-case is hilarious!

Heavy blocks (or even heavy scenes in general) might cause Rhino to stall during a render - I presume it has to do with data packets being sent into the blue after the host lags. Seems to happen when compiling prepasses, sometimes.
Might be worth trying to make the most powerful machine the host (unlike in previous Vray versions where the opposite was necessary).

Do you have GPU acceleration activated?
Or would you use cap CPU capability in Swarm?

Playing with these options has helped to solve stability issues on our end. It might slow down the process but yield results

[Travis Serio_cg]

It's all CPU and I cap the CPU on the host.

[Peter.Chaushev]

Thank you for your reply!

(...) I will fish out the swarm log. Whats the log's file path? And where should I email it?

Whenever you reproduce the problem, please grab the logs and email them at support@chaosgroup.com (also add a link to this forum thread).
The V-Ray Swarm log of the machine used to initiate the render job is located in %temp%
And the V-Ray Swarm log of the render node machine can be found at:
C:\Windows\System32\config\systemprofile\AppData\R oaming\Chaos Group\vray-swarm\work\vray-swarm

Kind regards,
Peter