Distributed rendering woes

[arobbert]

I am having a problem with DR. when I cancel a job I wait a min. before I resend the job it then crashes the workstation submitting the job. This is driving me crazy. How do you tell if a node is still working an a previously submitted job? Has enyone found a reliable consistant way to monitor, kill and restart jobs?

[SnipeyX]

I've always just found it best to not kill the renders, let it complete the frame and everything seems to play nicer.

However, I only use DR occaisionally, so it's not really an issue for me.

[vlado]

Re: Distributed rendering woes

I am having a problem with DR. when I cancel a job I wait a min. before I resend the job it then crashes the workstation submitting the job. This is driving me crazy. How do you tell if a node is still working an a previously submitted job? Has enyone found a reliable consistant way to monitor, kill and restart jobs?

Which 3dsmax/V-Ray version is that?

Best regards,
Vlado

[arobbert]

viz 2007 1.5 RC3.

[jacksc02]

your workstation might be running out of ram. If it does the first render it will load a lot of textures to ram and it wont clear the ram at renders end, So when you go to render again it wont have enough ram to keep viz going so it will dissapear

[arobbert]

It appears that some nodes are pusing at the pre-filtering light cache stage. If I cancel a rendering I usually wait a min. before submitting a new job. Some nodes seem to hanging at the pre-filtering stage for a long period of time. If I submit a new job while nodes are in this state it crashes the workstation. I often have to re-start spawner on all the nodes to get things up and running again which is a pain.

I wish there was a way to know if a node is still processing after a render has been canceled. It would be great if there was a tool that would help with monitoring and controling nodes. I tried the DR utility thing posted a while ago but could not get it to work.

I'm also trying to figure out why there is such a long pause at the pre-filtering light cache stage.

[jujubee]

I agree with Jackson - if you have less than 2 gigs of RAM it's probably trying to unload a heavy scene's memory before starting back up again causing you station to crash.

[arobbert]

These are not very complex scenes. All my nodes have at least two gigs of ram and the workstations have 3.

I notice that the nodes that are hanging are not showing any activity in the task manager and the bigger the Irradiance map the longer they hang. If they are this state and I cancel a rendering VIZ continues to hang until the nodes come out of this state. Which can take 5 min or more. During this time the workstation is locked up.

I suspect that this has something to do with a network time out setting or something and that my server is not delivering the maps fast enough.

It seems that a node should load the map when it becomes available instead of stalling. If I use a local path for the Irradiance map the problem goes away.

[3ddesign]

I've made it my practice to always have a Remote Desktop window open for each of my render nodes. Task Manager is open in each window. If I ever have to cancel a render, I always end the 3dsmax process at each node. I wait a minute and re-submit. I've never had any problems with DR after doing this.

Craig

[vlado]

Somewhat off the topic, but on 64-bit windows have you had a problem where a 3dsmax process will remain running and you can't kill it even from the task manager? I've had this a few times and it's very irritating, there is nothing to do except reboot (this is 3dsmax 8 ).

Best regards,
Vlado

[zeronove]

Following Vlado off topic,
I can say that on xp 64, Viz 2005 Vray 1.5RC3 sometimes (no pattern), the main application hangs at the end of render after saving the image; this happens only if this is done throught backburner (no DR); so the render node remains busy and not available for the next job; not so useful in night time render queue

a.