Backburner sim feedback (or lack of) causing timeouts

[elyptic]

I read the thread here about the Backburner machine stuck on 'Transforming vertices" and thought I had the same problem. However after checking the output cache folder, I saw it was actually running the sim and saving cache files properly. It takes about 5 minutes for a frame to complete, which is fine.

However, this morning I see it stopped around 9:35am although the 3dsmax instance is still pinning the CPU and memory is actively being consumed (both up and down). It's 11:40am now, so it's been sitting there executing nothing for about 2 hours. It's not on a 15-frame bump either, it's just stopped on 234 (of 900).

The biggest problem is the lack of feedback whatsoever. It still says 'Transforming vertices' frame 0 of 0 in the dialog panel (even though it successfully simulated 234 frames). Would it make more sense to write a batch file or something to execute at the command line rather than using Backburner? There is zero feedback to indicate what is going on or why it's stopped.

Update: The only hint I have at what went wrong is in my Backburner Monitor which says "3dsmax adapter error: Ping task timeout". Backburner still thinks it's on frame 0! I checked the Job Settings and the timeouts are 'Wait to load': 20, 'Wait to unload': 10, 'Wait to render': 600. I have a feeling this 600 minutes (10 hours) is what is killing it early. Of course if the script sent feedback to Backburner it would probably be fine, but it thinks frame 0 is taking more than 10 hours to render.

So how would I resume the job where max was when the ping timeout hit? I've updated the render timeout to 2880 (48 hours) which I think should finish out the job. Edit: Well I tried suspending and resuming the job ... and now we are resimulating everything from frame 0 ugh

[Svetlin.Nikolov]

Hey,

Which version of Phoenix do you use? If you aren't already on a very recent nightly, please check it.

The problem with Backburner is that it was never meant for anything other than rendering and it's quite not-customizable. The Phoenix simulation is executed as a pre-render task and there is no way I could find to connect to the progress in the Monitor and display the actual % finished. Also, as you've already found out, by default Backburner times out after 600 minutes, and since the pre-render task that Phoenix runs is not considered by Backburner a real job that would update the progress, you cannot have a simulation of more than 10 hours. In the more recent nightlies I added an option to submit the job suspended and this way you could manually edit the job and increase the timeout, which for some reason Backburner does not allow us to do from MAXScript. Also, since the Phoenix job is a pre-render task, this is also the reason why you can't pause and resume the job the way things are currently. I have to figure a way around that, thanks for reporting it!

An alternative would be to use Deadline and I have to update the submitter with the customization options that are available for Backburner right now.

[elyptic]

Well I don't want to pay for a network rendering solution when Backburner is doing most things properly.

We can set timeouts in maxscript with -waitRender:<integer> (The amount of time to wait for 3ds Max to render, in minutes. Default=600.)

This is working so far, it began (February 21, 2017, 12:04:39 PM) and still going at (February 22, 2017, 7:16pm). I edited the Job Settings and set the timeout to 2880 in Backburner. I'm using a nightly from maybe 2 nights ago.

So at this rate, I'm at frame 522 of 900 and I'm a day and a quarter in. The obvious best thing to do would be to set it to something ridiculous like 30 days timeout ... but in the meantime what's the best way to resume a partially done simulation via Backburner? If I set the sim start and end times to X and 900, will it pick up the simulation from where I left it?

[Svetlin.Nikolov]

To restore a simulation from where you left off, you need to alter the script just a tiny bit. Please check the forums - we already spoke about this yesterday in another thread Look for A_StartSim. I hope I'll have time to add this properly to the user interface soon.

Cheers!

[Svetlin.Nikolov]

Here you go: http://forums.chaosgroup.com/showthr...310#post724310

[Svetlin.Nikolov]

We can set timeouts in maxscript with -waitRender:<integer> (The amount of time to wait for 3ds Max to render, in minutes. Default=600.)

Yup, this is a parameter to calling cmdjob, but if we did it this way, you would not have automatic copying of the scene to sim machines - we're using the MAXscript Backburner API where this is not exposed. You can check how it's done actually - the entire implementation is in the PhoenixFDMenu.ms.

[elyptic]

OK thanks for the insight. My current sim is still running from two days ago, less than two hours left and still almost 200 frames of sim to go. Towards the end I'll edit the Job Settings and see if I can add another 1440 minutes to the timeout. If it doesn't work (I don't expect it to), I'll try to learn about resuming this sim.

I found a thread about some use of the http://forums.cgsociety.org/showthre...0&page=1&pp=15 here that I might dabble with after looking at the Phoenix scripts.

[elyptic]

After editing the job settings in progress, the backburner job restarted itself so I had to kill it. Then I went into the PhoenixFDMenu.ms and added the recommended changes:

When I filled in "" 720 in the line

simulateCommand = .... try( A_StartSim phoenixNode "" 720; ...

we just got an error, I thought because the double unescaped quotes. So I escaped them like so:

simulateCommand = .... try( A_StartSim phoenixNode \"\" 720; ...

saved max, closed it, reopened it, loaded my scene and went frame 723 (the next frame in the sim which wasn't saved). Set the Start Frame in the GUI to 723 and Stop Frame at 900. Set the timeline to frame 723 and submit to the edited script.

'Run only the Simulation' and 'Submit the Job Suspended' are checked. clik Run and Activate the job in Manager.

The job runs for a moment, then quits saying 'Complete' in Manager and no files are written anywhere. Backburner server says 'Received End of Job command'.

[Svetlin.Nikolov]

Alright, hang in there! Will send you the modified script in a few hours

The only thing you should check is if you have backup frames in the simulated caches - they are enabled from the Output rollout and you can also see if the currently loaded cache is a backup frame in the info box of the Simulation rollout.

[Svetlin.Nikolov]

Set the Start Frame in the GUI to 723 and Stop Frame at 900. Set the timeline to frame 723 and submit to the edited script.

Ah, you don't need to set the start and end frame - this is very probably why it's not working - the last backup frame is outside the start-end interval and so the sim can't start. If you've set the frame in the script, this is all that's needed.

[Svetlin.Nikolov]

And here is the updated submitter - it will also be in tomorrow's nightly PhoenixFDMenu.zip If you have time, please check if this works for you.

Cheers!

[elyptic]

Nice, I'll check it out once this render batch is over. I see in the job XML.details file on the backburner end there's a node called <MaxRender> inside the MaxJob node, maybe we can access the render timeout in the job submission part?