Ship on Ocean Sim Restore After Max Shutdown / Start

[Svetlin.Nikolov]

Hey,

Here is more info on sim licenses and how to get started: https://docs.chaosgroup.com/display/...ation+Licenses

I got the scene from support and hope I'll be able to check the issue soon. Thanks very much for helping catch this issue!

[Ivaylo Katev]

well, you have a point, really you will be not able to make experiments during the entire simulation.
but really separate simulation license would be a better solution.

[rporter8555]

Thanks Svetlin!
I am looking forward to seeing your fix, and will try the sim license setup this weekend.
Ron

[rporter8555]

Hey!
Running the sim on my render slave is working like a charm!
This is great, allowing me to continue working on my model at the same time.
Thank you!
Ron

[Svetlin.Nikolov]

Sweet! If you find any issues with it, please do bump us. We have to refine and extend the user interface for the next versions.

Cheers!

[rporter8555]

Hi Svetlin,

Progress report is not so good. Maybe you have some suggestions?

The sim appears to run on slave machine #1, at least for a while. The most aur files I have generated is around 550 before crashing. Several attempts have aborted earlier, around 230 frames. The error message from backburner is ping task timeout.
At first I tried the full size ocean sim of about 50million cells, with multiple crashes. Then tried a smaller version around 7million, and it crashed after more completed aur files. Now I have moved to slave machine #2 with backburner manager on #1 to see if it will run any better.

I am thinking this is probably my issue with the machines having only 16GB memory.

Below are the last two cycles from the log. Notice the last entry aborts before normal:



Sim cycle 1763 box:533 x 501 x 69 --------------------- cycle start---------------
system Liquid particles: 7120050
system WetMap particles: 225612
SceneInteraction 1 0.000001
SceneInteraction 2 0.000123
Enter transferCtrlToMainThread: 2
transferCtrlToMainThread simEventType set: 2
etNewCycle
onnc 0 0.000000
onnc 1 0.000167
onnc 2 0.000277
onnc 3 0.000182
setSolverParamsResimAdaptGrid
setSolverParamsCommon
setSolverParamsLiquid
onnc 4 0.001701
begin proc nodes 0.000171
proc node Scene Root
proc node Dummy Travel
proc node Dummy Hull
proc node DummyProxy
proc node HullProxy
proc node Emitter
proc node Emitter-Neg
proc node Emitter001
proc node Emitter-Neg001
proc node PhysCamera004.Target
proc node Group001
proc node Fin Stbd
proc node Fins Port
proc node Rudder Port
proc node Rudder Stbd
proc node Screw Port
proc node Screw Stbd
proc node Shaft Port
proc node Shaft Stbd
proc node Shaft Support Aft Port
proc node Shaft Support Aft Stbd
proc node Shaft Support Fwd Port
proc node Shaft Support Fwd Stbd
proc node PhysCamera004
proc node PhysCamera002.Target
proc node PhysCamera002
proc node PhysCamera003
proc node PhysCamera001
proc node PhysCamera001.Target
proc node PhysCamera005
proc node VRaySun001
proc node VRaySun001.Target
proc node VRayLight001
proc node VRayAmbientLight001
proc node PG [Foam] of system [PhoenixFDLiquid001]
proc node PHXFoam Shader
proc node PG [Splashes] of system [PhoenixFDLiquid001]
proc node PHXSplashes Shader
proc node PHXSource-Pos
proc node PHXSource-Neg
proc node PHXWet Shader
proc node PG [Liquid] of system [PhoenixFDLiquid001]
proc node PG [Mist] of system [PhoenixFDLiquid001]
proc node PG [WetMap] of system [PhoenixFDLiquid001]
end proc nodes 0.010094
PBCore: update body HullProxy code:1 mode:1
oldbas:528.640503 bas:528.640137 t:0.006450
push particles t:0.524975
scan body t:0.116923
update srf t:0.048267
end update bodies 0.697291
onNewCycle() end 0.000230
End transferCtrlToMainThread: 2
SceneInteraction 3 0.719158
SceneInteraction 4 0.000272
SceneInteraction 5 0.104403
s(1),s(2),s(3),s(4),s(5),s(6),s(7),s(,s(9),dynamics Liquid
dynamics WetMap
s(10),s(11),s(12),s(14),s(15),
ph:00 t:0.000000/0.000000
ph:01 t:1317.224901/0.828697
ph:02 t:84.784174/0.049370
ph:03 t:1.195581/0.000356
ph:04 t:84.296344/0.048607
ph:05 t:56.305845/0.034879
ph:06 t:402.030718/0.272054
ph:07 t:0.597804/0.000324
ph:08 t:17394.126292/12.096591
ph:09 t:7311.952851/5.060318
ph:10 t:1.146810/0.000740
ph:11 t:1949.011501/0.000193
ph:12 t:7353.021491/5.709796
ph:13 t:0.000000/0.000000
ph:14 t:0.694889/0.000216
ph:15 t:42.790332/0.024980
ph:16 t:0.000000/0.000000
ph:17 t:0.000000/0.000000
ph:18 t:0.000000/0.000000
ph:19 t:0.000000/0.000000
------------------------------------------------------------------------


Sim cycle 1764 box:533 x 501 x 69 --------------------- cycle start---------------
system Liquid particles: 7118910
system WetMap particles: 225950
SceneInteraction 1 0.000001
SceneInteraction 2 0.000144
Enter transferCtrlToMainThread: 2
transferCtrlToMainThread simEventType set: 2
etNewCycle
onnc 0 0.000000
onnc 1 0.000169
onnc 2 0.000270
onnc 3 0.000226
setSolverParamsResimAdaptGrid
setSolverParamsCommon
setSolverParamsLiquid
onnc 4 0.001926
begin proc nodes 0.000195
proc node Scene Root
proc node Dummy Travel
proc node Dummy Hull
proc node DummyProxy
proc node HullProxy
proc node Emitter
proc node Emitter-Neg
proc node Emitter001
proc node Emitter-Neg001
proc node PhysCamera004.Target
proc node Group001
proc node Fin Stbd
proc node Fins Port
proc node Rudder Port
proc node Rudder Stbd
proc node Screw Port
proc node Screw Stbd
proc node Shaft Port
proc node Shaft Stbd
proc node Shaft Support Aft Port
proc node Shaft Support Aft Stbd
proc node Shaft Support Fwd Port
proc node Shaft Support Fwd Stbd
proc node PhysCamera004
proc node PhysCamera002.Target
proc node PhysCamera002
proc node PhysCamera003
proc node PhysCamera001
proc node PhysCamera001.Target
proc node PhysCamera005
proc node VRaySun001
proc node VRaySun001.Target
proc node VRayLight001
proc node VRayAmbientLight001
proc node PG [Foam] of system [PhoenixFDLiquid001]
proc node PHXFoam Shader
proc node PG [Splashes] of system [PhoenixFDLiquid001]
proc node PHXSplashes Shader
proc node PHXSource-Pos
proc node PHXSource-Neg
proc node PHXWet Shader
proc node PG [Liquid] of system [PhoenixFDLiquid001]
proc node PG [Mist] of system [PhoenixFDLiquid001]
proc node PG [WetMap] of system [PhoenixFDLiquid001]
end proc nodes 0.009810
PBCore: update body HullProxy code:1 mode:1
oldbas:528.640137 bas:528.639832 t:0.006778
push particles t:0.525848
scan body t:0.117216
update srf t:0.048162
end update bodies 0.698754
onNewCycle() end 0.000250
End transferCtrlToMainThread: 2
SceneInteraction 3 0.721818
SceneInteraction 4 0.000306
SceneInteraction 5 0.113976
s(1),s(2),s(3),s(4),s(5),s(6),s(7),s(,s(9),dynamics Liquid
dynamics WetMap
s(10),

[Svetlin.Nikolov]

Yup, this looks like out-of-memory when saving the cache file - there is a memory spike there and the log in 3.0 still did not have good memory reporting. Try a machine with 32 and this week I hope we'll be able to open the nightlies and you would be able to try them out.

[rporter8555]

Hi Svetlin,
Upgraded slave machine with 32 GB yet crash continues with same result as previously reported.

EDIT: I went into the Backburner Monitor and edited the job settings: changed "Wait to render: from 600 to 60000. The default 600 is not enough for a large or long duration simulation. My ocean is both.
Since backburner is treating this entire sim project as a single line item frame for rendering, and is not getting reports of success from each frame in the simulation, it is timing out at the default 600 minutes or ten hours. It then flags the server as failed, waits for cooldown period, then restarts based on the manager rules on number of retries. If retries is more than zero, the server starts again from frame 0, which resets and starts the entire simulation from beginning.

Hope this helps,
Ron Porter

[Svetlin.Nikolov]

Ah, thanks very much for the info! Will have a look if it's okay to override the duration.

You can now request a 3ds Max nightly access from Support. Btw, if you are using particle-based smoothing, you might want to try out next Tuesday's nightly as well - just fixed a crash there that could affect liquid and especially ocean simulation.

Cheers!

[rporter8555]

Cool. Thanks!
Ron

[rporter8555]

Svetlin,

Another aspect of this for your consideration:

I ran a small grid sim of 1000 frames which was easily completed within the default 10 hour limit. Backburner Manager was set for 3 Retries of failed servers. The rendering settings in Max called for 50 frames since that was my last setup for workstation rendering test. Backburner sets up a render job for 50 frames, but uses only the first frame for sim.

The sim actually completed successfully generating 1000 AUR files in the cache, but since manager is not receiving a completion message from the server for the single frame job, it assumed failure after the default 10 hours and relaunched the sim from the beginning, overwriting the previously completed AUR files. I am guessing that if I used the cache with the first hundred files newer than those following, this would probably introduce an unexpected artifact in the fluid at that transition.

Better perhaps to turn off manager retries to avoid this, or is it possible that the sim can report completion to the manager, in which case the number of frames in the manager job should be set to one?

Ron