Resimulation Questions

[joshclos]

1.) Is it possible to restore a resimulation?
2.) Does phoenix have limits on grid res or cache file sizes?

I had a sim last night that was about 49 million voxels and I resimed it to amp the resolution by 1.0, which then reached about 392million. As a test, I tried to resim that same 49million base sim with an Amp of 2.0. Once the sim's adaptive resolution began to grow, the cache file sizes were approaching 3gb (which is fine because it had velocity and was probably enormous) but at a certain point the sim just stopped. Deadline didn't report any errors and said it was still simming but previously it was simming at about 10 mins per frame and then just halted for 4 hours. I tested the same sim 5 times simultaneously with slightly different amp settings and they all halted and their last files written were around 2.8 or 2.9gb and the frame they halted on were different. I'll try another sim without velocity since I'm sure the files will be 1/10th the size they were before In case it's relevant, the computers were dual xeon with 128gb of ram.

Thanks!

[Svetlin.Nikolov]

Hey,

There shouldn't be any limits, unless there is either a bug, or an OS or hardware limitation.

Can you send over the Phoenix log file right after the problem occurred? You can find it from the Simulation rollout -> Preferences...

PS: Sorry, I missed the first question - restoring a resim should be possible, unless something's wrong, and then we gotta fix it.

Cheers!

[joshclos]

Another round of tests revealed that there may be a network file size limit of 4gb. Last night's sims (with velocity channel removed) got much further in the sim and wrote files up to 3.9gb and then halted again so I'm assuming the next file would have exceeded 4gb and it isn't allowing it to write the file. Perhaps the results would be different if it were a local sim. I don't actually need the resolution to go that high for these particular sims so it isn't a problem - I was just curious.

Unfortunately I can't send the log file because I'm working with a studio that doesn't allow us to take files off the computers due to NDAs and such. The sim computers don't even have internet connectivity. Perhaps I'll try the test again at home to find out more and if I do, I'll gladly send the log file if the problem occurs again

Thanks for your help!

[Svetlin.Nikolov]

Ouch. Nice pointer though! Maybe it is indeed the network, because just recently I was debugging caches over 6 gigs, so it could be either the network, or the cache content - the 6 gigs were mainly particles since it was a liquid sim, and these are compressed a bit differently...

[DaniSan]

Hello,

Sorry to crash the thread, but I'm actually having this same issues when simulating a fountain simulation. I've tried this over the network with the deadline submission and locally on my own machine and both methods seem to stop after a certain time frame. I have a total of 263 frames and it seems to just stop after it reaches the 236th frame. Each frame was simulating at about 6 minutes but now hours have passed since the last frame simulated and I still don't have a new simulation file. The phoenix simulation doesn't crash or throw out an error over deadline, but it seems like it stopped simulating.

Any thoughts?

Thanks,
Dani

[joshclos]

Does the file size grow over time for your caches? And if so, what is the size of the most recent file? In my case, the limitation was with Windows, which has some 4gb file size limitations regarding network files. Since you tried simming locally as well, I'd be curious to see if the file size is still an issue there.

[DaniSan]

The file size grows slightly, but overall it stays pretty constant. My cache files are anywhere from 80 MB to 105 MB each because the water pressure is animated. I also noticed that after a certain frame the particle start to go crazy and fly all over the place. Shortly after this happens the simulation stops.

Here is an image showing what happens with my particles. https://www.dropbox.com/s/4lp9cmhk79...icles.png?dl=0

[joshclos]

Since your files are small, you don't seem to have the same issue that I was having but your simulation hanging very likely has to do with the particles going crazy and causing long (or stalled) calculations. Have you found out what was causing them to "go crazy?" And are you using an adaptive grid? If so, the particles flying all over may cause the grid to become really large?

[Svetlin.Nikolov]

In this case you should definitely send the scene over to support. There should not be such situations happening.

Thanks!

[DaniSan]

I'm not using an adaptive grid. Therefore, I'm not really sure what's causing this issue; however, I just sent the file over for review.

Thanks for all the help!

[tashko.zashev]

I'm not using an adaptive grid. Therefore, I'm not really sure what's causing this issue; however, I just sent the file over for review.

Thanks for all the help!

Thanks for the uploading the scene file.
What I've found so far is that the particle exploding is reproducible (quick test using low grid res.) and mesh errors like open geometry and double faces, which might be a reason for this first issue.
Now I'm waiting for the second simulation to finish where all settings are untouched to see if the hanging simulation will be reproduced.