foam missing from cache on some frames. nightly 2020080930269

[spaceRick]

hi,
on an ocean scene simmed in 4.20.01 Nightly, Build ID: 2020080930269 there's a few issues, but the main one right now is that the foam particles are missing on some frames. they don't display in Maya, or in the standalone, but do show up in the particle counts in Maya and in the logs, as well as the standalone.

as an example there's a stretch of 12 frames starting at f1072 (sim is 910-1121) where the foam disappears, then comes back again on the 13th goes again for another 6 frames, then comes back til the end of the shot. the movement of the foam looks correct when it comes back, as if it has simmed fine, and the particle counts check out too but no foam displays or renders.
82,516,698 liquid particles, 73,495,490 foam particles, Grid Size: 1280 x 738 x 258.

I set off 2 nearly identical sims at the same time, the only difference being droplet breakup. both sims have the same issue, but on different frames / sets of frames.


the other oddity is doubled up foam on backup interval frames when motion blur is enabled. backup intervals set to 10.

[Svetlin.Nikolov]

Hey, would it be possible to send over the scene, together with 3 caches - one where the foam is alright, one backup frame, and one where the foam is missing, to support at chaosgroup dot com?

If you also keep the log files from the simulation, they would be very useful as well.

Thank you in advance!

[spaceRick]

hi, I have some uploading now for that very thing, i'll send the two I have (1071, 1072) and will follow with 1070 which is a backup frame. watch out for 2 emails!

[Svetlin.Nikolov]

Awesome, thank you so much!

[Svetlin.Nikolov]

Hey, got the files, starting to look at what's going on.

Btw, the V-Ray log is of not much use to me unfortunately. There is a dedicated Phoenix log file that you can open from the Phoenix Preferences dialog: https://docs.chaosgroup.com/display/...al+Preferences; On Windows it's under C:\PhoenixFD - if possible, please send this over, captured exactly after the render, before a new Maya instance is opened.

Will ping you when we know more...

[Svetlin.Nikolov]

Hey, so for now it looks like you might have overflown the size of the AUR format. I will extend it and will run some tests here. I hope in a couple days you will have a new nightly build you can use. This is also the reason why the issues happen on backup frames - because more data gets written then. Will write again when I know more...

[spaceRick]

Cheers Svetlin, thanks, i'll be honest, i'm not sure what that means! backup frames were reduced from 15 to 10 at some stage when crashing was occurring. I did successfully finish that sim by restoring the sim locally [windows] from 1060 and then uploaded the caches and continued to render, sans motion blur. the first original problem frame was 1068 so restoring from 1070 didn't work either - that continued to write bad caches. I've replied to my support email a few times with extra bits & updates, including that 1060 frame I was able to restore from in case you're missing it.

I don't have access to that log, i'll ask for it to see if it's still there. I wonder if you could also consider writing these PFD logs out to the cache destination, or in a way they they're timestamped and retained in their usual destination? they seem pretty important for your trouble shooting and they're gone in an instant! we appreciate the changes and updates you have made to verbosity of the vray logs by the way, that's been very helpful.

[Svetlin.Nikolov]

Ah, it means that there are too many particles and they also have too many channels, so they reach the AUR format's memory limits. In backup frames, even more data is written, so the memory increases even more. The good news is that I can extend this, the bad news is that you will have to simulate again..

[Svetlin.Nikolov]

Hey, in tomorrow's (20 Aug) Phoenix 4 nightly builds there will be a change that should fix this issue. AUR files will be able to contain much more particles than they did before. Unfortunately the AUR files that you already have written out can not be saved Data was lost at the moment they were written by the simulation, so you will have to simulate them again.

The new AUR files written since the new nightly will not be readable by older Phoenix versions though! This is why I bumped the nightly version and it will be 4.20.03; Cache written by old Phoenix versions should open fine with new Phoenix versions though (except for the problematic caches you sent over).

Please give the new nightlies a try and please ping me if you notice any issues.

This change should also take care of the crash at backup frames and the double foam too. However, if you notice anything wrong, please ping us again.

Thank you very much for reporting this and for pushing the limits of our software!