"Simulation has stopped due to numerical errors!" > fairly reproducible error in simple setup

[tysonibele]

Hey guys,

I've run into Phoenix instability with a very simple setup. This sim fails or explodes pretty consistently in the first 15-20 frames of simulation (and this scene sims at 1s/frame so it shouldn't be hard to reproduce the problem), despite the fact that there doesn't seem to be anything wrong with the particles sent to Phoenix. Sometimes it gets further than 15 frames so you may need to run the sim a few times to get the error, but here on my machine I can get it to popup within a couple minutes of hitting resim a few times over the first 15 frames.

This is in 3dsmax 2017 w/Phoenix 3.14.00. I have a machine with 16 cores in case that matters.

The PhoenixFDSource is importing a tyCache (that has its particle interface on, so everything is happening through IParticleDataExt) and it's set to use the particle geometry as the source.

I've stepped through the tyCache particle interface code many times and all the data sent to PhoenixFD seems legit. I don't see any problems with either the logic, or the code. PhoenixFD calls UpdateParticles on my interface, then gets particle IDs, ages, shapes and TMs. All pointers I send to PhoenixFD should be valid and all mesh generations and array modifications happen inside mutex locks in case PhoenixFD is querying things in a threaded manner. And I can't reproduce the issue if I change the PhoenixFD source to a non-"use particle shape" mode. So the issue seems related to simulating particle shapes.

So basically I've exhausted all my troubleshooting options here and am reaching out.

Attached is the file, including relevant caches. Just retarget the tyCache in case it can't find it. Also if you get a tyFlow version error, it can be ignored so long as you have the latest release installed. Thanks!

[tysonibele]

It's worth noting that this issue also occurs in the latest nightly build.

[Svetlin.Nikolov]

Hey, thanks! We'll check it and ping you when we know more...

[Svetlin.Nikolov]

Dang, so - I reproduced it with 3.14 and 3.14 nightly, but we've got some beta builds coming with some more significant changes in the sources where these NaNs don't appear anymore. In a few days the new builds will be public and it should be resolved in those...

[tysonibele]

Ok great! I'll let you know if I run into the same issue again....

[Svetlin.Nikolov]

Hey tysonibele , I was just looking at a test scene that broke after my previous change, so there will a new change in the nightlies addressing that. Meanwhile I was playing with the attached scene and changed the Phx Source from "Use particle shape" to "Sphere, use size", and also increase the Steps Per Frame to 10. This way Phoenix is supposed to trace the particle trajectory and create fluid throughout, so it calls ParticlePosition() and ParticleVelocity() on the particle object at each substep with the respective TimeValue. However, the position always returns the same and does not change when advancing the time value by 16 for each of the 10 simulation substeps. Additionally, UpdateParticles() also gets called on the extended interface at each substep with the same time values. Could there be an issue in tyFlow in acquiring the particle position this way?

[Svetlin.Nikolov]

Oh, just tested "Use particle shape" as well and it seems like the same behavior appears there as well - GetParticleTMByIndex() always returns the same transform for substeps, and in "Use particle shape" this is even more important because Phoenix routes the geometries of particles through the processing for regular meshes and so it does not try to connect the particle trajectories and using more than 1 steps per frame is the only way to get an uninterrupted trajectory for fluid sources.

A few other things you gotta watch out for are that the fluid's conservation of momentum would damp the velocities the particles bring into the solver, so using conservation of 1 and 0 vorticity would minimize this effect. It's also useful to export grid velocity and keep and eye on the velocity channel range in the Simulation rollout's cache file content box - the voxel velocity is in voxels/sec. A thing I need to think about would be that a Source in Volume Brush mode would only apply a portion of the motion velocity is the brush effect is below 100%. Right now I'm not sure whether this is more of a desired effect or not, or if it would be better to have the motion velocity unaffected by the discharge settings.

[tysonibele]

Hey,

Not sure if you're using the latest build (v0.16046) but there was an issue with tyCaches not returning proper substep positions/transforms so I think that's what you may have been running (assuming you're using v0.16045 or below). So that should already be fixed.

[Svetlin.Nikolov]

Hooray! Gonna update now...

[Svetlin.Nikolov]

Ah, so using 0.16046 the tyCache particles sampling using ParticlePosition() at each Phoenix substep is now correct.

Unfortunately, sampling the particle shapes in "Use particle shape" mode of the source using GetParticleTMByIndex() still returns the same matrix for all substeps. Does this reproduce on your side?

[tysonibele]

Hey sorry, didn't see your message until now. I will look into the TM issue, although it's sampling the same data that ParticlePosition takes from so it's odd that it's not properly inteprolated...

Edit: ah yes, this is indeed a bug and will be fixed in v0.16048.

[Svetlin.Nikolov]

Sweet, thanks a ton!

[stevesideas]

I'm having the same issue with build: 4.30.01 Nightly, Build ID: 2020122330512

[stevesideas]

Not sure what happened but since resetting the sim to a preset it is working fine now....odd