Large liquid similation - Phoenix FD not using full CPU usage

[legi008]

Hello,

We are facing a strange behavior from our Phoenix FD simulation scene.

We are currently simulating a large liquid simulator of 12m x 3,5m x 6,5m with cell size of 1cm (Total 265.972.000 cells- because it's going to be projected on a 1:1 display with small viewer distance). It's basically an aquarium that need be fill in and where the water need to be push, rotate from ground to ceiling, etc..
The simulations is calculated on a AMD Ryzen 9 3950x (16cores-32threads) and the files are written directly to an M2 SSD into the computer. The foams and splashes is not yet calculated.

The problem is Phoenix FD only use an average of 49-53% of the CPU usage (With high peaks to 100% when moving from step to step) .

So we are thinking that it's 2x slower than expected to be calculated. Is it normal knowing the container is still 60% empty from liquid and will never go full.
At this moment (frame 460), one cache file is around 1Gb with 85.140.464 millions liquid particles and 2.544.825 particle for wetmap.
I've put a screenshot of our main settings.

We are using Max 2020 and Phoenix FD 4.0 because we are still under VRay 3.7.

Thank you

G

[Joelaff]

Phoenix moves a LOT of memory so things like bus speed and cache size come into play. I wouldn’t necessarily expect it to use 100% of all cores. In fact, try playing with the Max number of threads right by the start and stop simulation controls. Sometimes it will simulate faster with few threads.

[squintnic]

you need to get an intel chip with low cores and high clock speed (5.3ghz)
phoenix just doenst work that welll on AMD chips, they are crippled.

[Svetlin.Nikolov]

Hey legi008 , if you are using Phoenix 4.0, definitely get the latest Phoenix 4.30. We did a lot of optimizations regarding the simulation, rendering and previews since them and machines with many CPUs, as well as NUMA CPUs like AMDs models should be more efficient now.

Check the changelogs here:
https://docs.chaosgroup.com/display/PHX4MAX/4.10.00
https://docs.chaosgroup.com/display/PHX4MAX/4.20.00
https://docs.chaosgroup.com/display/PHX4MAX/4.30.00

Cheers!

[legi008]

Hey, thank you all for your feedback.

Joelaff, i'v tried to tweak with different numbers of threads (8-16-24-30) but it did not change anything, always higher calculations times.

Squintnic, yes it could be a workaround but it would need us to buy a new computer with out knowing what the best CPU and if it would be 5-10-40% more efficient.

Svetlin,as i said we are still working with V-Ray 3.7. And i think Phoenix 4.3 drop support for V-ray 3.7 right ?
Is it possible to simulate with Phoenix 4.3 and render with Phoenix 4.0 ?

[Svetlin.Nikolov]

Hey, thank you all for your feedback.

Svetlin,as i said we are still working with V-Ray 3.7. And i think Phoenix 4.3 drop support for V-ray 3.7 right ?
Is it possible to simulate with Phoenix 4.3 and render with Phoenix 4.0 ?

Wow, V-Ray 3 is terribly old and slow, both in general and for Phoenix volume rendering in particular, so I am kinda surprised you are not struggling with render speeds as well.

We extended the AUR cache format twice since 4.0 - in 4.20 and 4.30, so new Phoenixes can read old AUR caches, but old Phoenixes will not know what's inside of new AUR caches, so I'm afraid you won't be able to render with Phoenix 4.0, unless you write VDBs instead.

However, I don't think you need this - just grab a Phoenix nightly for V-Ray 3: https://nightlies.chaosgroup.com/#/p...ghtly/20201012
Even though we dropped V-Ray 3 officially, I will keep the nightlies until it's technically impossible for them to keep building (which might come soon as well though). So you can grab a latest nightly build and it will work with V-Ray 3.

Hope this helps!

[legi008]

Yes unfortunately i know V-Ray 3 is old but our company had to make some priorities before switching to V-Ray 5 ( If we ever do).

Oh i didn't know about the nightly build that still support V-Ray 3. I've installed the latest uploaded from this night and i can already see some improvement.
The CPU is now using around 65-74% and the frames are simulating 28-31% faster. Definitively a big improvement !

Thank you for pointing this out !


I have 2 more questions about the step per frame :


The first 500 frames of our shot have fast liquid movement so i've put a SPF of 8. Result are good. But after that, the liquid stay still (or very calm to arrive nearly still) for 400 frames. Can i decrease the SPF to, let say, 4 or 5 to save some time and when the liquid will go back to moving after that, readjust the SPF to 8 ? Will it work ?


And for the re-simulation of the foams and the splashes, can i set the SPF lower than the original simulation (If simulation SPF set to 8, Resim set to 4) to save time, or it's not recommended.

[Svetlin.Nikolov]

Hooray!

We have some more improvements in mind for many CPUs, we'll see if they do work as we hoped...

Animating the SPF is possible and you should be able to get away with it in some cases, if the fluid is not moving very rapidly and if the liquid is NOT very deep. Liquid which needs to keep its volume because there are many stacked liquid particles might lose its volume in case the steps are not enough. So if this happens in your setup, an alternative might be to move the keys of the interacting nodes closer together and shorten the 400 frames to 20-30, and then animate the Direct Cache Index in the Input rollout so that the animation is stretched when Phoenix loads the cache files for the viewport and for rendering.

Resimulation with SPF other than 1 can fail gloriously. You might be lucky, but in general only base simulation with SPF = 1 and Resimulation with SPF = 1 will work as me and you expected. Other combination might work, or might not. The reason is that if the Base simulation runs at 8 SPF - it does 8 substeps and writes the cache file with the velocity after the 8th one, then does 8 more and writes the velocity after the 8th one, etc. So when you resimulate, Phoenix reads the velocity from the cache files and drives the simulation with it, but if anything interesting happened during the 7 substeps that were NOT written to the cache file, the resimulation would not know about it. And these (small or large) differences will also accumulate over time, so the resmulated result might end up being completely different from the Base sim.

Hope this helps!

[stevesideas]

Hi there. I'm having the same issue. I have an AMD ryzen threadripper 2990WX 32 core processor. It has 64 threads and although Phoenix shows all 64 being used actual CPU usage is just 15% ! Is there anything I can do apart get a different system? Seems ridiculous!

To add. I am running the latest version of Phoenix. 4.30.00. Windows 10. 64 GB RAM. 3dsmax 2021

[Svetlin.Nikolov]

Hey,

Can you send the scene you are using? Also, a Phoenix log copied right after the simulation is run will show very clearly which part of the simulation is the bottleneck.


As a side note - for me it's common sense, but maybe it's worth mentioning:

- Phoenix has two entirely separate solvers - one for fire/smoke, and another one for liquids.

- Each has between 15 and 20 stages it executes during simulation, depending on the settings you use.

- These stages all can be single or multithreaded, some can scale well, others - not so well. Each is implemented separately and each can perform differently on different hardware.

- Each of these stages can depend on external factors - are there any forces in the scene (forces are read single threaded due to limitations of 3ds Max and Maya). How much geometry is involved. How much of the geometry is moving, or rigged, or are there any texture maps involved, or are there any sources, discharge modifiers, or tuners, or mappers....

Knowing this, I hope it makes sense when I say it's very unlikely that it's the same issue and very likely it's a special case.

Are all the toolbar presets simulating with no more than 15% CPU utilization all the time?

Btw, in today's nightlies there is an optimization for low CPU usage with texture mapped sources, do you have something like that in your scene?

Thanks!

[Svetlin.Nikolov]

Ah, one more thing - unfortunately AMD did something naughty with their CPUs and it affects Threadrippers more or less - if I remember correctly, 2990 is NUMA, and I am not sure if your Windows task manager and thus if Phoenix recognizes it as a NUMA architecture, allowing you to use only one of the underlying NUMA nodes. NUMAs are disasterous for fluid sims because of the different access time of different cores to memory, which is critical for such processing. I remember that many AMD models did not actually show in Windows as NUMA, thus not even allowing you to restrict the simulation only to the NUMA node which has equal memory access time for all cores. Often NUMA setups run simulations in a very suboptimal way, and in extreme cases using all cores can even be slower than using only the cores from a single NUMA node. So this one could be a factor in your setup as well...

As mentioned in the sticky forum thread with hardware advice for faster simulation, the bus speed is also crucial for the same reasons as above - fast memory access is paramount for fluid sims, so this is also important to mention. In the office we have a pair or Xeons which are beasts, but a poor motherboard cripples them severely..