Trouble loading .aur file from NAS

[Joelaff]

So we have a new NAS setup, and some clients randomly seem to sometimes have trouble loading large .aur files from it when a bunch of client concurrently hit the new server (which is an enterprise Dell server with good hw). However, if I put the file on the old NAS at 1 Gig they all load every time.

I know, this sounds like a hardware or NAS issue, and it very well may be. However, I am trying to find any info on WHY this would be. The clients don't time out. Everything else in the scene loads fine from the NAS, including the scene file, textures, and tycache files. But the .aur keeps failing to load, but only form the new NAS.

I thought I would post here in case the Chaos crew had some idea of what Phoenix is doing differently form the rest of the Max file loading. I am using a single .aur file as a still of some smoke frozen in time. The aur file is about 965 MB.

I tried loading the same .aur file using a VRayVolumeGrid and ran into the same issue. Do Phoenix and VolumeGrid have some kind of timeout on the load? I couldn't find anything in the VRayLog file. It never says anything about a failure to load the aur.

For right now this is Max 2021 VRay5 (latest release) and a nightly of Phoenix 3 (actually 4). It's time I upgrade all the Chaos stuff, but we had been buried in production.

Any ideas? Thanks in advance.

[Svetlin.Nikolov]

Hmmm, how about a vdb file with a comparable size, just as a sanity check?

It doesn't ring any bells though. The last time we had issues loading files was over the network with Windows XP...

Also, is there anything in the Phoenix log file?

Cheers!

[Joelaff]

Above I said Phoenix 3. I meant 4 Specifically Chaos Phoenix 4 ADV for 3ds Max 2021 / V-Ray 5.x, version 4.41.02 Nightly, Build ID: 2022031131297 from Mar 11 2022

Not seeing anything in the Phoenix log - the one in C:\ChaosPhoenix\ChaosPhoenix 3ds Max 2021.log

This is from a node that failed to render the volumetric:

Chaos Phoenix 4 ADV for 3ds Max 2021 / V-Ray 5.x,
version 4.41.02 Nightly, Build ID: 2022031131297 from Mar 11 2022

Log Started on 2022.07.15, 11:46
Windows version: 6.2
Platform: x64
Memory info: Total: 63.922 GB; Free: 60.063 GB

Loaded C:\Program Files\Autodesk\3ds Max 2021\plugins\phoenixfd\distance_measurer_phx.dll
Loaded C:\Program Files\Autodesk\3ds Max 2021\plugins\phoenixfd\phoenixfd_vray.dll
PHXToolbar: NOTIFY_TOOLBARS_POST_LOAD
PHXToolbar: NOTIFY_TOOLBARS_POST_LOAD
PHXToolbar: NOTIFY_POST_WORKSPACE_CHANGE
#BaseObjMax::updateNodeVersion() - file version 60 created with buildID 2022031131297
Pre render
Post render
Max close 2022.07.15 11:52
~FireSimObjMax 87a70160

I tried enabling slog and rlog in the pref and it wrote a zero byte file in G:\mypatht\Max\scenes\MyScene.max_Phoenix_frames\s ettings\rendering\record_y2022.m07.d15_11-27-39.rlog
But it is zero bytes. Also, would all nodes be writing to that same file?


BUT, the good news:
I converted the .aur file to a VDB and it loads every time! So I guess the question is, in what ways are aur files loaded differently, and why might they silently fail to render with a slow load. The VDB file is three times the size (~3GB).

I guess it is time to upgrade to Phoenix 5 and VRay6. SHould I jump straight to the latest nightly, or install the release?

[Morbid Angel]

Yeah the vdb loads for now, but its 100% hardware related. Its probably due to the size of file and likely files like large alembics and textures would also fail at one point. What OS is your nas running? Is your hardware running using smb2 or smb3? does your nas have static or dynamic ip? it can also be related to your domain and how the nas is connected to your active directory (if that's what you are using)

[Joelaff]

FreeBSD TrueNAS Core. Samba4 I believe. I did try setting the SMB version back down to SMB2 and that didn’t have any effect. 256GB RAM.

The IP is set as a static via local DHCP, but will set static manually just in case. Nothing else fails either in rendering scenes or benchmarks. It will saturate 10 Gigs. Intel NIC (Dell part)

Weird that the vdb loads fine at over three times the file size. The aur is 950MB, while the vdb is over 3GB and loads fine. So I am curious if they are using different read methods or one is memory mapped something.

And why is Phoenix not reporting an error?

[Svetlin.Nikolov]

Yeah, if anything breaks during the load, it should be in the log. Otherwise, the loading method is completely different, there's very little in common between the two formats.

Please grab the latest 5.01 official or nightly 31513 (it's the last one all our tests have passed on), and from the Phoenix Preferences, set the log verbosity to Debug and shoot me the log if possible. Don't forget to turn the verbosity back to the default Important verbosity, or it might slow the simulations down. Ah, btw, was the AUR cache simulated with the same 4.41.02 nightly, or was it from an older Phoenix version?

Ah, and one last thing that could tell us something - can you try opening the AUR with the Phoenix Previewer, instead of inside Max?

PS: There is just one rlog file with all simulators in it. Strange why it is zero bytes though... Could you try a brand new scene?

[Joelaff]

Thanks for the replies.

I believe I have the solution, but want to do further testing before I jump the gun on sharing it.

Where are the nightlies for Phoenix 5? I can't find them with the PHX3 or PHX 4 nightlies.

We installed the release version of Phoenix 5 and VRay 6 on all the nodes. This didn't change anything in term of the problem, though nice to have for many other reasons!

I wonder if the zero byte rlog was because we were normally cancelling the renders once we noticed the failure (visible in the light cache calc).

Is the setting to write the log, and the verbosity saved in the scene, or globally? If globally, I guess we would have to set it on every node then?

[Svetlin.Nikolov]

Ah yes, the verbosity is per 3ds Max version, it would apply to all open scenes in Max 2023 for example.

If you can't see the nightlies for Phoenix 5, please make sure to log out and log in so the visible folders refresh

[Joelaff]

Ah yes, the verbosity is per 3ds Max version, it would apply to all open scenes in Max 2023 for example.

What about render nodes? Where are they getting this setting, from some local-to-the-node file or registry setting?

If you can't see the nightlies for Phoenix 5, please make sure to log out and log in so the visible folders refresh

Aha! I wish I had known that yesterday when we installed all the latest versions! Thanks for getting me setup with this, though.

[Svetlin.Nikolov]

Yeah, the nightlies site needs sooooomelotta love.

On the render nodes question though - this verbosity only affects the Phoenix log from the Phoenix plugin for 3ds Max / Maya - when you render with V-Ray standalone, it has its own verbosity levels.