Vray Proxy Workflow - Very Slow Startup times, need to resolve..

[^Lele^]

You are very right: given you are loading physically different pieces of geometry from disk with each proxy, the speed at which the operation completes will be loosely proportional to their total number.
For that count, it's not even a terrible performance, don't go beat the IT guys with it, by any means.

Have you found anything at all that would allow you to move, say, just the proxies to the same folder as the max file, and repath each as relative (perhaps each with its subfolder if required.)?
Removing the crowding that happens when a fileserver has to serve many clients at once will pay by far the greatest dividends, i'd wager.

[brobins]

You are very right: given you are loading physically different pieces of geometry from disk with each proxy, the speed at which the operation completes will be loosely proportional to their total number.
For that count, it's not even a terrible performance, don't go beat the IT guys with it, by any means.

Have you found anything at all that would allow you to move, say, just the proxies to the same folder as the max file, and repath each as relative (perhaps each with its subfolder if required.)?
Removing the crowding that happens when a fileserver has to serve many clients at once will pay by far the greatest dividends, i'd wager.

It sounds like my best bet is to try and work with the Deadline guys and our TD's to see if we can implement the local asset caching as you suggested. I am sure its possible with a little bit of work.

[^Lele^]

It shouldn't be a terrible amount, provided the scope is kept simple (i.e. get the job done, don't be too fancy with the requirements.).
Let me know (via email, likely better) if i can be of any help in the process.

[Joelaff]

I doubt making fewer than 3000 files per directory would have much effect. It’s usually only when you get up around 5-10k or more that this comes into play. But it is hardware and software dependent, and file system dependent.

For the local caching we have used a simple system when required. Each node shares an asset directory which is mapped to a consistent path (could be C: or otherwise).

Then there is one machine (could be a Vm, or the file server itself if you wish, or on the client machine such that you can easily run it as a pre render script of some sort when you Render) that uses some incremental file syncing software such as rsync on the command line or the Windows program GoodSync is also quite powerful and has a good GUI.

The sync operation gets called each time you render. Since it is a sync operation only the changed files get transferred; so it is fast.

Then you also need some sort of cleanup script to delete old files, or you have to manually delete when that project is done, etc.

Of course before you set this up for every node try copying the files locally and see what sort of speed improvements you get. It will likely be substantial given the latest NVME drives are 7 or more times faster than 10Gig networking, and there is only a single process reading the files at a time, rather than all nodes at once.

[brobins]

Funnily enough I literally just finished the local test from my PCIe SSD and although it was quicker, it still hung massively at 2% - Preparing Load phase. This is consistent with loading from the Network. During this time, there is hardly any IO from the disk either.

It still took a good 10 minutes to open the file with local caching and 90% of that time was stuck on this Preparing load phase.

[Joelaff]

Wow. I would try Morbid Angel’s idea of combining as many nodes as you can into one.

[Joelaff]

Oh, also might be a good plan to try to render (could just do a 3dsmaxcmd render from the command line locally), as perhaps some of that delay is the preview and other interactive aspects of the load.

Speaking of which I would try disabling all the previews of the proxies completely. I would assume that this would be utterly ignored when rendering, but assuming rarely works in anyone's favor.

You could also run a quick test and do an export to vrscene and render (via the VRay menu in Max ). This would bypass Max entirely and render with VRay Standlone. Note that this does have numerous limitations and gotchas, but they are surmountable if you have time to figure them out.

[yavor.rubenov]

Hi there,

This was submitted yesterday, I believe.

Should you require anything further please let me know.

Thanks

Thanks for the scene.
Some preliminary tests and results I got:
- as you've noticed - having the scene on a local drive is faster. The slow preparing loading part looks related to some work 3ds Max is doing on resolving asset paths (in this case the proxies). It looks to me as if there's something suboptimal there in case of thousands of scene objects with asset paths. Will try to dig a bit more into it and maybe contact Autodesk on it.
- having the proxies on a network drive can be quite slow indeed. In some short test I did here I see somewhere between 6-10x slow down on reading some of the proxies data. Unfortunately - all proxies are loaded even hidden ones and all of them need to read some data from the file.

We have some ideas on improving the loading time for proxies especially in such cases but those will take some time.

[brobins]

Oh, also might be a good plan to try to render (could just do a 3dsmaxcmd render from the command line locally), as perhaps some of that delay is the preview and other interactive aspects of the load.

Speaking of which I would try disabling all the previews of the proxies completely. I would assume that this would be utterly ignored when rendering, but assuming rarely works in anyone's favor.

You could also run a quick test and do an export to vrscene and render (via the VRay menu in Max ). This would bypass Max entirely and render with VRay Standlone. Note that this does have numerous limitations and gotchas, but they are surmountable if you have time to figure them out.

I tested the vrscene and VRay Standalone render job a few days ago. It is sent via Deadline to the farm as 2 jobs, the first is the creation of the vrscene file and the second is the rendering job. The rendering job did just pick up and render right away with no 'Startup time', however, the job to create the vrscene file had the same 25 minute Startup time (and only a 32 second 'Render Time' in making the vrscene) so it was just a trade off in a way.

I have tried with all Proxies set to display as Box, this had the same performance when opening the scene.

[Morbid Angel]

Is it 25 min startup time per frame or for the whole thing? I think the trade off is you load the scene once output vrscene and then render the actual frames fast. Also not sure, but I think in max you can load vrscene as a container (I can't remember, been long time since I used max) but in that case you can write your scene directly from max ui when you have your scene loaded, then submit a separate scene with the container loaded to farm already.

[Joelaff]

Yes, you can tell Deadline SMTD to create the vrscene locally, which would mean the scene was already loaded, and would likely avoid a lot of the delay.

However, vrscene really is a can of worms. Just be prepared for weird bugs where things render differently with standalone than with Max. We went through a lot of this lately, reporting numerous bugs. I don't think any are fixed yet (other than the ones in Phoenix, as those guys seem to be some kind of super heroes and churn out fixes in no time).

Most of the differences involve bugs when using render elements, also VRayDisplacementMap renders a little differently (offset does not work the same), and the Max procedurals render differently, etc.

[Joelaff]

The fact that standalone loads so fast makes me think this is a Max slowdown. Perhaps set the proxies to display NO preview at all (I guess point mode is the most rudimentary).. And anything else you can think of that would limit any processing Max would need to do when loading the scene. Definitely disable all material preview (set to shaded Object Color in the Display Tab of the command panel seems to do a good job of this).

[Morbid Angel]

That's why I would suggest to load vray scene as a container into max and still utilize the render aovs from max interface. Almost treating vray scene like vray proxy in a way.

[^Lele^]

It's being profiled right now, but it looks like Max may have indeed something to do with it.
Dmitry's solution may well be a fix, as Max in that case would be essentially bypassed.
It's not official yet, though.

[yavor.rubenov]

Funnily enough I literally just finished the local test from my PCIe SSD and although it was quicker, it still hung massively at 2% - Preparing Load phase. This is consistent with loading from the Network. During this time, there is hardly any IO from the disk either.

It still took a good 10 minutes to open the file with local caching and 90% of that time was stuck on this Preparing load phase.

This long wait is related to Max's missing paths cache. You can test turning it OFF before opening the scene. Especially with locally cached proxies it should be noticeably faster.
You can turn it off with the following MAXScript:

missingPathCache.cacheEnabled=false

Some more details on it here: https://help.autodesk.com/view/MAXDE...2-E02186B34C5C