We are experiencing very slow startup times with our 3ds Max file containing VRay Proxy objects. 26 minutes Startup time on our farm (and workstations) for the first load and 3 - 4 minutes render time after load so the render time is not the issue, just the file startup time.
Same for our artists loading the 3ds Max file, its 20 - 25 minutes to open.
The scene file contains 16,000 VRay Proxy objects spread across 5 Layers with only ever 1 Layer visible at once, so 3,200 Proxy Objects on screen at any one time.
We are using 3ds Max 2023 and VRay 6 Hotfix 3.
The Proxy workflow is quite important to us to keep the core chunk of data we are rendering in a single place and to have it referenced into the lighting files. We have chosen the ‘One Proxy per Object’ route for maximum flexibility but didn’t really expect this level of slow down. These animations are 3 - 4 minutes per frame to render but there is a serious delay at the beginning.
Really looking for help on how to identify the cause and hopefully resolve the slow startup times.
Are the proxies over the network? Is it much faster if they are local? You could use a push script or software (e.g. rsync or GoodSync) to send the resources to each node if that does help.
It does sound like there ought to be some other solution, though.
Yes, they are over the 10Gbe network which is what we use across all projects.
I have managed to speed things up by telling our render pass submission pipeline to only include visible objects in the file per pass, rather than keeping all hidden layers too. This has actually brought the Startup times down from 25 minutes to under 5 minutes.
It seems like even hidden VRay Proxy objects are automatically being loaded during startup?
It would help if you could send one such scene together with the proxies to our support so we can profile it and see what we can do to speed up opening. 25 minutes is definitely a lot..
Other than that - as Joelaff already mentioned - can you test if having all those proxies locally makes any difference. Even if the network is 10Gbe it could be an issue when doing many many small reads..
Regardless of the outcome of any dev investigation and any further optimisation, you should consider local caching to the rendering machines.
Adding a single quick drive to each machine can shave years of wasted network comms time.
Notice this is a known problem over networks (that of many small reads as opposed to monolithic, big ones.) that has no practical solution, besides caching.
It happens with xMeshes and with tiled textures, when those are being cherrypicked for their smallest mips.
The caching can be awfully simple, and it’s something also included in (free) render farm managers.
After a quick look I have not been able to find anything regarding local asset caching for 3ds Max jobs. Do you happen to have any information on this?
Umh, i am not current, i *assumed* it was present in deadline 10 given this thread on their forums.
It seems it’s implemented for maya alone, so you’d have to write your own pre-render process to distribute assets.
Use Local Asset Caching: Enabling local asset caching can reduce network overhead. When enabled, Deadline stores a local copy of all assets used during Maya renders on the Worker. The renderer then uses the assets from the local machine, rather than having to access them from their network location. The assets are stored in the Worker’s cache and can be used by subsequent frames or scenes submitted with the same assets without having to reach out to the network.
Which is likely for the best, anyway, as only you will know the precise requirements for a file overwrite or a skip (something that can very much impact overall efficiency, as many assets end up being reused across shots and jobs.).
There may already be something, an app or a script, doing it.
Otherwise, you could theoretically try and use DR while submitting frames, maybe with two clients each, as it does have an asset transfer and reuse mechanism.
But it’d have to be tried, i’ve no idea of how you are configured, and how this would work, or not, in your case, i’m afraid.
This is me just thinking out loud, but could you try caching this out to vray scene perhaps and seeing if this is better to load that on the farm.
You do have quite the case though for the proxies, max generally and other software is not great at handling thousands of anything, are those unique objects or instances? do you have the proxies all loaded directly in the scene or are they xref or other type of refs into a main scene?
The vrscene idea might be worth a test, I will try and do that this week.
all the proxies are loading directly into the main max scene and spread over 5 layers, approx. 3000 on each layer and only one layer is ever visible at one time for rendering. Think is each layer as its own render pass.
There are no instances, all individual objects.
Saving the max file with only one layer loads in approx 5 minutes. Saving the max file with all 5 layers loads in approx 25 minutes so it does seem to be a matter of ‘the more you have in the file the slower it is to open’.
15,000 files is also a number where putting these files into multiple directories (if you haven’t already-- something like 1000 items per directory max) will likely speed up the file accesses.
Also, consider configuring your fileserver (assuming *nix) to be case sensitive, which helps with many files in a directory vs. being case insensitive as, e.g. NTFS, normally is.
You could also try to using bonding/link aggregation (LACP) for your fileserver to be able to use multiple 10Gbit links, or upgrade it to 25 or 40 or 100 Gbit if your switch supports any of those. Obviously, this is a much bigger tweak, but with many benefits for the future.
Also, Deadline has a function that will remove non-rendering assets from the .Max file that is saves for rendering. I would give that a shot as well if you have not already. (I think that may be what you refer to in post #3)
I bet you will see a significant speedup from Morbid Angel’s suggestion, though.
Please keep us updated; would love to hear how things work out for such a scene.
The vrmesh files are stored across 5 directories currently, with approx 3,000 in each directory. I could try testing breaking it down further.
Regarding Deadline, yes that is the feature I am using and relying on atm, it helps massively and tbh makes sense. Why should we keep data in the render job that is not required for that particular job. It has some draw backs with are minor such as for example, if the shot camera is hidden it will not be saved with the job file and the job will fail. As I say, this is a minor consideration.
You are very right: given you are loading physically different pieces of geometry from disk with each proxy, the speed at which the operation completes will be loosely proportional to their total number.
For that count, it’s not even a terrible performance, don’t go beat the IT guys with it, by any means.
Have you found anything at all that would allow you to move, say, just the proxies to the same folder as the max file, and repath each as relative (perhaps each with its subfolder if required.)?
Removing the crowding that happens when a fileserver has to serve many clients at once will pay by far the greatest dividends, i’d wager.
It sounds like my best bet is to try and work with the Deadline guys and our TD’s to see if we can implement the local asset caching as you suggested. I am sure its possible with a little bit of work.
It shouldn’t be a terrible amount, provided the scope is kept simple (i.e. get the job done, don’t be too fancy with the requirements.).
Let me know (via email, likely better) if i can be of any help in the process.
I doubt making fewer than 3000 files per directory would have much effect. It’s usually only when you get up around 5-10k or more that this comes into play. But it is hardware and software dependent, and file system dependent.
For the local caching we have used a simple system when required. Each node shares an asset directory which is mapped to a consistent path (could be C: or otherwise).
Then there is one machine (could be a Vm, or the file server itself if you wish, or on the client machine such that you can easily run it as a pre render script of some sort when you Render) that uses some incremental file syncing software such as rsync on the command line or the Windows program GoodSync is also quite powerful and has a good GUI.
The sync operation gets called each time you render. Since it is a sync operation only the changed files get transferred; so it is fast.
Then you also need some sort of cleanup script to delete old files, or you have to manually delete when that project is done, etc.
Of course before you set this up for every node try copying the files locally and see what sort of speed improvements you get. It will likely be substantial given the latest NVME drives are 7 or more times faster than 10Gig networking, and there is only a single process reading the files at a time, rather than all nodes at once.
Funnily enough I literally just finished the local test from my PCIe SSD and although it was quicker, it still hung massively at 2% - Preparing Load phase. This is consistent with loading from the Network. During this time, there is hardly any IO from the disk either.
It still took a good 10 minutes to open the file with local caching and 90% of that time was stuck on this Preparing load phase.
