infuriating delays in working

[super gnu]

ive got a scene that uses 20 very high resolution textures ( 40k-80k wide, total about 19GB of data, mostly uncompressed files) displacement etc..

obviously id expect some heaviness working, but considering i have a stupidly powerful 64 core machine, with 128 gb ram, and a pcie4 NVME raid 0 array as my work drive (nearly 11GB/sec) the performance is surprisingly terrible.

opening file is not toooo bad, about 5 minutes, but opening material editor takes about 10 minutes (my drive should load those bitmaps in 2 seconds in an ideal world...) this delay happens nearly every time i open material editor.

changing multiplier or gamma on background (its a 64k wide exr, the biggest file, nearly 3GB) sometimes takes 2-3 minutes.. other times its instant. . it appears to reload the map when i tweak a setting in the vraybitmap loader.. but only sometimes.

rendering takes about 5-6 minutes to start, with cpu sitting at 1-2 percent usage. this is the case with and without displacement on.

the BIGGEST annoyance i have, is on render end..

ive been waiting 6 minutes so far for max to become responsive after finishing a render.. cpu is at 2-3 percent usage, ram is not doing much.... this happens on every render... this one in particular, i have a black override on all textured geometry, and displacement off.. its literally rendering some glowing splines for a mask.. the render itself started instantly and took 30 seconds to finish.

in the last week ive probably done about 10 hrs of work.. the rest has been waiting.


i would be prepared to share a scene with Chaos if they give me an ftp login so i can upload something.. it wont help me now, ive got to submit the job to a renderfarm in 2 days. however if i could avoid this agony next time i would be very happy.

i would also love to know why it takes so long to work with bitmap files that a) the drive should be able to load in seconds, b) mostly need no decompression c) are not displayed in viewport and d) have 63 cores sitting idle when loading.

and.. PLEASE FIX THE DELAYS AFTER RENDERING! IPR does not have these delays.

thanks!

[kosso_olli]

and.. PLEASE FIX THE DELAYS AFTER RENDERING! IPR does not have these delays.

Also had the same. We still get this quite a lot especially with very high resolutions. This also happens without any proxies whatsoever, it is the resolution: https://forums.chaosgroup.com/forum/...fter-rendering

changing multiplier or gamma on background (its a 64k wide exr, the biggest file, nearly 3GB) sometimes takes 2-3 minutes.. other times its instant. . it appears to reload the map when i tweak a setting in the vraybitmap loader.. but only sometimes.

Had this, too: https://forums.chaosgroup.com/forum/...ue-of-vrayhdri

Chaosgroup changed the behaviour a few releases ago. Basically, every time you change Gamma or Multiplier, the texture gets reloaded. However, if you change it to a value you already had before, it gets loaded from Cache, which is instant. I do not like that behaviour, as it makes iterating between HDRIs for example very slow. The HDRIs we have are up to 32k in width and several GB in file size. No fun!

[3d-laboratory]

10 Minutes in the material editor is quite much but as some other posts regarding material editor speed suggest, it will be the texture size, that slow it down. When I open CME it takes short time to render the previews of most materials, but when it comes to the dome light HDRI (now VRay Bitmap) it takes significant longer. I would be interested in what resolution your renderings have when you use such very high resolution textures? If I had the need to work with such massive textures I would replace it with a good file management system (I use Directory Opus) and replace the small textures with the big ones only for final rendering. You can use scripts for it in DirectoryOpus, it is then a one click task. But on the other hand: working with CME and SME is slower with MAX2021 and VRay5 compared to Max2019 VrayNext. There has to be some work to be done (Microsoft+Autodesk+ChaosGroup, in that order I think

[super gnu]

thanks for the quick replies... the rendering is at 4k, but the scene is earth and requires shots where we zoom from a full view, in on individual countries.. the spherical environment at 64k wide initially seemed overkill, but if you stick a long lens in the scene (necessary in some shots) the stars become blobs, so its still nowhere near "too high res"

early in the project i did play around with converting files to .tx but i noticed my renders were a little slower, and it didnt seem to reduce ram usage, so i didnt continue experimenting ( the 64k exr i cant even open in photoshop to resize, Gimp and vray are the only two applications that will read the damn thing, thanks Nasa!)


i expect using low-res proxies is the way to go, but ive been doing a ton of test rendering, and swapping back and forth would be painful.

[Joelaff]

Tiled EXR or .tx files should help, but I haven't used them much in recent version of Max/VRay. If you use these check your Tile Cache settings, perhaps give it more memory if you have it. Note that if you load these tiled textures off a network server it will use a lot of network bandwidth while rendering. You won't notice it with a single machine rendering, but if you have a good sized farm, say 30-100 nodes all reading off the server at once, that will get slow. We push the textures to the local hard drives of the render nodes in this case. (Share each node's drive, and have one machine sync the textures with Good Sync. This way you don't need a "server" OS because there is only one connect to each node rather than multiple connections to a server.)

You might use a Shell material with a low-res texture for preview in the viewport. Then the texture should not be loaded until render. In this way it may be easier to make any tweaks to your texture in an external program and just update the file, rather than using the material editor. It will render faster too.

There is no doubt that the material editor is terribly slow in the latest releases, even with the VRay HQ previews disabled. It is driving us nuts as well.

Oh, also disable automatic texture reload in the prefs. That always drove me crazy anyway.

[super gnu]

Tiled EXR or .tx files should help, but I haven't used them much in recent version of Max/VRay. If you use these check your Tile Cache settings, perhaps give it more memory if you have it. Note that if you load these tiled textures off a network server it will use a lot of network bandwidth while rendering. You won't notice it with a single machine rendering, but if you have a good sized farm, say 30-100 nodes all reading off the server at once, that will get slow. We push the textures to the local hard drives of the render nodes in this case. (Share each node's drive, and have one machine sync the textures with Good Sync. This way you don't need a "server" OS because there is only one connect to each node rather than multiple connections to a server.)

You might use a Shell material with a low-res texture for preview in the viewport. Then the texture should not be loaded until render. In this way it may be easier to make any tweaks to your texture in an external program and just update the file, rather than using the material editor. It will render faster too.

There is no doubt that the material editor is terribly slow in the latest releases, even with the VRay HQ previews disabled. It is driving us nuts as well.

Oh, also disable automatic texture reload in the prefs. That always drove me crazy anyway.

some great advice there, very timely since for some reason the renderfarm, despite having the same amount of ram as me (128gb) is unable to render my scene due to massive ram usage and paging... ive no idea why, it fits quite nicely (just) in 128gb here... ive just converted all textures to TX (and resized what i can)


so ill need to ask if they can copy the files locally.. the scene is now using about 70 gb of ram here, is there a way to tell if its "streaming" the textures during rendering or not?

back to your other suggestions.. how do i a) disable HQ material previes and b) disable automatic texture reloads? its in the max prefs or vray settings?

[Joelaff]

The way I see the network usage is to look at the network usage stats real-time on the server. You can try it without copying the textures locally, but if the server link gets saturated it will be slowing things down. Note that VrayProxies with the same way, and are usually much faster if copied locally.

The HQ material previews I was talking about are controlled in the System settings if the VRay render settings. Native 3ds Max Material Swatches. Turn that on. It just makes the swatches that render in the material editor faster, but less accurate... like they were in previous VRay versions.

The texture reloading is a Max setting in the File tab of the Max prefs.

Another note when using tiled images is never set the blur value super low, or it will always load the full res texture and defeat the purpose of the multiple resolutions (MIP) present in the file. Of course, IMHO, you should never set the blur amount below about 0.5 minimum except in special circumstances anyway (maybe close up packaging).

[Joelaff]

BTW, what is using so much memory? Is it just textures? Or geometry or volume grid, or what?

The conserve memory setting under the embree settings might help. It never seems to slow things down that much.

[super gnu]

BTW, what is using so much memory? Is it just textures? Or geometry or volume grid, or what?

The conserve memory setting under the embree settings might help. It never seems to slow things down that much.

its a high resolution globe with 86k wide diffuse texture, 43k bump, specular, and a matching cloud layer with displacement, a handful of lights and a few high resolution satellite models.

it was set up to allow zooming right in, and any other potential uses down the line, and , since it fitted in my ram, and rendered fast i just left everything at full res.. seems optimisation is needed in this case. wasnt too painful though, with teh cameras i have to render tomorrow it looks identical with all the textures halfed in size. good old "bitmap tracker/resizer" from pixamoon, its been a lifesaver on quite a stressful evening.

regarding the network usage, ill have to ask the renderfarm to check that, ive got just the one workstation here.. i was hoping there might be a message or sign it was streaming stuff..

in the end ive set the texture cache size to "0" and it still only uses 70gb now.. i assume setting that to 0 should avoid streaming textures.

interestingly i note that my c: seems to be seeing quite a lot of activity during rendering...? this does not make sense if its streaming stuff, since the assets are all on the d:

[Joelaff]

Oh, if you are not using tiled textures then you might try 8bit dithered palette (indexed color) images (like a GIF). It sounds terrible, but with most renderers this saves you 2/3 the size of the highest res MIP map (probably about 1/2 the memory).

I am not sure how VRay /Max handles this though.

Often you can’t tell the difference in the final render. Testing is in order.

I don’t recall what a cache of zero means. It might mean unlimited, which would NOT be what you want of course.

[super gnu]

Oh, if you are not using tiled textures then you might try 8bit dithered palette (indexed color) images (like a GIF). It sounds terrible, but with most renderers this saves you 2/3 the size of the highest res MIP map (probably about 1/2 the memory).

I am not sure how VRay /Max handles this though.

Often you can’t tell the difference in the final render. Testing is in order.

I don’t recall what a cache of zero means. It might mean unlimited, which would NOT be what you want of course.

i am using tiled textures (now) , and yes, setting 0 means unlimited.. i tried this to see if it maxed out the ram again -i.e. how much ram would i consume if i definitely wasnt unloading/reloading tiles during rendering.

, but increasing the cache size (even unlimited) does not cause more ram usage, which is now quite stable at 60-70gb.. i assume a) resizing the textures has helped greatly and b) maybe the tx files are loading a lower mip map level due to view distance, or loading only visible parts of the texture as opposed to the full thing?


ill see what happens at the farm tomorrow, ive uploaded my tweaked scene..

anyway got a bit Off Topic here... i jumped on you cos i was stuck and nobody had read my other thread begging for help! i was in a right flap but i think (hope) its resolved now.. many thanks

[Joelaff]

Oh, so you want unlimited? I kind of thought you were running out of memory, in which cache you want the texture to be unloaded as needed.

Yes, I believe only the appropriate MIP map is loaded. So if the texture in smaller in frame it uses less memory. Note that as your camera moves in the memory usage may go up, though.

Let us know how it all turns out.

[super gnu]

Oh, so you want unlimited? I kind of thought you were running out of memory, in which cache you want the texture to be unloaded as needed.

Yes, I believe only the appropriate MIP map is loaded. So if the texture in smaller in frame it uses less memory. Note that as your camera moves in the memory usage may go up, though.

Let us know how it all turns out.

i was running out of ram (on the farm, not at home, despite having 128gb here and there) then i converted all textures to TX and reduced res of all the ones i could.. it cut ram usage in half, but i was worried since you said streaming textures could hammer network, so i thought id see if i could tune it to use as much ram as poss without going over.. as a test i set the cache size to unlimited, and it stayed using the same amount of ram... so im now testing at the farm set like this... given ive no idea what the correct value is.. maybe unlimited is a bad option since, as you say, it could creep up during rendering and give me a nasty surprise..

also, what do you think of "clear cache on render end.." disabling that certainly makes the next render start quicker, but again, it might end up just filling my ram during sequence rendering..

[Joelaff]

also, what do you think of "clear cache on render end.." disabling that certainly makes the next render start quicker, but again, it might end up just filling my ram during sequence rendering..

Interesting. I was never so bold as to use Zero for the cache size. So I am not sure what would happen.

If the animation has all the same textures throughout then it may not be an issue, and would likely be faster not to clear it. In theory those tiles of the texture are already loaded and can be used again. I would just set the cache size to something kind of big (like a few gigs less than the free memory) and leave it. I am not sure how intelligent the cache is (like does it count accesses to each tile and preserve the tiles that are access the most, or is it just a FIFO or FILO). Keep in mind the OS does some caching of pages as well; so it may not always be a full read from disk anyway.

[trick]

I have just been reading this thread and am getting a headache thinking of working with large textures in the past, far smaller than the ones I see here. Have any of you tried working with Proxy Textures (by Spline Dynamics)? I don;t know if it works fast and errorfree with huge textures, but in my case where I always try to work with LOTS of original 4K or 8K textures and do not want to spend time searching for a balanced size until arrived at the very end of a project, it works very well.