Optimizing/organizing scenes.

[Vizioen]

As computers become faster, and deadlines become tighter (in my case), I never find the time to find a good organization/optimization for my scenes. Nor have I got the time to do extensive research. One big problem is for instance if I'm done rendering it takes a very long time for max to be released after the rendering has been completed. (Memory use is going down slow). Or I have texture files all over the place (NAS, desktop, project folder).

So how do you guys optimize/organize your scenes? How do you keep memory usage as low as possible? How, what, why, when,...?

Examples:
-VrayHDRI (mipmapping?)
-LOD Trees?
-Saving to vrimg?
-Using scripts to resize textures, organize,...
-What do you put in different layers?
-Memory buffer disabled?
-Do you create a project folder and does it help with the organizing?
-Where do you put proxies that you use to scatter (I always place everything far away from my scene, but that results in the scene bounding box error, although not that bad I don't know if it influences something. Also it takes me while to navigate there sometimes.
-If you repeat a few times a piece of furniture from 30K or something is it used as a proxy?
-Scenes in Xref?
-Groups?

How do you keep your viewport performance as fast as possible with as much feedback as possible about what you're doing:
-show bounding box?
-point clouds?
-nitrous texture resolution?
-do you prefer shaded/realistic with edges/...
-how do you display textures (I can't find a way to display VrayHDRI textures in my viewport with real world map scale for example)
-Do Vraylights display properly their illuminance values properly?
-How do you have feedback from a Vrayhdri on a domelight in reflection without putting it in environment?

It's a very broad subject, I know, but I hope to get the discussion going a bit on how you guys work the fastest and hopefully we can all learn from this.

[Dave_Wortley]

I'm planning on releasing a training course similar to Grant Warwick's Mastering VRay which will be aimed at Mastering the Technical Side of 3d. Scene management, optimising, structure, organisation, layers, XREFs, VRayProxies, Allembic / XMesh are all things I plan to cover.

[Vizioen]

That would be great, maybe all the topics that could be included can be discussed here, maybe some tips/tricks from other people that can be included so we have a nice overview. Do you have an estimate on when you'll be doing this?

[Joelaff]

Easiest ways to reduce memory usage are:

If you use Embree, enable conserve memory.

Watch your BSP settings. Don't set face level coefficient too low (though lower is usually faster it uses more RAM), or limit the depth

Always use VRayHDRI. It will only load textures that are seen in the current frame (this is often all of them due to GI bounces or reflection, though.) Regular Bitmap nodes load the entire texture at the start of the frame rendering, even if the object is out of view.

Use tiled TIFF files (.tx). These can be made with maketx (Google it). Pixamoon has a script on script spot that I was working with him on to get maketx support added. That can help automate this. Tiled .tx files must be loaded with VRayHDRI. It will load only the tiles needed (assuming you have texture filtering on, and GI texture filtering on, which you should anyway). The tiles are loaded into a cache global to the scene. The cache size is set once for the scene (in any VRayHDRI mode). If this cache size gets exceeded then tiles will be flushed, presumably first in first out.

Another option is tiled EXR files. These are good for floating point textures, but use double the memory of the 8 bit per channel .tx files. Tied EXRs can be made with Pixamoon's script, or with the image2tiledexr.exe file installed by VRay (which his script calls). Note that the entire file is loaded into memory if you use a bitmap node rather than a VRayHDRI node. So don't.

VRayProxies are incredible. If your scene has tons of geometry, or if some dense meshes are only seen for a few frames they can help a lot. Only the portion seen in the render (or reflections or Gi) is loaded. The amount that is cached in memory is controlled by the dynamic memory limit. Smaller render block sizes will tend to load smaller chunks as well (depending on the voxel size). They can be displayed as boxes in the viewport, which is really quite fast.

There is a script on script spot to convert a selection of instances to VRayProxies such that they all reference the same proxy file. This saves geometry cache space, as a single proxy file is only loaded once if different proxies reference the same file.

When using either tiled textures or proxies the sequence order becomes more important. Triangulation is usually pretty good, but just think about what has to be loaded into memory and choose the best order based in that. Or if you have a few slow blocks that tend to cause Last Block Syndrome (where you are waiting a long time for the last block to render) you may want to set the order to start those blocks first (or reduce the block size). Dynamic splitting is good, but it needs to be much more aggressive in my opinion, or really needs an aggression tuning value to subdivide earlier.

Xrefs, either object or scene, help with the scene file size and loading times (and, as Vlado pointed out elsewhere, can be cached for DR rendering).

Putting a delete mesh modifier on complete geometry and setting it to off in Render can speed up your viewport by hiding things that can still render.

Limiting your render elements can save a lot of memory, and don't forget the old trick to shrink your Max render size to something small and manually set the VFB size.

Dense light caches can take up a ton of memory, often with very little benefit (enable retrace and increase the threshold if you have to).

The Summary Info (bottom of the Max menu) can export a .csv file that you can load into a spreadsheet (Libre Office free calc) and sort to see which objects have the most polys and should be targeted for conversion to proxies.

If you have a ton of small animated objects then the RayFireCache (if you have RayFire) can help a lot with interactive playback (and with retiming).

In most cases Max is MUCH faster, both in the viewport, and rendering, if you have fewer complex objects vs. tons of small objects. So Attach-ing is your friend. In a recent project I attached about 1000 objects into one and saved about 35% on render time per frame, for the same result!

Hope this helps. You can save a ton of memory like this. We employed all these tricks on a recently complex commercial that involved an entirely CG scene with people shot on motion control (and some locked off). Pretty much everything in the scene had to animate, mechanically flipping and spinning and sliding around to change between different environments. So there were a lot of parts and pieces, and a lot of textures, etc. most were only seen for a few seconds. So the tiled textures and proxies helped tremendously.

[Joelaff]

I *think* putting things far away in your scene, at least renderable objects, can mess up your BSP and make the rendering slower. I know some renderers have this issue. I have not tested it with VRay... Vlado??

[vlado]

No, I don't think it matters much how far away the scene is, at least for the kd tree and Embree. It is not a good idea for other reasons though.

Best regards,
Vlado

[Joelaff]

Good to know. Thanks.

[3LP]

Great read Joelaff!
Thanks for the infos!

Stan

[Vizioen]

Indeed great read, it'll take some time to fully integrate all these in my current work-'notsomuch'flow. The only options I touch in the BSP is enabling embree and dynamic memory limit to 0. Conserve memory is slightly slower it says in the help. How much slower is slightly? I'm not really sure even after reading the help what the rest does, might take some testing.

[vlado]

There is a script on script spot to convert a selection of instances to VRayProxies such that they all reference the same proxy file. This saves geometry cache space, as a single proxy file is only loaded once if different proxies reference the same file.

There is no need for this anymore with the latest builds; V-Ray will do this internally. The advantage is that it works even if the proxies are X-Ref'd in different files.

Best regards,
Vlado

[Dave_Wortley]

The Shell Material is quite useful for having lower resolution textures in your viewport than in your render. Takes a bit of organising and batch processing of textures to make low resolution versions but gives you a lot of control.

[Dave_Wortley]

Also occasionally when you are changing your textures and re-loading or replacing textures there's often still textures in the memory.

Using the maxscript lines:

freescenebitmaps()

and

gc()

Can free up a load of memory!

Especially good when using downloaded scripts and the author has memory leaks in their scripts.

[super gnu]

doesnt nitrous handle showing lower res textures in the viewport already? for me the pain (until 2014) was getting it to show the textures at a half decent res at all.

also, although i dont use it, i believe the bitmap pager will generate low res proxy textures for you. im not sure if this plays happily with vray though.

+1 for gc().. in the past you had to use it repeatedly while attaching lots of meshes together as there was a memory leak there and youd run out of ram. think this is fixed now though.

[vlado]

also, although i dont use it, i believe the bitmap pager will generate low res proxy textures for you. im not sure if this plays happily with vray though.

It actually does; not with VRayHDRI obviously, but for regular bitmaps it works.

Best regards,
Vlado

[Dave_Wortley]

Bitmap Pager can cause a lot of headaches! Shouldn't be necessary to use it with modern computers, maybe 10 years ago with 32bit and 2gb of ram yes, but with SSDs, 100Gb network and VrayHDRIs I wouldn't bother.

Fills up your render blades with files too.