Easiest ways to reduce memory usage are:
If you use Embree, enable conserve memory.
Watch your BSP settings. Don’t set face level coefficient too low (though lower is usually faster it uses more RAM), or limit the depth
Always use VRayHDRI. It will only load textures that are seen in the current frame (this is often all of them due to GI bounces or reflection, though.) Regular Bitmap nodes load the entire texture at the start of the frame rendering, even if the object is out of view.
Use tiled TIFF files (.tx). These can be made with maketx (Google it). Pixamoon has a script on script spot that I was working with him on to get maketx support added. That can help automate this. Tiled .tx files must be loaded with VRayHDRI. It will load only the tiles needed (assuming you have texture filtering on, and GI texture filtering on, which you should anyway). The tiles are loaded into a cache global to the scene. The cache size is set once for the scene (in any VRayHDRI mode). If this cache size gets exceeded then tiles will be flushed, presumably first in first out.
Another option is tiled EXR files. These are good for floating point textures, but use double the memory of the 8 bit per channel .tx files. Tied EXRs can be made with Pixamoon’s script, or with the image2tiledexr.exe file installed by VRay (which his script calls). Note that the entire file is loaded into memory if you use a bitmap node rather than a VRayHDRI node. So don’t.
VRayProxies are incredible. If your scene has tons of geometry, or if some dense meshes are only seen for a few frames they can help a lot. Only the portion seen in the render (or reflections or Gi) is loaded. The amount that is cached in memory is controlled by the dynamic memory limit. Smaller render block sizes will tend to load smaller chunks as well (depending on the voxel size). They can be displayed as boxes in the viewport, which is really quite fast.
There is a script on script spot to convert a selection of instances to VRayProxies such that they all reference the same proxy file. This saves geometry cache space, as a single proxy file is only loaded once if different proxies reference the same file.
When using either tiled textures or proxies the sequence order becomes more important. Triangulation is usually pretty good, but just think about what has to be loaded into memory and choose the best order based in that. Or if you have a few slow blocks that tend to cause Last Block Syndrome (where you are waiting a long time for the last block to render) you may want to set the order to start those blocks first (or reduce the block size). Dynamic splitting is good, but it needs to be much more aggressive in my opinion, or really needs an aggression tuning value to subdivide earlier.
Xrefs, either object or scene, help with the scene file size and loading times (and, as Vlado pointed out elsewhere, can be cached for DR rendering).
Putting a delete mesh modifier on complete geometry and setting it to off in Render can speed up your viewport by hiding things that can still render.
Limiting your render elements can save a lot of memory, and don’t forget the old trick to shrink your Max render size to something small and manually set the VFB size.
Dense light caches can take up a ton of memory, often with very little benefit (enable retrace and increase the threshold if you have to).
The Summary Info (bottom of the Max menu) can export a .csv file that you can load into a spreadsheet (Libre Office free calc) and sort to see which objects have the most polys and should be targeted for conversion to proxies.
If you have a ton of small animated objects then the RayFireCache (if you have RayFire) can help a lot with interactive playback (and with retiming).
In most cases Max is MUCH faster, both in the viewport, and rendering, if you have fewer complex objects vs. tons of small objects. So Attach-ing is your friend. In a recent project I attached about 1000 objects into one and saved about 35% on render time per frame, for the same result!
Hope this helps. You can save a ton of memory like this. We employed all these tricks on a recently complex commercial that involved an entirely CG scene with people shot on motion control (and some locked off). Pretty much everything in the scene had to animate, mechanically flipping and spinning and sliding around to change between different environments. So there were a lot of parts and pieces, and a lot of textures, etc. most were only seen for a few seconds. So the tiled textures and proxies helped tremendously.