Not enough VRAM to open projects that previously worked. What happened?

[larsandreas_iversen]

Hello. I have a challenge and have been facing this issue for two months now. Previously, I had no problem running large projects in Vantage, but now the program crashes all the time. I can easily carry out simple projects, such as checking an architectural model, but it takes very little for the project to become too large and crash during startup or rendering. This scene worked without problems before, but now I can't open it: Youtube.

To me, it seems like the PC is no longer able to utilize shared RAM (Shared GPU memory usage), but crashes as soon as I have used up VRAM (Dedicated GPU memory). I have 12 GB of VRAM (RTX 3080ti) and 256 GB of RAM (of which 128 is shared with the GPU). I tried using Unreal Engine instead, but quickly encountered the same problem. I have no prior experience with Unreal, so I'm not sure if the problem lies in hardware, Windows 10, or Vantage.

I have tried:

• 3 different Nvidia Studio drivers
• Vantage versions: 1.7.2, 2.0.0, 2.1.0, 2.2.1, and 2.2.3
• Updating BIOS on the PC.
• Updating and rolling back Windows updates.
• Turning Vantage on and off with Safe Allocations. Turning off has been the only thing that has previously worked for large scenes.
• Changing my PSU.

I have also been experiencing issues for several months where I cannot render with Vray CPU in 3DS Max for longer than approximately 45 minutes. The program crashes after that time. If I quickly stop and start the render again, I have another 45 minutes before the crash. Sometimes I can render for 3 hours without any problem. I have previously discussed this in this thread.

I will be producing several animations in the near future, but I currently lack the tools to proceed. Does anyone have any suggestions for what I can do?

My setup:
CPU: AMD Ryzen Threadripper Pro 3975WX
GPU:Nvidia Geforce RTX 3080 Ti (12 GB VRAM)
Motherboard: Asus ProWS WRX80E-SAGE SE WIFI
RAM: 32 GB Kingston 9965745-020.A00G DDR4 X 8 (Total 256 GB RAM)
PSU: Be Quiet Dark Power 13 1300W
Ethernet connection
Connected to inhouse server (1 server and 2 workstations)

Kind regards
Lars Andreas Iversen
Premani AS (www.premani.no)

[vladimir_krastev]

Hi, thanks for posting. Do you remember when you last opened your scene? We added support for displacement with Vantage 2.2. This could be a reason why your scene consumes more memory.
Check if there is a file named like Vantage_****.dmp in your %TEMP% folder and share it if it exists. Also if possible share the file %APPDATA%/Chaos Group/Vantage/debug_log.txt.
This file gets overwritten every time you start Vantage.

[larsandreas_iversen]

Thank you very much for your response. Even when I rolled back to 1.7.2, I couldn't open projects that I had previously opened. And the few that I could open, I couldn't render. Only navigate around in low resolution.

I wasn't allowed to upload the .dmp file, so I'm sending it to you as you requested via WeTransfer: https://we.tl/t-FQcrW7HFuG

I'm not surprised that the scenes are too large for the memory in my GPU, but I am surprised that I could work with very large files before, but I can't anymore. Previously, I could take a project from 3DS Max that used 100 GB RAM and transfer it to Vantage. Not with live link, but as a Vray Scene, and that would work like a charm.

[npg]

Interesting. I see you've tried a lot of things, this is a whole saga. I would have noted that your cpu was overheating in the first video from the other thread, it was showing max 92* when these processors have Tjmax 90* ( https://en.wikichip.org/wiki/amd/ryz...per/pro_3975wx ). But your second video showed 10 degrees lower. In the case of V-Ray and high CPU load and temperatures, you're also stressing the motherboard VRMs so these are suspect as well (the whole area around the cpu socket is probably super hot. If you can get an IR camera you could try checking that, though there are probably heatsinks getting in the way, etc). But Vantage doesn't stress the CPU once the scene is loaded, so a VRM-related issue is less likely. You've already tried various app versions, OS and driver updates. Your PSU seems solid enough.

One thing I wanted to suggest was already suggested in the other thread but I don't know if you tried it: plug the ram sticks again and run a memtest. You might as well check all the cable connections in the machine. I would also suggest removing half the memory or even go down to 64GB. Lots of sticks have always been less stable than just a couple, even when they come in a kit.

Overall, this is 99% likely a hardware problem because you're getting it with different apps and system updates.

[npg]

Also, if you have somewhere to borrow individual components from, this could help you run a process of elimination, swapping things one by one (cpu, mobo, gpu, ram, maybe even psu).

Oh and yes, 2.2 uses more memory if your scene had displacement the whole time, but you can set tessellation level to "none" in the preferences dialog. This is not related to the original topic because you had issues with 1.7 etc.

Side note: toward the end of that video there was quite a bit of flickering around the house. It looks like increasing light tree samples or turning on reservoir resampling should help, because you have many light sources.

[larsandreas_iversen]

Thank you very much for many insights, Nicola. I will start by testing the RAM modules first.

Some thoughts I have gathered:

• I find it very strange that both of our workstations (Marthe and me) started to fail with the same problem at the same time (CPU - Render crashing after about 45 minutes).
• The cooling surface on the water cooler (Corsair iCue 150i Elite) is somewhat smaller than the massive CPU. The iCue 150i Elite was recommended on AMD's website. When I monitored all cores in a separate monitoring program, no cores stood out with higher temperatures than the others.
• I replaced the PSU for a reason. The previous one I had was 850W. When I calculated it, it was undersized if I ran both the CPU and GPU at full load. The noise from the PC (probably the PSU) has been SIGNIFICANTLY reduced since I replaced the PSU.
• I find it very strange that I crash 3DS Max and Vray after about 45 minutes of rendering in most cases, if I render only one image. However, if I render an animation with Vray in 3DS Max with 300 images and let the PC work all night, it works perfectly fine. I would have thought that the latter case would stress the PC the most.

[npg]

The dump files show a crash when allocating memory for mesh lights. I can't see the number of triangles. Maybe the memory was already exhausted by this point. The log file also shows memory allocation failures toward the end of the scene loading process, but the application was later closed normally, it didn't crash.

[vladimir_krastev]

larsandreas_iversen if you'd like send us your scene for review on a ticket so we can at least test it, check how much memory it takes on the GPU and so on.
Use the support contact form for this and include a link to this thread for better tracking.

[vladimir.nedev]

The "Failed to allocate buffer of size" messages are printed in the log only if you have "Safe GPU allocations" enabled.
If you want Vantage to use "shared GPU memory" i.e. your system RAM, "Safe GPU allocations" should be off.

Can you send the scene to our support, so we can try it too ?

Greetings,
Vladimir Nedev

[larsandreas_iversen]

Thank you very much for the information, guys. I have now submitted a new dump file and log file on a ticket, along with the associated scene. This scene is different from the one I sent the dump file and log file for earlier. I can open and navigate around this one, but as soon as I increase the resolution, attempt to render, or click on one of the set cameras, I receive a message indicating insufficient RAM.

I have now noticed that the scenes I am testing utilize up to 20% of the GPU memory (dedicated and shared) (see attachment). Beyond this, the program crashes.

I ran a memory test (Standard test), which yielded no errors. I have yet to test with half of the memory or memory from another machine. I will try that now.

[npg]

This is 20% of the combined memory, but the onboard memory is 97% full (11.6/12). Even if you have disabled "safe gpu allocations" not everything works by using system memory.

I'm wondering if you have two separate issues: one, the crashes that occur even with V-Ray CPU - these look like a hardware problem and should not be related to the GPU in any way; the other - that your very large scenes fail to load because of insufficient memory. The latter is not a bug, it's expected. The only question is how it worked before and now even old builds don't. Maybe it was luck and barely working...

[larsandreas_iversen]

Apologies for not being clearer about this earlier. These are two separate issues. I have been experiencing crashes in 3DS Max during rendering with Vray for at least six months now. The problem with not being able to open Vantage files has arisen in the last two months. I don't believe there is any direct link between the issues, other than both possibly being related to hardware, although different components.

I find it very strange that I've been fortunate for two years regarding the GPU matter, and suddenly I'm not lucky anymore. As soon as I have the opportunity, I will conduct some tests by swapping out RAM modules. It's so odd that I crash at around 20% utilization every time.

Given that no one else seems to have the same challenges as me, as far as I understand, it undoubtedly must be related to my PC. Regarding the CPU issue, it also affects my colleague's PC. The problems arose simultaneously, which I find very, very strange. There was a Windows update around that time, but I recall that rolling back to the previous version didn't resolve the issue.

All thoughts and opinions are greatly appreciated. These are very expensive PCs to replace, so I hope to narrow down the problems rather than having to purchase new ones.

When working with very large scenes, is the only way to achieve successful results to be economical with resources? Small file sizes in bitmaps, few polygons, etc.? Because it's not possible to have more than 24 GB of VRAM at the moment, right?

[npg]

My suggestions for testing the RAM etc. still stand, but these are about the V-Ray crash and similar crashes in other software. It won't help Vantage if the issue is that the scene is just too large.

I want to stress on this again, the 20% number is misleading. Your GPU memory is near 100% used up when task manager shows this combined usage at 20%. So 20% is not the special number, 100% is the special number here.

GeForce GPUs have up to 24GB, but there are (expensive) workstation GPUs with up to 48GB. If it doesn't hurt scene quality too much, optimizing the assets is always useful. Even if they already fit into memory it will render faster with less geometry and smaller textures.

[larsandreas_iversen]

I understand that I need to optimize my scenes and make my workday a little easier by purchasing RTX 4090.

I will add more information to the thread here if any unexpected results arise during further troubleshooting.

Thank you very much for all the input!

[vladimir_krastev]

Hi, larsandreas_iversen I received your scene on the ticket. We will test it and see how it goes.
Please don't rely on Windows Task Manager to monitor your hardware's performance. Especially for GPU use some dedicated tool instead (for example this one).