GPU crash with Vantage / and LiveLink-connection question

[BeneZ]

Hello,

bug report:
I was loading a huge scene, and the GPU crashed and workstation needed to be started without chance to close applications.
Attached a screenshot taken with smartphone.
Using 2 x gtx1080 ti

Another question:
How do I activate the Livelink?
I opened the scene in Max, I exported a vrscene and I opened the vrscene simultaneously in Vantage.
The LiveLink was shown as active (at least shown in menu of Vantage)

But there wa no connection between 3dsMax (2021 Vray5) and Vantage.
I did not find a description how to connect both.

Thanks

[npg]

Hi

We have a doc page about live linking here: Chaos Vantage

Is this option enabled in your Vantage? It should prevent most crashes, but it’s possible it would still crash.

[BeneZ]

Thanks for the LiveLink page.

The GPU options were already both enabled (I just installed it fresh and ran it as it was)

[charlyxxx]

Hi, dear support.

The same on my side too…

- I installed the latest V-Ray 5 Stable nightly.
- Then opened this scene → https://drive.google.com/file/d/1fvSkVWUz3vKeC09P0k96YgqTJg1sBoyh/view?usp=sharing in 3ds Max.
- Then opened Chaos Vantage.
- Started the Live link and my display was off, then turned on and I saw this crash.

I have 2x2080Ti with Nvlink with Nvidia Studio Drivers 457.30 and when I try to start IPR without Chaos Vantage - everything is OK. And I can open .vrscene without 3ds max in Vantage. What’s wrong with LiveLink? =)

Then I saw another strange issue.

When I started Vantage again - my screen was frozen… Then I tried to restart GPU Driver with WIN+CTRL+SHIFT+B and then again saw this D3D Error.

How to fix this?

Thank you and have a great day
Best regards,
Andrew.

[charlyxxx]

Tested another scene and had the same crash. But fixed it with turned off “Safe GPU”

And had this FPS Speed =)

But the scene working without any crash.

One question - Does Vantage support NVLink?

Thank you and have a great weekend
Best regards,
Andrew.

[npg]

The low framerate is likely worsened by the OS swapping memory back and forth between the GPU and main RAM. That happens when you allow overallocations by disabling the “safe” option.

I think we discussed NVLink in another thread already. We support it as a faster bus for transferring data between the GPUs. It does not double the amount of memory each GPU can use. This would most likely require complicated code changes, it is not simply a magic switch. We will be looking into this issue though.

Thanks for the scene, we will try to reproduce the crash. It sounds like the driver itself crashed or had a TDR timeout, because your screen went blank.

[vladimir.nedev]

What Nikola said, but I have some more questions.

You are most likely running out of GPU memory, which causes the crashes and slow down.

I have 2x2080Ti with Nvlink with Nvidia Studio Drivers 457.30 and when I try to start IPR without Chaos Vantage - everything is OK. And I can open .vrscene without 3ds max in Vantage. What’s wrong with LiveLink? =)

When you start the live link, do you have the “use local host” option enabled, i.e. are you simultaneously running V-Ray GPU inside 3ds Max ?
That will basically double the amount of GPU RAM you need, as both V-Ray GPU and Vantage will have copies of the data.

Other GPU heavy applications running in the background, like 3dsMax, games or other renderers can also eat up the memory available to Vantage.
You might have to run Vantage on a GPU that’s not simultaneously used by 3ds Max in order to have more memory available.

- Then opened this scene → https://drive.google.com/file/d/1fvS…ew?usp=sharing in 3ds Max.

Can you upload the full scene (.max file and textures) and send me a link on my e-mail.
We need all of it to try to reproduce the issue and improve the code that checks for an “out of memory” condition, we might be able to avoid the crash.

Greetings,
Vladimir Nedev

[charlyxxx]

Hi @npg
Thank you for your answers. Yes, I understand, that NVlink doesn’t double GPU RAM in Chaos Vantage, but this will be awesome if you will add this in the future. =)

The low framerate is likely worsened by the OS swapping memory back and forth between the GPU and main RAM. That happens when you allow overallocations by disabling the “safe” option.

So, as I understand, this is like turning on the “Out of Core” in V-ray. Right? =) And this is slow down the FPS =) Because scene eats more free GPU RAM than I have on 1x2080Ti, right? =)

Hi @vladimir.nedev

So, I increase FPS to 20-30 with “use localhost”. Thank you =) In the previous test, it was disabled…

are you simultaneously running V-Ray GPU inside 3ds Max ?

Like as in the tutorial. Selected “distribution render” - on and started IPR.

Other GPU heavy applications running in the background, like 3dsMax, games or other renderers can also eat up the memory available to Vantage.

I just started 3ds max + Vantage. There is no game and etc =)

You might have to run Vantage on a GPU that’s not simultaneously used by 3ds Max in order to have more memory available.

So, your advice is to disable one GPU in V-ray 3ds max? Or how to do that? =) But why, when I don’t use “Safe GPU” option - everything is ok? =)

Can you upload the full scene (.max file and textures) and send me a link on my e-mail.

Yes, no problem. Done =)

We need all of it to try to reproduce the issue and improve the code that checks for an “out of memory” condition, we might be able to avoid the crash.

Ok. I already sent the scene to your email. Please check =)

Thank you and have a great day

Best regards,
Andrew.

[npg]

Yes, it’s a bit similar to “Out of core”, but it’s handled by DirectX and the driver, not our code. The further you go beyond the available budget, the worse it could get for framerate, because it would need to swap more things. And it could crash at some point, so “safe” should be disabled with this in mind and only when necessary.

If you do not disable “use local host” V-Ray will also render the scene and you will both eat up your VRAM and compute cores. Vantage can run on a single GPU, but we haven’t added UI for it yet (a bit more complicated than it sounds). If you run vantage.exe from the console with argument -d 0 it will use only the first device or -d 1 for only the second.

[vladimir.nedev]

So, I increase FPS to 20-30 with “use localhost”. Thank you =) In the previous test, it was disabled…

Wait, that doesn’t sound right, 'Use local host" should be unchecked/disabled .

So, your advice is to disable one GPU in V-ray 3ds max? Or how to do that? =)

I meant to disable one GPU for Vantage. Like Nikola explained, by using the command line options for Vantage.

But why, when I don’t use “Safe GPU” option - everything is ok? =)

Because you are running “out-of-core” in this case.
Also, note that the Windows operating system will tell Vantage what its GPU memory budget is.
When the safe allocations are enabled, Vantage will try to fit within this budget.
There could be more GPU memory available, which the OS has set aside for other apps.
Hovering over the status bar in Vantage will tell you how much additional (marked as available) memory Vantage can use in addition to the memory already used - shown in the status bar.

Ok. I already sent the scene to your email. Please check =)

Thanks. The scene uses 8.08 GBs on my 2080 Ti at 720p resolution.
Not sure what resolution you use, but higher resolutions will use more memory.

The scene has very dense geometry (32 million unique triangles, most visible from the camera) which is part of the reason it’s running relatively slow.

Greetings,
Vladimir Nedev

[charlyxxx]

Hi @vladimir.nedev

These are my comments:

Wait, that doesn’t sound right, 'Use local host" should be unchecked/disabled .

Strange, but at that time I turned on it =))))) Enabled =) Checked =) And FPS was increased =) But now I tested again and yeah, with this option Vray VFB rendering in parallel and this decrease FPS =)

I meant to disable one GPU for Vantage. Like Nikola explained, by using the command-line options for Vantage.

I’m not very familiar with the Vantage command line and I think, If I will disable the 1 GPU - FPS will be decreased twice =) I don’t want this =)

Because you are running “out-of-core” in this case.
Also, note that the Windows operating system will tell Vantage what its GPU memory budget is.
When the safe allocations are enabled, Vantage will try to fit within this budget.
There could be more GPU memory available, which the OS has set aside for other apps.
Hovering over the status bar in Vantage will tell you how much additional (marked as available) memory Vantage can use in addition to the memory already used - shown in the status bar.

So. this means that on large stages that use more than 11 GB - I should turn off it every time. Right? Because I tested 5 scenes and every time I had this crazy crash, with turned off displays, then freeze and restart PC =) Because every time after the freeze, when I restart Driver with WIN+CTRL+SHIFT+B and started Vantage again - it was frozen again =) Until I restarted PC =)

Thanks. The scene uses 8.08 GBs on my 2080 Ti at 720p resolution.
Not sure what resolution you use, but higher resolutions will use more memory.

The scene has very dense geometry (32 million unique triangles, most visible from the camera) which is part of the reason it’s running relatively slow.

Yes. this is a scene from Evermotion =) They create very dense scenes. That is I wanted to test it with my 2x2080TI and NVlink =) haha. Vantage is very fast, even with “Out of Core” vs standard IPR with NVlink =)) ahha. Very soon I will release a tutorial with my tests =) So thank you for this awesome software.

About the resolution. Hmm, I didn’t change it. Just started 3ds max with this scene and Livelink =) So I used 720p too.

Thank you and have a great day
Best regards,
Andrew.

[squintnic]

installed this thinking it would be useful for animatics - but it cant even handle a simple scene (albeit one with lots of proxies and 4-8k texture sets) nothing crazy though
this is on 2 x 2080s with NVlink

i will test this on my 3090 next but have to say unimpressed so far. how are other people loading such massive scenes in like in the feature video?

[squintnic]

ok appear to be much better without live link - that seems to kill the GPU memory available.
looks like live link is not working as expected for me

edit - actually it fails on taking a snapshot of a 4k frame lol…

will have to try it on a simpler scene, still suffers from the same memory issues as vray gpu it seems

[charlyxxx]

@squintnic

ok appear to be much better without live link - that seems to kill the GPU memory available.
looks like live link is not working as expected for me

edit - actually it fails on taking a snapshot of a 4k frame lol…

will have to try it on a simpler scene, still suffers from the same memory issues as vray gpu it seems

Maybe they turn off the “Safe GPU” option? =)

ok appear to be much better without live link - that seems to kill the GPU memory available.
looks like live link is not working as expected for me

Still experimental, as mentioned in the docs =) I noticed the same, that Vantage without LiveLink works better =)

edit - actually it fails on taking a snapshot of a 4k frame lol…

On 3090? Or 2x2080Ti? If 2080Ti, try to disable the “Safe GPU” option. It helped me with the dense scenes.

[vladimir.nedev]

but it cant even handle a simple scene (albeit one with lots of proxies and 4-8k texture sets) nothing crazy though

That doesn’t sound like a simple scene to me.
In any case, Vantage is currently lacking on-demand mipmapping, which means your full textures will get loaded, with 4K textures this could be a lot of memory.
We will be working on this in the future.

this is on 2 x 2080s with NVlink

Like mentioned before, Vantage doesn’t use NVLink to share memory.
We will most likely attempt this in the future, but it will surely lead to much worse performance, so it’s not very high priority.

ok appear to be much better without live link - that seems to kill the GPU memory available.
looks like live link is not working as expected for me

Vantage shouldn’t be using any more memory in “live link” mode.
However, 3ds Max can be using some of the GPU memory (this happens if 3dsMax is just running in the background and you open a .vrscene in Vantage).

edit - actually it fails on taking a snapshot of a 4k frame lol…

We need to improve memory usage with large resolutions.
There is quite a lot we could do there, but didn’t have time for the release.
However, Vantage was designed to have the best interactivity possible, rendering large/print resolutions was never its goal.
V-Ray CPU/GPU will be better at this at least for some time.

Greetings,
Vladimir Nedev

[vladimir.nedev]

So. this means that on large stages that use more than 11 GB - I should turn off it every time. Right? Because I tested 5 scenes and every time I had this crazy crash, with turned off displays, then freeze and restart PC =) Because every time after the freeze, when I restart Driver with WIN+CTRL+SHIFT+B and started Vantage again - it was frozen again =) Until I restarted PC =)

In theory, you should keep the “Safe GPU allocations” enabled to prevent these crashes.
However implementing this option is tricky, so obviously it needs to be improved.

When you have the safe allocations disabled, do you get the crashes ? Do your scenes always render in this case ?

Still experimental, as mentioned in the docs =) I noticed the same, that Vantage without LiveLink works better =)

The “experimental” tag refers to some of the changes in 3ds Max not getting applied in Vantage.
It does not relate to memory usage.

Can you elaborate what works better ? You get less out of memory crashes ?
Do you close 3ds Max when using Vantage with .vrscene files ?

Greetings,
Vladimir Nedev

[vladimir.nedev]

ok appear to be much better without live link - that seems to kill the GPU memory available.
looks like live link is not working as expected for me

Just to mention it again, make sure that “Use local host” is disabled or you will be loading the scene twice - once in Vantage, once in V-Ray GPU.

Greetings,
Vladimir Nedev

[squintnic]

ok thanks for the clarifications - i guess having 10 objects with 4-8k texture sets is not *that* simple, perhaps the demo videos showing complex scenes led me to believe it was more capable with heavy geo and maps
i would say doing a 4k snapshot should not defeat a modern GPU tho.

i was hoping to be able to use Vantage as a way to render out quick and dirty shots for animatics to save me render time and money, but perhaps thats not the best use case for this software.

i feel kind of stuck between a rock and a hard place and i flip flop around looking for new solutions to save me time but generally run into issues with vray GPU based products at the moment.

[vladimir.nedev]

Can you send me this scene ? It will be very useful for testing purpose.

Greetings,
Vladimir Nedev

[charlyxxx]

Hi, @vladimir.nedev

These are my comments.

In theory, you should keep the “Safe GPU allocations” enabled to prevent these crashes.
However implementing this option is tricky, so obviously it needs to be improved.

When you have the safe allocations disabled, do you get the crashes? Do your scenes always render in this case?

No, really no. I tested 5 dense scenes and there is no any crash =) With “Safe GPU …” enabled - every time during import or open vrscene =) Btw, I working on the tutorial and use Bandicam NVenc GPU, and Vantage still working without crashes. So I think you should implement this option more =)