Nvlink issues

[pedro_rivera]

Good evening all,

I've been having an issue when rendering a scene with more than 25GB of RAM. I'm using 2x Nvidia A5000 GPUs connected with an nvlink bridge. I activated the sli on the nvidia control panel (Maximize 3D performance). At the moment of the rendering I'm getting an error like this:

Error [V-Ray] [MemoryManagerGpu::setupManagerMemoryInstance] 2: Could not allocate device buffer with size: 5472768 bytes

My motherboard is sli compatible and I'm using 96gb of ram on the system.

I have no idea how to solve this issue. Please help.

[Muhammed_Hamed]

Hi pedro_rivera and welcome to the forums,

Change Filter Level in log window to show all, then you will be able to see if V-Ray GPU recognized your NVlink bridge or not.
Please upload this log here, it will be helpful to us

Best,
Muhammed

[pedro_rivera]

Hello Muhammed,

Thank you for your quick response. I'm attaching the log info inside a note file because it did not let me attach it as a .log format. In addition, I attached the .log in a wetransfer download link (https://we.tl/t-60BGFt01l2). Please let me know if you need anything else.



Pedro

[Muhammed_Hamed]

It seems that NVlink is not working. Could you please list your system specs including windows build?

[pedro_rivera]

Hello, Muhammed-Please let me know if you need any other information.

System Specs:

CPU: AMD Ryzen Threadripper 3970X 32-Core Processor [latest chipset drivers installed]
Motherboard: ROG STRIX TRX40-E GAMING PCIe [4.0 Ready according to the box it supports NVIDIA 2-Way SLI Technology]
RAM: Corsair Dominator Platinum RGB 96GB DDR4 3200MHz [6x16GB]
GPU: 2x NVIDIA A5000 24GB each GPUs [driver version:472.84] I had the latest driver installed [511.09] but Nvidia technical support told me to use the older version because I was getting a message on V-ray log about the driver version being too new.
Storage: 1xhard drive 3TB / 1x Samsung M.2 970EVOlus 500GB / 1xM.2 980PRO 1TB [Windows 10 pro installed on the 980PRO M.2]
CPU Cooler: CORSAIR - iCUE H150i ELITE CAPELLIX CPU Cooler
Power Supply: CORSAIR RMx Series, RM1000x, 1000 Watt, 80+ Gold Certified, Fully Modular Power Supply
Monitors: 2x4k monitors LG connected through Display Port
OS: Windows 10 PRO

[Martin.Minev]

Hi pedro_rivera

We have a current recommended driver for GPU rendering you can see here. The driver check from V-Ray will print in the log file if the driver you have is newer than our current recommended driver.

There are two ways to enable nvlink for Quadro GPUs. The first one is to use SLI and a motherboard that supports it (the one you are trying) and the second option is the use TCC mode. You can find out how to active NVlink for Quadro GPUs using TCC mode here.
Please note that when GPUs are switched to TCC mode you can't have a monitor attached to them. You will need a spare GPU for monitors, so maybe this is not an option for you.
In case the NVlink is not working using SLI you can contact NVidia support so they can check. There are tools online you can use to check if your NVlink bridge is working properly.

Thanks.

[pedro_rivera]

Hello, good morning.

I looked at that particular driver version but it says it's for GTX30 series GPUs. Nvidia A5000 is not listed. Can I install those drivers anyways?

[Martin.Minev]

Hi,

It may work, but It's better to use Quadro drivers. If the current recommended driver is not available for your GPU, please use a newer one. The latest one for your GPU is 511.65 you can make a test with it.
Please make a test using a tool for NVlink (you can find that online easily search for NVlink test). If the tool reports that NVlink is not working, you will need to contact NVidia or your hardware vendor to check.
If NVlink is working, V-Ray will print in the VRay log file - NVLink detected and will use it.

[pedro_rivera]

Hello and thank you for the response,

According to Muhammed above it seems the NVlink is not working according to the V-ray log file I sent on this forum. I will try the nvlink test though. In relation to the nvidia driver, I had driver 511.09 but I was getting this message on the log" Warning:checking CUDA driver version:5001:Installed driver version (511.09) is newer than the latest verified one (497.09)!"

What does that mean?

[pedro_rivera]

I did the NVlink test and according to that the NVlink is working properly on my system.

[pedro_rivera]

Good morning,

We know nvlink is active on my computer. Why V-ray is not recognizing it? How can we solve the issue?

[Martin.Minev]

Hi,

Sorry for the delay.
We will need to make a remote session so we can troubleshoot. We will send you more information on your ticket about how we can proceed.