Vray GPU Vram related error VRay / Maya 2020.4

[jonathandigigiant]

Hi There,

Since updating to the latest release of Vray for Maya
V-Ray 5 for Maya, update 2.1
(v5.20.01 from Dec 16 2021)

We are experiencing errors where frames will not render due to the following error,

V-Ray error: [VUtils::VOptix::OptixPipelineWrapper::launch] 7050: OptiX error encountered in vray_optix7.cpp[320]. Error Code 7050

Which seems to be related to VRAM overusage? Unfortunately, I am not able to post the example scene here, but we can reproduce on a scene with a full CAD automotive model, rendering with motion blur. We have turned off displacement, subdivisions, and even texture mipmapping / resizing. Adaptive Lighting is on.
The failed frames only happen on machines with dual 2080TI's but the frames do render successfully on machines with RTX6000 or 3090 cards which have more VRam.

We have tried both bucket and light cache for the GI contribution but still have the same result. The scene is not unusually heavy, with a car driving over a vray Plane with a 4K concrete texture applied. This error has only started happening since we have upgraded to the latest version. I could downgrade, but then we have the issue of light textures not motion blurring correctly.
We have the latest nvidia driver as well as the last one 479 I think which Vray was verified on.


Here is an additional strange result. If we render locally to the VFB, the first time we hit render, we get the error and the render stops. The next time we hit render, the frame renders totally black and with the error. The third time we render, the render will render the full image, but still with the error. So this obviously fails on a farm / batch type render. This is the case on the 2080ti machines.



2022-02-18 19:31:23: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: PTX max register count 128:128

2022-02-18 19:31:23: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: Building optimized CUDA kernel (1) for comp cap 7.5 for device 0...

2022-02-18 19:31:23: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: PTX file generated with CUDA Toolkit v10.0 for CUDA compute capability 3.0.

2022-02-18 19:31:23: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: Compiling kernels for device NVIDIA GeForce RTX 2080 Ti

2022-02-18 19:31:24: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: Optimized CUDA kernel assembled successfully

2022-02-18 19:31:24: 0: STDOUT: [2022/Feb/18|19:31:23] V-Ray: Total memory for device 0 : 11264 MB, free 6273 MB

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Building OptiX top level acceleration structure for device 0...

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Compacting OptiX top level acceleration structure for device 0...

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray warning: The bucket sampler with V-Ray GPU has optimal performance when used with Light Cache.

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Using adaptive lights ver2

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Rendering image...

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Using 6 threads per GPU device

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray: Using 6 threads per GPU device

2022-02-18 19:31:26: 0: STDOUT: [2022/Feb/18|19:31:26] V-Ray warning: Num samples per thread reduced to 245856, rendering might be slower

2022-02-18 19:31:27: 0: WARNING: V-Ray error: [VUtils::VOptix::OptixPipelineWrapper::launch] 7050: OptiX error encountered in vray_optix7.cpp[320]. Error Code 7050


Thank you very much.

[Muhammed_Hamed]

Thanks for the report,

Do you have NVlink enabled on these machines with 2x 2080tis?
Could you please try with the latest stable nightly build from this folder

Other useful advice to better manage GPU memory,

-Enable "Hardware-accelerated GPU scheduling" in Windows 10 specially in the recent Windows builds, this frees up more VRAM on the GPUs
It should be under Graphics Settings, and will require a system restart
https://cdn.discordapp.com/attachmen...mage_66836.jpg
https://cdn.discordapp.com/attachmen...mage_66837.jpg

-Use progressive sampler mode in V-Ray GPU, it uses less VRAM overall, and makes use of V-Ray's smart sampling and scene adaptivity
For rendering animations you can set a maximum render time in minutes per frame, this works well for rendering animations and avoids any stuck buckets near the end of rendering
On average progressive mode will consume 3-4 GB less GPU memory than bucket, on high end GPUs

-Use Cuda mode on the nodes that crash, it consumes less memory than RTX and could avoid those crashes

We are aware of this behavior, newer builds should avoid most of the memory crashes, and we are working on improving memory management on GPU.

Best,
Muhammed

[jonathandigigiant]

Hi Muhammed,

Thank you very much.

The 2X2080TI's are not NV-Linked.

I just tried the latest nightly build for Vray 5 ( screenshot attached.) But we have the same issue.

I did try the progressive renderer and that did work, thank you for that suggestion and we can use that as a workaround for now, however the bucket render'er does seem to render the scene much faster and provide cleaner results.

Thank you very much.

[Muhammed_Hamed]

Thanks for the feedback!
Bucket mode is currently near the top of our priority list.
On another note, Progressive mode is usually faster than bucket for the same noise levels. That is something we will improve.
Using progressive on V-Ray GPU is not a bad practice at all

Best,
Muhammed