Batch render crashing randomly. (light cache related?)

[B037973a]

Hi there,

I have had an issue over the past few days with a batch render that keeps randomly stopping. The render in total is a few thousand frames, and we stopped it on the evenings and started again the next morning. During this past week, it has stopped by itself/crashed around 10 times. I can restart the render, and it does that frame fine, so it doesn't really seem to be an issue that specific frame.

I have looked in the mayaRenderLog.txt file, and every time it stops it's during the 'Building GPU light cache' stage. Here is instance of this:

[2024/Mar/14|21:15:23] V-Ray: Updating frame at time 2222.00

[2024/Mar/14|21:15:23] V-Ray: Updating frame at time 2222.00 (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray: Total time updating frame  0h  0m  0.0s (0.0 s)

[2024/Mar/14|21:15:23] V-Ray: Preparing scene for frame...

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 10 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 20 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 30 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 40 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 50 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 60 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 70 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 80 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 90 % completed.

[2024/Mar/14|21:15:23] V-Ray: Preparing scene for frame... (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray: Compiling geometry...

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 10 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 20 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 30 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 40 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 50 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 60 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 70 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 80 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 90 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 100 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 0 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 10 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 20 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 30 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 40 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 50 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 60 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 70 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 80 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 90 % completed.

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 100 % completed.

[2024/Mar/14|21:15:23] V-Ray: Compiling geometry... (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray:   Using Embree ray tracing.

[2024/Mar/14|21:15:23] V-Ray:   Building Embree static accelerator...

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.1s): 10 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.1s): 20 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 30 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 40 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 50 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 60 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 70 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 80 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 90 % completed.

[2024/Mar/14|21:15:23] V-Ray:      (Estimated time remaining:  0h  0m  0.0s): 100 % completed.

[2024/Mar/14|21:15:23] V-Ray:   Building Embree static accelerator... (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray:   Building Embree static trees took 27 milliseconds, memory used 78.13 MB

[2024/Mar/14|21:15:23] V-Ray:   Scanning scene for light plugins.

[2024/Mar/14|21:15:23] V-Ray:   9 lights found.

[2024/Mar/14|21:15:23] V-Ray:   Using adaptive light evaluation with 8 lights.

[2024/Mar/14|21:15:23] V-Ray:   Using 2.21 MB for adaptive lights buffer
[2024/Mar/14|21:15:23] V-Ray: Initializing CUDA renderer...

[2024/Mar/14|21:15:23] V-Ray: GPU render kernel type: 2

[2024/Mar/14|21:15:23] V-Ray: Number of CUDA devices: 1

[2024/Mar/14|21:15:23] V-Ray: nVidia driver version: 546.01

[2024/Mar/14|21:15:23] V-Ray: Initializing environment kernel

[2024/Mar/14|21:15:23] V-Ray: Transferring static geometry tree with 758294 faces to CUDA devices...

[2024/Mar/14|21:15:23] V-Ray: Building dynamic meshes trees

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 11 % completed.

[2024/Mar/14|21:15:23] V-Ray:    (Estimated time remaining:  0h  0m  0.0s): 77 % completed.

[2024/Mar/14|21:15:23] [2024/Mar/14|21:15:23] V-RayV-Ray: :  (Estimated time remaining:  0h  0m  0.0s): 100 % completed.

Building dynamic meshes trees (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray: Building trees for 9 meshes with all threads per mesh took 0ms.

[2024/Mar/14|21:15:23] V-Ray: Total number of dynamic triangles: 100.

[2024/Mar/14|21:15:23] V-Ray: BUILDING Embree dynamic geometry tree for GPU

[2024/Mar/14|21:15:23] V-Ray: Embree dynamic geometry tree contains 9 primitives.

[2024/Mar/14|21:15:23] V-Ray: Embree dynamic geometry tree built in 0.02 ms.

[2024/Mar/14|21:15:23] V-Ray: Embree BOX tree statistics:

[2024/Mar/14|21:15:23] V-Ray:   Primitives count: 10 (Mem usage:0.00 MB)

[2024/Mar/14|21:15:23] V-Ray:   Meshes count: 9 (Mem usage:0.00 MB)

[2024/Mar/14|21:15:23] V-Ray:   Number of tree nodes: 17

[2024/Mar/14|21:15:23] V-Ray:   Tree takes 0.00 MB. Build time 0.23 ms.

[2024/Mar/14|21:15:23] V-Ray: Uploading GPU nodes buffers...

[2024/Mar/14|21:15:23] V-Ray: Uploading GPU nodes buffers... (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray: Building GPU dynamic nodes...

[2024/Mar/14|21:15:23] [2024/Mar/14|21:15:23] V-RayV-Ray: :  (Estimated time remaining:  0h  0m  0.0s): 10 % completed.

Building GPU dynamic nodes... (finished in [ 0h  0m  0.0s])

[2024/Mar/14|21:15:23] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 90 % completed.

[2024/Mar/14|21:15:23] V-Ray: Scene contains 8 lights

[2024/Mar/14|21:15:23] V-Ray: Enabled Light Tree on GPU

[2024/Mar/14|21:15:23] V-Ray: Building optimized CUDA kernel for devices...

[2024/Mar/14|21:15:23] V-Ray: Compiling kernels

[2024/Mar/14|21:15:23] V-Ray: Optimized CUDA module 1 for device NVIDIA GeForce RTX 4090 compiled successfully

[2024/Mar/14|21:15:23] V-Ray: Optimized CUDA module 0 for device NVIDIA GeForce RTX 4090 compiled successfully

[2024/Mar/14|21:15:23] V-Ray: Compiling kernels (finished in [ 0h  0m  0.1s])

[2024/Mar/14|21:15:23] V-Ray: Maximum kernel work group size: 1024

[2024/Mar/14|21:15:23] V-Ray: Using adaptive lights ver2

[2024/Mar/14|21:15:23] V-Ray: Maximum kernel work group size: 1024

[2024/Mar/14|21:15:23] V-Ray: Device[0]: NVIDIA GeForce RTX 4090 selected for GPU light cache calculations.

[2024/Mar/14|21:15:23] V-Ray: Building GPU light cache.

[2024/Mar/14|21:15:23] V-Ray: Tracing 1000000 image samples for light cache in 64 passes.

[2024/Mar/14|21:15:23] V-Ray: Using 63.53 MB for light cache preview buffer
[2024/Mar/14|21:15:23] V-Ray: Building GPU light cache...

Stack trace:

  rt_cuda.dll!<7FFC2F896322>

  rt_cuda.dll!<7FFC2F89BA5B>

  rt_cuda.dll!<7FFC2F898910>

  vfb2_qt.dll!deleteVFB2QTUI

  Qt5Core.dll!QThread::start

  KERNEL32.DLL!BaseThreadInitThunk

  ntdll.dll!RtlUserThreadStart

00007FFC89539E10Result: //themaneframe/Colonel Duck Productions/4) CDP Brand/2) Assets/2) Showreels/16) 2024 - Fuel Cell/2) Animation Files/1) Maya/scenes/FuelCell_Render_01__23084.ma

Fatal Error. Attempting to save in C:/Users/colon/AppData/Local/Temp/FuelCell_Render_01__23084[Recovered-colon.2024-03-14-21.15].ma

// Maya exited with status 1
​

My system setup is the following:
Maya Version: Autodesk MAYA2024.2
V-Ray Version: V-Ray 6 for Maya, update 2

Computer:
Microsoft Windows 11 Pro
ASUS
13th Gen Intel(R) Core(TM) i9-13900K, 3000 Mhz, 24 Core(s), 32 Logical Processor(s)
NVIDIA GeForce RTX 4090
NVIDIA Studio Driver: 546.01
64.0 GB RAM

Thanks for any help with this, and please let me know if there is any more information you might need.

[vladimir_krastev]

Hi, thanks for posting.
Check and make sure you are not running out of memory. Monitor your hardware around the time the crash happens.
One way to do this is to record your screen if you don't know exactly when it will crash. Leave open your VFB and a resource monitoring tool (please use something more advanced than the Windows Task Manager).
Look for CPU and GPU temps and memory consumption.

crashed around 10 times. I can restart the render, and it does that frame fine, so it doesn't really seem to be an issue that specific frame.

This still could be an issue with the GPU memory. Could you archive and send us the scene for review? You can do that here or trough the support contact form.

[B037973a]

Hi Vladimir,

Thank you for your reply. I'll do the screen recording, as you've mentioned, to check to see if I'm running out of memory on our next batch render.

I can also see there is an option within the 'Render settings' for 'Memory Tracking'. I've not previously noticed this option, and I'm unsure if it'll be helpful, however I'll still enable it during that render.

In regards to sending you an achieved scene, is there a specific email I should include within the 'CC' option to ensure it's directed to you?

Thanks,
Nathan

[vladimir_krastev]

Hi, please use this link if you want the ticket sent directly to me. Other ways of getting the ticket directly to me is pasting a link to this thread in you message or pasting the ticket number here.

I can also see there is an option within the 'Render settings' for 'Memory Tracking'

This is for CPU rendering. Even if we had such functionality for GPU it is best if we could monitor the whole process externally. And also hardware performance etc.

[B037973a]

Hi,

Thanks for that information. I have used the link you supplied to upload the achieved scene.

The ticket ID is: #223134