Batch GPU render will only render 1 frame

[rimcrazy]

I'm attempting to render an animation using GPU rendering. I'm using the latest version of Vray for Maya. I'm using the latest version of Maya 2017 update 3. It renders 1 frame perfectly but then quits. I don't believe I have anything that is preventing the render from running correctly as it runs fine in the Vray Framebuffer.

I have attached the log file from the latest run.

[Morbid Angel]

According to the log gpu runs out of ram. I've actually run an animation in batch on my 1080 ti, generally on any simple scene it uses around 4-6 gb. How much gpu ram do you have?

[rimcrazy]

Hmm... interesting. My cards are Asus GTX 1080's with 8Gb of ram each. These are not complicated scenes, few lights and very few low density polygon objects. I'm surprised it ran out. I will have to run some tests.

I ran this in Maya GUI mode, ie, I had Maya open and just started a Batch Render from the menu. I'm sure it can be done but I'm guessing if I start from a Windows command shell I'd use less memory than running from the GUI. I'll go RTFM and figure out how to start from a shell and see if that makes any difference. Short of killing some lights I'm not sure how I can simplify my scene much more. Thanks for the feedback and support.

[Morbid Angel]

Do you have a gpu meter? or some way to see how much ram is being used during rendering. It could be that you have a few textures that are large, perhaps resizing them down could help. Also, heavy geometry count might also require some ram. Its funny but 8 gb, or even 12 gb (what my card has) is barely enough to do what I usually do. All my systems have 48...that seems to be enough

[rimcrazy]

I don't normally run them as they can be a resource pig but I'll install one to look at things. Wow, you have 48Gb of video ram? Holy pasta fazoli!! Ok, my system has 32Gb of system ram and 16Gb of video ram. I just re-ran things with some lights removed. What I'm not understand, if I"m reading the log correct, is the light cache's are not that big but they seem to be what is blowing out the scene. This one has 2 less lights than the first but it is still not making it. If you have 48Gb of Vram and you have issues I may as well bag it for production at least at 4k resolution. I've not looked yet but I assume that if I reduced my resolution from 4k down to say 1080p it all might work. That is a 4x reduction in pixels.

What is maybe confusing me the most here is these renders run in the VFB perfect. No issue. I kick off a VFB render of a frame at 4k and it fly's like a banshee and finishes in no time. All I'm doing different, from my obviously dumb perspective, is asking Vray to do this for a number of frames in a row. That obviously appears to be very different so I'm missing something.

[2017/Apr/25|07:12:54] V-Ray: Merging light cache passes...
[2017/Apr/25|07:12:54] V-Ray: Light cache contains 14841 samples.
[2017/Apr/25|07:12:54] V-Ray: Light cache takes 6.6 MB.
[2017/Apr/25|07:12:54] V-Ray: Prefiltering light cache...
[2017/Apr/25|07:12:54] V-Ray: Average rays per light cache sample: 39.59 (min 1, max 271)
[2017/Apr/25|07:12:54] V-Ray: Using premultiplied light cache.
Failed to allocate memory for buffer
./src/lightcache_gpu.cpp(347) : CUDA error 2 : unable to allocate enough memory to perform the requested operation (out-of-mem)
Failed to allocate memory for buffer
./src/lightcache_gpu.cpp(337) : CUDA error 2 : unable to allocate enough memory to perform the requested operation (out-of-mem)
Failed to allocate memory for buffer
./src/lightcache_gpu.cpp(347) : CUDA error 2 : unable to allocate enough memory to perform the requested operation (out-of-mem)
[2017/Apr/25|07:12:54] V-Ray: Lightmap successfuly loaded.
[2017/Apr/25|07:12:54] V-Ray: GPU light cache: 14841 samples stored.
[2017/Apr/25|07:12:54] V-Ray: GPU light cache: 2634 nodes stored.
[2017/Apr/25|07:12:54] V-Ray: GPU light cache: 2635 leaves stored.
[2017/Apr/25|07:12:54] V-Ray: Mtls 7.07MB, Geo 6154.42MB, Lights 0.00MB, LightCache 1.03MB, Other 65.46MB (data index 0)
[2017/Apr/25|07:12:54] V-Ray: (Estimated time remaining: 0h 0m 58.8s): 99 % completed
[2017/Apr/25|07:12:54] V-Ray: Building optimized CUDA kernel (0) for comp cap 6.1 for device 0...
[2017/Apr/25|07:12:54] V-Ray: PTX file generated with CUDA Toolkit v7.5 for CUDA compute capability 2.0. CUDA runtime version 7050.
[2017/Apr/25|07:12:55] V-Ray: Optimized CUDA kernel assembled successfully
[2017/Apr/25|07:12:55] V-Ray: Total memory for device 0 : 8192 MB, free 0 MB
[2017/Apr/25|07:12:55] V-Ray: Building optimized CUDA kernel (0) for comp cap 6.1 for device 1...
[2017/Apr/25|07:12:55] V-Ray: PTX file generated with CUDA Toolkit v7.5 for CUDA compute capability 2.0. CUDA runtime version 7050.
[2017/Apr/25|07:12:56] V-Ray: Optimized CUDA kernel assembled successfully
[2017/Apr/25|07:12:56] V-Ray: Total memory for device 1 : 8192 MB, free 0 MB
[2017/Apr/25|07:12:56] V-Ray error: ./src/ocl_tracedevice.cpp(3254) : CUDA error 2 : unable to allocate enough memory to perform the requested operation (out-of-mem)
[2017/Apr/25|07:12:56] V-Ray error: ./src/ocl_tracedevice.cpp(3254) : CUDA error 2 : unable to allocate enough memory to perform the requested operation (out-of-mem)

[Morbid Angel]

Sorry I wasn't clear. I have 48 gb of system ram, not vRam. If you are rendering 4k it most likely might be the issue since large frame buffer requires a lot of memory on its own.

[rimcrazy]

Here are some screenshots with GPU meters running

So this is really what I'm not understanding. Below are two screenshots with CPU/GPU meters running. The first screenshot is for a 1920x1080 GPU render using my machine. The two meters are for the two cards. The card on the right is both a GPU compute node and the main display GPU. The card on the left is just an additional GPU. These are configured correctly for GPU rendering (ie there is no SLI connector). As you can the memory useage is as follows

1920x1080
GPU 1 = 2243Mb
GPU 2 = 1017Mb

4096x2304
GPU 1 = 2847Mb
GPU 2 = 1581Mb

These are all well below the 8Gb limit on each card, so why in batch render mode are they blowing up?

[Morbid Angel]

Sounds like some other issue then, perhaps a bug.

[savage309]

It seems like it tries to load some huge memory buffers for geometry. Can you share the scene ?

Best,
Blago.

[rimcrazy]

I can send it to you. I don't want to post it on a public form, how may I get it to you? It is a little over a 4Mb zip file.

Just sent you a PM with a link to the zip file of the scene.

[savage309]

I can send it to you. I don't want to post it on a public form, how may I get it to you? It is a little over a 4Mb zip file.

Just sent you a PM with a link to the zip file of the scene.

Thanks! Will check and get back.

Best,
Blago.

[rimcrazy]

For a point of information, the graphs that were posted on my VFB renders were done rendering frame 160 from the camera named RenderCam.

[rimcrazy]

Found out something more this afternoon. This issue appears to only happen when I've opened up Maya in GUI mode. I just kicked off a small command line batch render of 4 test frames, all with the same settings and at a 4K resolution and it appears to be running smoothly. Memory use on both of the graphic cards is 2366Mb and 1580Mb on the two cards which is far below the 8000Mb capacity of each of the cards. This being the case, fixing this for me at the moment is not nearly that critical. I can easily, and actually prefer, to run my production renders in shell command batch mode. It should be probably understood and fixed if the root cause if Vray. If the root cause is me, I'd like to understand what it is that I'm doing with my scene file to cause such issues so that I may prevent that in the future but again, it's not do-or-die at the moment.

UPDATE: Nope... spoke too soon. The problem seems to do with what is being rendered. In rendering frames 140 and above the scene has less polygons and that seems to work fine. The lower number frames less than 100 have a little denser scene and these seem to have issue. I'm not understanding still as the "heavy" scene is only about 450k faces. That is not a dense scene I would think.

[shaio]

Hello,
I was looking at your scene and and saw that you have viewport subdivisions ON for most of the geometry which is quite memory intensive. The scene has about 450k faces but they are all subdivided and those subdivision aren't calculated and shown in the viewport. 32GB of RAM wasn't enough for me to render frame 80 for example. You could try using the OpenSubdiv option from the Attributes->VRay->OpenSubdiv menu on the geometry shape node instead of the Viewport Subdivisions. This way the scene renders the same result (frame 80) with only 12-15 GB of RAM usage based on my tests.

[rimcrazy]

That is a very good catch, I'm not sure I would have caught that. I only thought that VFB renders. I will dig much more into this in the future. Thank you much for your assistance.

UPDATE: I went through my scene and took each object and either did no smoothing because it was not required or used the Vray Attributes on the mesh as suggested and it now renders fine without blowing up. Thanks again for your assistance.