Render slaves are producing black frames 3.5.04

[martinium]

Hello lovey people,

We have some problems after upgrading to Vray 3.5 (both 3.5.03 and 3.5.04). Some of the scenes we are currently working on, produce multiple random black frames without showing any errors on deadline logs or without any sign of logic or order behind it.
Some random frames are being rendered black with no alpha and for about 50 seconds to 2 mins on scene that should render a frame for 1-2 hours a frame. The Slaves producing these black frames are different every time and on a different frames each time, so it doesn't look like a machine issue (software or hardware). Also we started noticing these after our upgrade to 3.5 so I believe it could be version related (unless it's our stupidity with messing things up of course ).

At first we had this behavior on only one-two scenes, but now it is happening on other shots as well. I did some tests with different settings, but was not able to find what the issue is.
I've attached a pics that shows a test job set on progressive to render for 10 mins and you can see the selected frames that are black with rendertimes between 30ish seconds to 2 mins instead of 10 mins. The logs shows that on the normal frames it is spending some time on calculating light cache and on the black frames - the log is just stopping there and skipping to rendering and then frame completed and writes the output...

Is it possible for you guys to have a look? Maybe I can upload a scene?


Many thanks and once more congrats for the Oscar Vlado and the whole team! Whoooooo!

Martin

[tashko.zashev]

Hi,

We have some reports about Black render frames on cases where the hardware acceleration for Denoising or lens effects calculations fails. Do you use any of those effects? If yes, please run a test render with hardware acceleration Off.

[martinium]

We don't use them, however on one of the scenes I've been testing the new lens flare which is in the VFB, however on the scenes set to render on the farm, both lens effects and denoising are off (there is no denoiser RE).

[tashko.zashev]

We don't use them, however on one of the scenes I've been testing the new lens flare which is in the VFB, however on the scenes set to render on the farm, both lens effects and denoising are off (there is no denoiser RE).

Allright then, it would be best to test your scene here and see if the issue is reproducible. Do you have access to our FTP ? If not, please drop an email to support@chaosgroup.com including a link to this thread.

And one more question, how do you submit the render job? using the Deadline script in Max or directly from the Deadline monitor ?

"Many thanks and once more congrats for the Oscar Vlado and the whole team! Whoooooo!"
I forgot to say thank you from the whole Chaos team!

[martinium]

Hi Tashko,

Great! I think we have access to your ftp I will check with our producer and write you again.

We are using deadline submission script from 3ds max.

There is one update however. For now, when the job is submitted with Vray frame buffer on (we usually turn it off when sending jobs on the farm so we can have the render elements split out in a separate folders) doesn't have black frames. So it looks to be happening only when rendering with vray frame buffer is off.

I will prepare the scene and upload it. Thanks!

Cheers,
Martin

[tashko.zashev]

OK, thank you.

[martinium]

The scene is sent!

Thanks,
Martin

[tashko.zashev]

The scene is sent!

Thanks,
Martin

Do you have a ticket number ?
I will need the name of the file you've uploaded on our FTP.

[martinium]

Hi,
Yes sorry - "forChaosGroupInspection.rar"

Contains max file and all the textures, ies lights etc.

Martin

[martinium]

Hi again,

I can confirm that the problem appears only when vray VFB is off. I think this can maybe help narrow the the area where to look for.

Also, it has happened on local rendering, not only on the farm by renderslaves. My local machine started the rendering process and the rendering just ended after loading textures and mesh process before starting the actual rendering, again without any error or warning.

For now we are rendering everything with Vray VFB on on the farm, if we notice the scene has black frames on it.

Cheers,
Martin

[moso]

Did you find the cause for this?
We are having almost the same problem... and in our case we are using the VFB with denoiser... the strange thing for us, is that a render node does the first assigned frame properly, and saves it, and then in the next it goes through the process, finishes the render, does denoising, but it only saves a file that is 12Kb in size... so is very misleading as deadline thinks is correct... and the first frame for each render node is correct. Is also worth mentioning that we have render many times like this without issues.. (using denoiser and VFB on the network) so is very hard to avoid or predict... we will keep on looking but any insight would be great.

M.

[moso]

OK Hoping this can save someone some headaches we found our problem.
Somehow a duplicated render element had crawled into the list (so there were two raw reflections elements -with same name). We tested and every time we had two duplicated render elements there will be an error writing the EXR(this can even be replicated locally).. but this only happens when denoiser is also included as a render element on the list.


Hopefully Vray can implement a fix later so that if that mistake of adding to of same render element one is omitted but the render does not fail to write the EXR.


M.

[martinium]

Hi guys,

thanks for the suggestions, but unfortunately it is neither one. We can't have duplicate Render Elements names since Deadline Submitter is performing a vanity check for that, and we can't send a job to the farm unless we have made all the duplicate RE names unique!
Also, it happens on scenes that don't have denoiser RE and we are not denoising them at all. This happens on many scenes here. The guys from Chaos group found some very weird issue with changing the render region (although we have it off) before starting the render, being outside of the image resolution boundary. Not sure how that happens on our scenes (could be caused by our pipeline scripts or something but not sure...).

Anyhow, there should be a solution to this soon. For now, the only way to solve this is by turning ON the Vray Frame Buffer and Separate Render Channels. That VFB doesn't render black frames but also doesn't sort the REs in separate folders and the comp department complains a bit

Hope that helps.

Cheers,
Martin

[martinium]

Hi again,

Seems like we've found what is causing this issue with the empty frames - or black frames saved without any error complains.

Before sending the render job on the farm, make sure to TURN OFF the VFB render region, because even if the "Enable build-in frame buffer" is off, leaving the region rendering as shown in the image - is causing some of our scenes to render black frames randomly as I've described in the beginning of the topic!

Cheers,
Martin