Half rendered frames

[RichardBlank]

Hi, we have experienced a strange problem in the last couple of months. It happens every now and then and we can't find a reason and it's just happening in one project and not in any other projects. Both in maya batch and vray standalone. We think its some kind of network hiccup that forces vray to interrupt the render and save the half rendered frame. But its so strange that its just happening in one of several ongoing projects.

Hers's a link to the deadline log...
https://we.tl/t-etgvxMpDiD

[Zdravko.Keremidchiev]

Have you noticed whether V-Ray renders the whole frame but saves only that part? Or it just stops render there?
Also, does it happen on the same frame or randomly?

[RichardBlank]

No it doesn't render the whole image. And it is random frames.
We think it has something to do with a large number of tiled exrs. I rewrote the exr with another version of the tiled exr app (41202 - 29275). And now the problem seems to be gone.

[Morbid Angel]

Hey RichardBlank, did you notice if the machine run out of ram during that given frame? According to your log it does not finish rendering then tries to save the image. I saw a mix of exr and png textures. You do seem to have motion blurred fur geometry, it can easily require a large amount of ram if the fur is subdivided at render time, so please check and see if that's the case. My advice would be to use .tx tiled tifs for final rendering as its the best option for reducing memory from textures but also do check the subdivision / ram usage.

[RichardBlank]

It doesn't look like the machines go out or ram, our rendernodes have 128gb, and we do not subdivide the fur during render. My thought was that it had something to do with how the tiled exrs are loaded and unloaded during render. And we sometimes experience small network hiccups and if that happens when a tile is read, could it be that vray stops and saves the file? We tried it on a workstation and pulled the network cable and we got the same result.

[Morbid Angel]

Yeah, this sounds like a reasonable concern. I think Chaos group did a fix for a similar issue. I personally had an issue where I've upgraded the switch to a 10 gig capable switch and my server got overloaded and completely lost network connectivity when I ran a test where 30 + machines all at once accessed the server to read a 20 gb file, so perhaps this has something to do with that also.

[RichardBlank]

We have a very similar setup with the 10gb switch and 40 computers reading huge files... I suspected that our nas caused the problems but you think it’s the switch that gets overloaded?

[Zdravko.Keremidchiev]

In this situation, you can take advantage of our resumable rendering which will resume incompleted renders:
https://www.chaosgroup.com/blog/unde...able-rendering
https://docs.chaosgroup.com/display/...able+Rendering

[RichardBlank]

It doesn't work with multi exr's?

[Mihail.Djurev]

From the logs it seems that V-Ray is aborting mid-render for some reason. Normally, if there's an exception, there would be an error message in the log.

I have a strong suspicion that one of the image libraries we use is hitting a max file open limit. Most probably libPNG or OpenEXR.
Could you tell us the number of png textures and the number of EXR textures that you have in the scene?

[RichardBlank]

It's somewhere between 350 to 400 depending on the scene. We have the exact same setup in Max and we don't see this happening there. It's only in maya batch and vray standalone.

[Mihail.Djurev]

Ok, most probably that's not it. It's very strange, since errors and exceptions should normally leave an error in the log. Could Deadline abort the job for some reason?

[RichardBlank]

No I don't think it's deadline that's aborting the job.