Distributed Rendering - speed issue

[MannaTheBerserk]

Hi everybody!

I am experiencing a rendering speed issue with the distributed rendering, here where I work.

What's happening is that the render goes fairly quickly (15 or 16 minutes) up to 98%, then it stays there for 30 minutes, sometimes more.

This makes the distributed rendering pointless at the moment, given a local render takes about 47 minutes to complete one frame.

Our animations scenes are between 80 to 200 frames long, each.

It seems to happen only during the batch render, not when I render in the VFB (one frame).

So, I guess something wrong is happening either on our network side, or the V-Ray side. We are really not sure.

Did some of you went through something similar?
What are the best practices for an effective Distributed Rendering set up in V-Ray for Maya? Maybe we are missing something.


Thank you in advance for any hint!


Mauro

[waaazoo]

you have probably some hanging buckets. Are you rendering in bucket mode ?
If you do distributed rendering, there is no dynamic splitting of buckets at the end (splitting up to 1 pixel). If you render animation, I would highly suggest rendering without distributed (1 frame per node), and via vrscene Vray standalone. It is much more effective and you have the ability to skip already existing frames, which you can't with mayabatch.

[MannaTheBerserk]

you have probably some hanging buckets. Are you rendering in bucket mode ?
If you do distributed rendering, there is no dynamic splitting of buckets at the end (splitting up to 1 pixel). If you render animation, I would highly suggest rendering without distributed (1 frame per node), and via vrscene Vray standalone. It is much more effective and you have the ability to skip already existing frames, which you can't with mayabatch.

I have never had any problems like this before. Yes, buckets hanging sometimes, but not this time.

In fact, quite the opposite.

Another thing that happens: when the other render machines finish their area, for some reason they just leave the rendering process to my local machine to finish, without keeping the rendering going on their side. Every time this happens, my machine CPU utilization drops to something like 12% or less, which is totally not right.

In previous versions of V-Ray, and I am talking about 6 years ago, this had never been an issue.

Rendering without distributed isn't an option when you have 200 frames and only few machines to render with. Also it would totally defeat the purpose and benefit of DR.

When I render in Autodesk VRED, with its cluster rendering (which is a DR by the way), nothing like this happens.


Thanks for the tips though, much appreciated.


Mauro

[SonyBoy]

A bit confused by several points in your posts, but when you say "Rendering without distributed isn't an option when you have 200 frames and only few machines to render with. Also it would totally defeat the purpose and benefit of DR" you seem to be misunderstanding the point of DR, which is fundamentally to distribute very high resolution (and therefore usually slow and expensive) image renders across multiple machines. Since you've already said that you're rendering animation, I'm not sure why you're using DR at all -- especially since it's causing you issues. Is there some particular reason you're not using standard rendering techniques with your render farm, that is simply queueing the frame sequence and letting the render nodes pick up one or more frames each exclusively? This is the normal way to render animation.

[SonyBoy]

Or perhaps you don't have a render farm set up or render management software (it's always a bit unclear when people talk about DR without making this clear, since it can be utilized directly through Vray or through software such as Deadline). If you're using this approach just because you only have a handful of boxes and no proper render farm, I'd save yourself a headache and just launch separate sequences on each machine, by breaking up the shots into chunks (machine A, frames 1-50, machine B frames 51-100 etc.). Or another way to do it is to launch the full sequence on every machine but render "By Frame" depending on how many machines you have, sequencing the start frames on each box -- this way you get your full sequence rendered from the beginning.

[chuckie7413]

I am on the same page as SonyBoy. Your comment is about the purpose of distributed rendering with animation seems a bit misguided. I am not sure how many rendering machines you have that you are distributed a single frame across but let's have a guess based on the render times you provided and give an example. 1 frame takes 47 minutes for you on a single machine. Distributed 1 frame across all your machines takes 16 minutes. Do you have 3 machines?

Let's say you have 3 machines and 300 frames to render (to make the example maths easier) and plan to render 1 frame per machine using a render manager like Deadline (which is now completely free)
If each machine renders 100 frames and each frame takes 47 minutes that would mean you would get all 300 renders done in 4,700 minutes (78.3 hours)
If you were to do distributed instead (with everything running with no issues and it takes 15-16 mins per frame) it would take between 4,500-4,800 minutes. So let's say the same amount of time.

So rendering a frame per machine is no worse and more stable because you do not have to worry that if a machine goes down during a render it ruins your whole frame, or even the actual issue you are running into with buckets lingering.

But the way you wrote your messages does indeed make it sounds like you running the render through Maya and not via a render manager like Deadline.

Your issues with distributed rendering are still valid though and that is something you should submit a direct support request with Chaos for.

Best,

Richard

[MannaTheBerserk]

Sorry for the late reply.

So, we do not have a dedicated render farm, otherwise we would be using solutions like Deadline. I should have pointed that out immediately.

For that reason, the batch render is running through Maya from the client machine (my workstation in this case).

I know how distributed rendering works and what the purpose is in those situations.

That said, I wanted to post an update regarding this "issue".

For the time being I have managed to speed up the process by reducing the buckets size from 64, which is default, to 24. This shaved the render times from 47 minutes to 11 minutes per frame. The buckets lingering at the end are still, well, lingering (spending a lot of time on certain areas), but at least they are taking a lot less time to solve their render part.

Thank you both for the time you spent helping me out guys!

Best regards,

Mauro

[waaazoo]

Sorry to insist again, but from what you are experiencing, I highly recommend that you change your setup.
I don't have either a big renderfarm nor a lot of nodes. This means you don't need to have a "renderfarm setup".
Running Mayabatch from Maya interface is already wrong for me. Rather use mayabatch from the command line window and create a batch file "myrender.bat"

"C:\Program Files\Autodesk\Maya2020\bin\render.exe" -r vray -proj X:\projects\myproject -cam mycamera -s 1 -e 1200 -b 1 -x 1920 -y 1080 -im myimages X:\projects\myproject\scenes\myscene.mb
-> by using mayabatch you will have to setup a batch file per node: for example node01 -> frame 1 to 200, node02 -> frame 201 to 400, etc...

That said, it's much better to use Vray standalone batch by exporting a vrscene, and create again a batch file:

vray -sceneFile="X:\projects\myproject\myscene.vrscene" -imgFile="X:\projects\myproject\images\myimages.exr " -distributed=0 -imgWidth=1920 -imgHeight=1080 -skipExistingFrames=1 -display=1 -autoClose=1 -frames=1-1200,1

In the batch text file, you can add several lines of processes. Once the first is done, it goes to the next. The good point with Vraystandalone is the "-skipExistingFrames=1" (it's obvious), and as you are not in distributed, you will not have hanging buckets.
Launch this same batch on all your available nodes, and it will work to get your animation job done.
If one node crashes, the other continue. If windows decides to update (oh damn, you forgot to delay the automatic updates!), the other nodes continue.
And, to speed up a little further, you can even deactivate the display...

[MannaTheBerserk]

Sorry to insist again, but from what you are experiencing, I highly recommend that you change your setup.
I don't have either a big renderfarm nor a lot of nodes. This means you don't need to have a "renderfarm setup".
Running Mayabatch from Maya interface is already wrong for me. Rather use mayabatch from the command line window and create a batch file "myrender.bat"

"C:\Program Files\Autodesk\Maya2020\bin\render.exe" -r vray -proj X:\projects\myproject -cam mycamera -s 1 -e 1200 -b 1 -x 1920 -y 1080 -im myimages X:\projects\myproject\scenes\myscene.mb
-> by using mayabatch you will have to setup a batch file per node: for example node01 -> frame 1 to 200, node02 -> frame 201 to 400, etc...

That said, it's much better to use Vray standalone batch by exporting a vrscene, and create again a batch file:

vray -sceneFile="X:\projects\myproject\myscene.vrscene" -imgFile="X:\projects\myproject\images\myimages.exr " -distributed=0 -imgWidth=1920 -imgHeight=1080 -skipExistingFrames=1 -display=1 -autoClose=1 -frames=1-1200,1

In the batch text file, you can add several lines of processes. Once the first is done, it goes to the next. The good point with Vraystandalone is the "-skipExistingFrames=1" (it's obvious), and as you are not in distributed, you will not have hanging buckets.
Launch this same batch on all your available nodes, and it will work to get your animation job done.
If one node crashes, the other continue. If windows decides to update (oh damn, you forgot to delay the automatic updates!), the other nodes continue.
And, to speed up a little further, you can even deactivate the display...

Hi Waaazoo,

I use the command line to launch my render scenes. Though all the test renders (low-res of course) I just go for the command within Maya render menu.

I haven't used the V-Ray standalone batch though. I will try with the v-ray scene and see how it goes.

Thanks for the help!

Mauro

[chuckie7413]

Hi Mauro,

Make sure to submit a support ticket to chaos about your distributed bucket issues.

And then just download and use Deadline for your rendering jobs. IT IS FREE!

There always a bit of setup pain with it but honestly it will be so much easier to manage and monitor your render jobs once setup.