Can stuck bucket syndrome finally be fixed, please?

[Joelaff]

You can also use very small bucket sizes. They are not as inefficient as you might think (less efficient with larger image filter sizes), and often a smaller bucket size finishes the render faster due to Last Bucket Syndrome (LBS).

Of course we all know the ideal— keep all the cores busy as efficiently as possible and finish the rendering as quickly as possible every time.

[^Lele^]

kosso_olli So, there is no counter test to those images being rendered entirely locally?
That the buckets are stuck at 100% is by no means a guarantee that the data is there and being processed.
While i can see why a bucket may get stuck due to scene setup (say, headlights), when i see stuck buckets in the middle of nothing at all i can't help but wonder about the infrastructure.

[kosso_olli]

Yes, I did counter check some of the scenes. My machine on its own will also get stuck buckets. Usually in areas of high blur, DOF or ray depth.

[^Lele^]

As i said, that part where the recursive computations take longer we think we understand well.
And for those, there is not going to be a "fix", if you want the recursion, the dynamic range, and the convergence : it's the price to be paid, it's down to you to find the best strategies to mitigate it, as other suggested.

I'd like to see a scene like the one hanging in the middle of nowhere.
Until we can see this happen, we won't know if there is in fact anything at all to fix.
Currently, we have precious little (read: nothing at all) which would help us with this, and it's sadly not obvious from a code perspective, either.

[kosso_olli]

I don't have time to prepare any scenes right now. Hit me up in the coming weeks. There are many scenes to debug.

[labulle]

To add my grain of salt to the conversation, we have very different render computers, the very high core count xeons being indeed very slow but compensate with number. We are, of course, plagued with LBS.

Some ideas I would absolutely love to see implemented, as they don't have to be fully automatic and magical :

• an optional and manual timeout to set in the DR dialog (as already suggested) where the buckets are handed to another client or subdivided and reassigned (the mecanism is already here, sort of, as when a client disconnects, the bucket is reassigned after a timeout)
• a possibility of setting a priority per render client (in the client list, set a core performance level, from say 1 to 5 maybe, then for the last 25% of the render hand the buckets to the best performing machines). I'll try to give it more thought, as in my mind at least, it could be a flexible way to have a pretty tailored LBS mitigation system.
• a vray object property to set a render difficulty level (or maybe just a "difficult" tickbox), where the overlapping nodes in the image are handed only to the best performing nodes (from the previous system)
• a fully manual reassign, right click a stuck bucket, click reassign and select the target render node

All of these could be behind an "expert" setting, hidden from view but a godsend.

The only way I handle this for now is by setting locking the bucket starting point and placing it at the worst possible place in my scene. Sometimes I have to move the bucket assignement with "follow mouse" during the render to place the buckets in the hot areas first. But of course it's less than optimal.


EDIT : my bad as I misunderstood the point here, our problem is more with core performance disparity than stuck buckets due to scene local complexity. I'll give a try later in a better suited post

[^Lele^]

For computational complexity across the frame there's an automated solution being worked on, which will rely on the LC to prioritise buckets.

For random stuck buckets, the request stays the same: we will need a scene which can reliably reproduce the issue for *any* solution to be worked on.

[kosso_olli]

For random stuck buckets, the request stays the same: we will need a scene which can reliably reproduce the issue for *any* solution to be worked on.

There are no random stuck buckets on our end. They get stuck on heavy DOF, motion blur or refractive objects with high ray depth.

[ALEX_SPYROPOULOS]

I think you can draw an average time of the surrounding buckets, then if the average render bucket time limit is exceeded by ... let's say two, and there are idle cores then you delete the bucket and split it to the remaining cores.
I switched to progressive with time limit for Distributed Rendering just because of that . But there is a strong use case for buckets.

[^Lele^]

As i said, that part where the recursive computations take longer we think we understand well.
And for those, there is not going to be a "fix", if you want the recursion, the dynamic range, and the convergence : it's the price to be paid, it's down to you to find the best strategies to mitigate it, as other suggested.

The LC-based prioritisation will do absolutely nothing if you have a bucket sitting there ten+ times longer than the rest of the image takes to complete, so it's still going to be back to manual optimisations an LnR TD ought to implement manually: make the shader simpler (recursion, gloss, reflectivity), make the lights' energies to be distributed fainter, be happy with a higher noise level for that part (i.e. Pass renders, checkerboarding, etc.), and so on and so forth.

If you're averse to any form of pass rendering, post production, or scene optimisation, for whatever valid reason, all you will ever be able to do instead will be to wait.

[^Lele^]

I think you can draw an average time of the surrounding buckets, then if the average render bucket time limit is exceeded by ... let's say two, and there are idle cores then you delete the bucket and split it to the remaining cores.

And so waste time which was spent properly to converge what the engine was tasked with? Nope, not going to work.

I switched to progressive with time limit for Distributed Rendering just because of that . But there is a strong use case for buckets.

This is smart a move for these scenarios.
As mentioned, if you need buckets, consider rendering image chunks locally on slaves.
I believe Deadline has the scripts to do it inside max, but there may be a number of other such tools laying around.

[matthew999]

i have exaclty the same problem with vray getting stuck on car headlights. i have come to accept it and what i do is have 2 deadline slaves on one machine so i can get all my other passes rendering while vray sorts out those pesky last few buckets around the headlights for an extra hour. By the time it has finished all my extra passes have rendered making use of the unused cores.

I think its down to the massive amounts of glass on modern day head lights and vrays slowness at rendering refraction. When i first migrated from mental ray to vray this was the first thing i noticed how slow it was at rendering multiple refracted surfaces on top of each other ie car headlights. water splashes etc etc, compared to mentalray. Looks like nothing has changed in 10 years

On deadline although i have never used there is a jigsaw style tile rendering where it can remember where the slowness is and subdivide that area and give it priority. This could be a solution to fixed camera renders in vray perhaps, get the buckets over the slow areas first so at least all your cores are being used for longer during a render.

Also if im rendering via vfb then i will hover my mouse pointer over head lights to get buckets started on head lights then render as normal once buckets are populated over troublesome area.

thats my 2p

[^Lele^]

All good approaches.
Remember you can lock the bucket start point, which could be somewhat hepful on a short enough animation interval.

[matthew999]

All good approaches.
Remember you can lock the bucket start point, which could be somewhat hepful on a short enough animation interval.

is lock the bucket start point a max feature, how can i do this in maya it sounds useful

thanks

matt

[^Lele^]

Click on the mouse follow icon, then RMC on the VFB window and select the option.