Can stuck bucket syndrome finally be fixed, please?

[matthew999]

thanks lele

[ALEX_SPYROPOULOS]

And so waste time which was spent properly to converge what the engine was tasked with? Nope, not going to work.

This is smart a move for these scenarios.
As mentioned, if you need buckets, consider rendering image chunks locally on slaves.
I believe Deadline has the scripts to do it inside max, but there may be a number of other such tools laying around.

Well I think it could be nice as an option. Especially when you have 180 cores sitting idle waiting for two chunks to be rendered at 3 ghz. I am using progressive for production for a few years now to be honest, I believe though that there is an issue that buckets crash (used to crash) sometimes. I understand that in areas with displaced frosted refraction and caustics things are "intense" but I witnessed buckets going haywire with relatively low complexity. Also a method like that could accommodate for hardware failure. I had one of my cpu fans dropping dead in the middle of the day while rendering a small animation, I went to see the finished result and it was stuck at frame 50 or something. There were 4 perfectly good machines to do the job. Sometimes it could be a network issue, or a weird windows update. I think it makes sense to have an option to time out things and split them to the remaining available machines or simply redistribute them to more cores if they take too long. I am not supporting things like this for an 8 core machine ... but if there is available power there.
I am not very keen to get to amazon's deadline ecosystem, but maybe one day. I kind of see this as vray territory to be honest and I don't use Krakatoa.

[^Lele^]

Sometimes it could be a network issue, or a weird windows update.

Well, this is the crux of the matter, isn't it?
While there could be optimisations coders could make for severe computations (the LC-driven prioritisation. f.e.), there is no warrant for us to care about a client's infrastructure to this level.
Rather, as you're doing, depending on the kind of workload and hardware you have available, you should split and generally optimise your workflow.
It's a render farm manager's task to decide a frame has gone on too long and restart it, to split long tasks into chunks, to assign the machines with the higher power to the hardest areas, as for all V-Ray knows, it's doing its job just fine.
Adding the option to the engine is in my opinion redundant (as it's in the wrong place.), dangerous to the wider public, and not really solving the basic issue, which is infrastructure-bound.

[kosso_olli]

Why does every thread derail as soon as Lele starts joining? I did not hear a single word from any of the developers.

[yavor.rubenov]

If there was a "quick fix" or some easy way - it would have been done years ago. Although, you as users can help us very much by sending such scenes to test possible solutions.

What Peter said
There is no quick fix for this issue and we need real life scenes that demonstrate it - last 1 or few buckets stay hours after the rest are done.

[kosso_olli]

I can prepare three different scenes, but it will take some time to strip them for you.

[yavor.rubenov]

That'd be great.

[^Lele^]

Why does every thread derail as soon as Lele starts joining? I did not hear a single word from any of the developers.

I thought you were sorted with providing scenes, and I was talking to users other than you.

Regardless, my roles include talking with the devs in the users' stead, and replying here after deliberation, as their time is much more valuable than mine.
If you have a special preference on who should be answering your queries, you should use our emails instead.

[Joelaff]

What Peter said
There is no quick fix for this issue and we need real life scenes that demonstrate it - last 1 or few buckets stay hours after the rest are done.

Forgive me, but can't you just make a simple test scene with some slower areas (refractive/reflective/SSS geosphere with high subdivision and a bunch of noise on it.) vs. a simple plane in the rest of the scene? The turn up the samples and lower the threshold.

I don't think the blocks need to be stuck for hours. Most of the time the issue is that the last block takes maybe 400% or more longer than the average block.

When you end up with four inactive cores you immediately have those cores work on an unfinished block (which you split into four). If the original core working on the un-subdivided block finishes the block first it is done. If the four cores finish the block first then the block is done. Rinse and repeat recursively.

If you find you have 16 cores free then you double-subdivide the block and do the same thing. Keep as many cores as possible working on the remaining area until done.


Another thing that would be helpful would be to simply be more aggressive about dynamic subdivision, starting this earlier than you do now. Ideally you let the use set a threshold for this, as only the user knows what works best for their scene.

This is a renderer function, and has nothing to do with render managers or infrastructure. The renderer should use all the cores as much of the time as possible.

[yavor.rubenov]

Forgive me, but can't you just make a simple test scene with some slower areas (refractive/reflective/SSS geosphere with high subdivision and a bunch of noise on it.) vs. a simple plane in the rest of the scene?

In our experiments with such scenes in most cases the difference between some optimal bucket splitting that uses all cores till the end and worst case scenario is just a few percent.

[Joelaff]

In our experiments with such scenes in most cases the difference between some optimal bucket splitting that uses all cores till the end and worst case scenario is just a few percent.

A few percent is generally all we get when we buy new hardware these days. (Processors haven't been improving that much except the jump around when AMD broke out). In other words, a few percent is a big deal to pretty much anyone doing this professionally.

5% faster rendering can literally be the difference between blowing a deadline (and thus losing tens of thousands or more in future work), and keeping a client happy. That's 1.2 hours a day!

And it gets worse... because remember that the shot is not done until the LAST block in the LAST frame on the SLOWEST machine is finished rendering. This is always what screws us every time. We are always waiting an extra 20mins or so for that final frame (since our renders are often 20-60mins/frame on modern hardware). Even with a totally homogeneous render farm you still end up waiting on that last block of the last frame.

If VRay6 is 5% faster than VRay5 that would be a big selling point... Every little bit counts.

[yavor.rubenov]

Maybe I should've explained better - what we've seen is things like 5-10 seconds difference and making the render longer doesn't change it much. Scene that has the last one or two buckets stay for extra 20 minutes is the thing we are looking for.

[yavor.rubenov]

And to add to that - here we are mostly looking into rendering on a single machine and having only one or two buckets left.
DR with multiple different machines and hanging buckets is another different issue. It is also in our todo list to solve.

[Joelaff]

Maybe I should've explained better - what we've seen is things like 5-10 seconds difference and making the render longer doesn't change it much. Scene that has the last one or two buckets stay for extra 20 minutes is the thing we are looking for.

Maybe I am confused too. Could you not just take a complex model and make it very small in frame where all the other blocks render almost instantly, but the ones with the complex model are slow?

Or do you for some reason need these slow blocks to meet specific criteria? I just mean for the splitting algorithm R&D.

Of course if you are trying to do something more intelligent (like base it on the LightCache or something) then maybe that is it? But even with a simple scene like described above you could still time the LC sample times and modify the splitting based upon those times. Let the user set the minimum block size for the complex areas, but then increase block sizes based on how fast the LC samples in the area.

Sometimes the "naive solution," as a CS professor would call it, is the best one. Love the idea of something better, though.

[kosso_olli]

This is a renderer function, and has nothing to do with render managers or infrastructure. The renderer should use all the cores as much of the time as possible.

Thank you, this is exactly how I feel. For a studio of our size, we got a pretty decent render farm. And yet, we spend most of the time looking at two or three buckets rendering more than twice the time of the rest of the image. It is not just a few percent!