Stuck buckets

[kosso_olli]

I know this has been discussed numerous times in the past, but is there anything that can be done against it?
It is so frustrating to see your render basically being finished in an hour and then getting stuck on two single buckets for an additional four hours! This is happening so many times now, on nearly all our scenes.

[3d-laboratory]

Hello Oliver, as I saw on your WEB-site you often work with car paint. I have the same problem sometimes. When two object sare illuminated by a sunlight ore a light dome and the two objects are reflecting each other I have the seme problem. For example; I have a facade with some elements who reflect cars wich are located on the ground and get sun from a sunlight or a HDRI. If the sun makes a light spot on the cars which are reflected by the parts of the facade these buckets renders a long, long time. When I remove the cars all is fine and I have normal render times again. May be that your problem it is caused by the reflections...(as you are German you can write an email to me in German language info@3d-labor.de if you didn´t understand what I mean)

[^Lele^]

I know this has been discussed numerous times in the past, but is there anything that can be done against it?

Well, what is it that is taking long?
If it can be sample-stopped (f.e. a strong highlight where noise won't be visible), then cap the max AA.
V-Ray will sit on those highlights until either they are within noise threshold range (often quite difficult indeed.) or max AA is reached (much more common.).

Otherwise, if you don't plan to do loads of post, clamp the renderer to, f.e. 13 float.
It'll leave you a few stops of play, but maybe it'll bring those highlights home in more reasonable times.

But if it's not highlights, then strategies may be different.

[Joelaff]

Actually, there is a very simple strategy that Backburner (the original Backburner, not Max stolen name version) used in 1994 or so. Backburner was the network rendering software with Specular International’s Infini-D software.

Backburner did in 1994 what I have been asking every other renderer to do since. Once the number of remaining remaining blocks is less than the number of free machines/cores /4 you chop the remaining blocks up into fourths and put four processes on it. But you keep the existing core or machine rendering it as well. Whichever processes finishes first wins and the job is done.

In other words every core works on the remaining blocks by subdividing them while they are still being rendered by the original core or machine. I keep saying core or machine because this should happen with net rendering because some render nodes are an order of magnitude faster than others. Backburner was for network rendering, similar to VRay distributed rendering. It worked so well so long ago. With distributed render you could subdivide and put multiple machines on the subdivided blocks.

Last Block Syndrome, or LBS as it is more commonly known, has been an insidious issue for decades, but it has a solution. Of course for now we just have to use a smaller block size. VRay5 has better dynamic subdivision than 3 did, but I still think the dynamic subdivision should start faster still. Any speed loss due to smaller blocks is more than made up for by time saved from less severe LBS.

[vlado]

It's on the "to do" list to finally try and tackle this.

Best regards,
Vlado

[Joelaff]

Thanks, Vlado.

I am sure in most cases it is really not that severe. However, it does drive the users insane to see a single core working on a block trapped in LBS (Last Block Syndrome). About 10% of the time this a real issue. I would love to see the brute force approach applied again in 2020. Throw all the resources at it, and whichever finishes first wins! And the user wins!

I love the idea of the dynamic splitting. I just think it needs to be even MORE aggressive. With most image filters (says the guy who likes the Soften filter at 3.98-- not lumped in here) the smaller block size makes very little difference because it resolves LBS much faster.

I was impressed the Specular International guys got it done so well in 1994. A shame they sold out to Macromedia who then killed Infini-D in favor of their own RayDream Designer (which was nice, but also got killed.)

Thanks.

[kosso_olli]

Well, what is it that is taking long?

Hey Lele,
it's the headlights of course, in all the cases. Sure, with all the bright light, refractions and reflections they take longer to render, but for some reason, there is an inconsistency. Out of let's say 100 buckets rendering the headlights, two or three always get stuck in different places. In the case of the image I started last night, the car itself was finished in under an hour, but it took a whopping 7 hours to finish the last two buckets.
In the past, I often canceled the render and rendered the remaining small area with progressive, but this is not an option while rendering over BB.

[Joelaff]

Hey Lele,
it's the headlights of course, in all the cases. Sure, with all the bright light, refractions and reflections they take longer to render, but for some reason, there is an inconsistency. Out of let's say 100 buckets rendering the headlights, two or three always get stuck in different places. In the case of the image I started last night, the car itself was finished in under an hour, but it took a whopping 7 hours to finish the last two buckets.
In the past, I often canceled the render and rendered the remaining small area with progressive, but this is not an option while rendering over BB.

Wow! What bucket size? Have you experimented with smaller buckets? We usually end up going that route in these cases. Makes very little difference in render time vs. larger buckets even without LBS (Last Bucket Syndrome).

I know headlights are notoriously slow to render here too, and for us is is an HD or 4K shot where the headlight isn't seen very long. I can only imagine for a high res still.

[kosso_olli]

It's on the "to do" list to finally try and tackle this.

Okay, great. If it comes to the point where this will be tackled, please don't hesitate to ask for examples. There are quite a few scenes illustrating the problem.

[kosso_olli]

Have you experimented with smaller buckets?

Yes. It does help a little, but smaller buckets just means more buckets getting stuck. If it's just two or three buckets from a render node, I can still render on the other ones, but more stuck buckets mean more blocked nodes. Bucket size right now is 16.

I know headlights are notoriously slow to render here too, and for us it is an HD or 4K shot where the headlight isn't seen very long. I can only imagine for a high res still.

Imagine rendering a closeup of the front fascia of a car, full-frame at print res with the headlight out of focus somewhere in the frame. The horror.

[Joelaff]

Yes. It does help a little, but smaller buckets just means more buckets getting stuck. If it's just two or three buckets from a render node, I can still render on the other ones, but more stuck buckets mean more blocked nodes.



Imagine rendering a closeup of the front fascia of a car, full-frame at print res with the headlight out of focus somewhere in the frame. The horror.

LOL @ the horror, Colonel Kurtz!

But yeah, that sucks, man. Good luck.

[fixeighted]

This is probably a dumb idea but I'll voice it anyway; wouldn't it be possible, when getting to a problem bucket, to search the pixels in the bucket and if they are of a similar brightness then simply stop that bucket at an average of the pixel values?
If this was set to a maximum time per bucket, or something similar, then wouldn't that be sort of like clamping but on a very bucket-specific level and without having to clamp anything per se?
Apologies if that is just plain nonsense

[joconnell]

Worth your while doing the headlights as a second pass with clamping / progressive and invisible to camera in the main one for now? Have you got a good automated rendering solution for all your stuff? Changsoo Eun from fuseFx is writing his own logic based render manager with lots of passes / tokens and so on that could be good for this type of thing.

[^Lele^]

This is probably a dumb idea but I'll voice it anyway; wouldn't it be possible, when getting to a problem bucket, to search the pixels in the bucket and if they are of a similar brightness then simply stop that bucket at an average of the pixel values?
If this was set to a maximum time per bucket, or something similar, then wouldn't that be sort of like clamping but on a very bucket-specific level and without having to clamp anything per se?
Apologies if that is just plain nonsense

That's super easy to do in post, for individual pixels (that's how fireflies removers, like the simple one i built, work.).
It is a good idea.

Until, that is, one has a strong specular covering an area.
Or the area in question doesn't lend itself to naive blending (like Olli's headlights.).

The problem with time alone is that we don't really know what is being computed (that's, technically, why we render: to find out.) : it may need that time to clean up or look bad beyond repair.
In other words, some areas take a long time even if they aren't Olli's headlights, purely through shading complexity.

That's why i suggested manual user intervention: a user knows things a renderer doesn't until it's done, however long that may take.

[fixeighted]

Good points Lele; I was thinking after I posted that fireflies were actually the best application of what I was thinking of.
Back to looking around for that special wand thingy then