Everybody knows the “stuck bucket” phenomenon. Dynamic splitting being imperfect, it’s still an issue from time to time.
I don’t know if it’s easy or hard to implement, but what about an “early termination timer” that stop the rendering of a bucket if the limit is reached.
I had yesterday 2 buckets 64x64 stuck for hours (litterally 4 hours for 2 buckets on a 4k rendering usually rendered in 1h) while in a rush, no time to debug. We just had to save the passes manually from the farm and “draw” the missing buck in photoshop (10 minutes )
A timer, for example 10 minutes/bucket (which is already huge for 4096 pixels ) stops the rendering of this part where it is and moves on.
Reducing the bucket size was not really an option since we still work with IM.
If using Distributed Rendering I’ve found removing whichever machine via the rendering dialog whilst rendering then starting it again can help if a bucket has been stuck for a crazy amount of time due to an error of some kind, not perfect but I’ve found is a good work around to restart a certain bucket, when problems arise.
Yep trick we are using for years now, only it work well when the current bucket is on anything else than the master node that drives the DR, When it falls back to that one, if it’s stuck you have no option to kick it back to someone else.
The time limit is just a great idea and makes a lot of sense.
I would even add that if the time limit is reached, that bucket get divided in 4 smaller onces, that would be a great addition.
On a similar note, if I manually kill a stuck bucket (or even automatically) I’d like the remaining bucket to be divided using the usual dynamic splitting feature. Occasionally (very rarely honestly) a machine will freeze up on all of it’s buckets (maybe it ran out of memory or something similar) and I’ll have 8 or more buckets that need to be restarted so it would be nice to have all of my DR machines available to pick up the remaining smaller buckets.
Why not do the following: Divide all the remaining buckets by the amount of DR nodes? A little explanation: Lets say you have 10 rendernodes with 8 cores each. There are 8 buckets left rendering hard. That means it is basically one machine that renders them. Why not divide the bucket size so far, that all the other DR nodes also have something to do?
+1!
I have an animation that I am working on that gets stuck on certain objects (glossy refractive objects) and the rest of the frame takes about 15mins but 2 or 3 buckets sit there for an hour and most of the CPUs are idle.
Could this idea be expanded to have a by “object” property of dividing buckets so as soon as certain objects are hit to render the buckets are automatically split?
It will first start to render then it has to time out, then it will have to be loaded again to more cores. If there was anyway to identify the difficult areas in advance so you can automatically scale down the bucket size that could be great.
Maybe doing a minimal draft render to create a ‘it’s complicated’ map : )
A timeout feature could be very interesting in case something goes wrong with one bucket.
I’d like this “split buckets after n seconds” function, although I hope this doesn’t mean those new buckets have to start rendering all over again. It would be cool if those buckets could at least use what the stuck bucket has already calculated and continue from there.
I’m rendering an animation at the moment that takes about 2 mins for most of the frame except for 2 buckets that take another 2 mins just for them. This is effectively doubling the render time.
I have ‘optimised’ it by having the render start on the side of the image with the offending buckets, so that it does not get stuck at the end of the render but rather at the start of the render, this reduced the render time by 50%
So there is a potentially a lot of speed to be gained by some smarter bucket splitting algorithms.
The best option out of all the idea’s i’ve seen here would be that the remaining buckets split (as many times as there are cores or nodes) when there is no new buckets left.
)
