How to estimate size/benefit ratio for DR renderfarm

[simmsimaging]

Thanks Robert.

I think the Rebus calculator isn't much help because they do not use DR, as I understand things anyway.

You make a good point in terms of setup time, but most of my renders are pretty high res (only do stills) so I'm in the second category where setup time is not often a major component of the overall time for me. Having more cores in one box would still be advantageous though. I just need to figure out precisely how the falloff works over the number of cores.

I don't know if 2 machines is twice is fast as 1, but it would not surprise me. What I'm not sure about is whether that changes. Is 8 2X faster than 4, or 10 2X faster than 5? For some reason I don't think it's going to work that way, but I don't know why really Maybe because no one seems to be able to confirm it for sure. Anyway, once I know that I can optimize the number of cores per box against my budget.

I'm going to try and work something out with another forum user to run some tests. If we get that done I'll let you know how it works out.

Thanks
b

[Kyle_Grace]

I think the main problem is there are too many variables to create an accurate 'model'.
It really does depend on the kind of work you deal with...

I think Rmejia hit the nail on the head when he spoke about transferral time... You say that you predominately do high res stills work, but are you working with very dense scenes? (in terms of geometry) then transferral time could be a real problem.

As far as I can understand... The 'Falloff' you speak of can only really be down to the transferral time and the speed at which the host machine can collate the data, both of which will be linked to the speed of your network... and of course as mentioned earlier some of these things can be combated with bucket size etc.

For example we have 2 host machines (dual quad core, 8gig) and 3 slaves (single quad core, 4gig). We often do high res stills that are extremely geometry heavy, even on a small gigabit network the transferral time is painful. But the overall benefit of throwing 28 cores and 28gigs of ram at a high res and very heavy scene is huge, however if I were to double the number of machines but half the power, then transferral time would become a real problem.

I think it really is a cost issue above anything else, because at the end of the day a 28core 28gig machine will easily thrash the hell out of my set up... but it is not feasible. Personally I try to keep the overall number of slaves down, and the number of cores & ram up... without paying a premium for the latest technology.

[simmsimaging]

I guess I have a hard time understanding how the variables can make such a huge difference to the raw power aspect, but it's probably just my ignorance showing

Setting aside transfer times for the moment: what makes sense to me is that that all the variables would dictate the basic "speed" of any given render on a single given core. Adding more cores would just subdivide that speed in some way, in a linear way or some other etc, but should not necessarily be affected by the 'variables' (again, aside from transfer times to get the scenes to the machine).

Is this just a mistaken understanding of how it works?

Returning to transfer times: maybe it's my misunderstanding of the terms, but I think of the transfer times as the point between clicking 'render' and when the first buckets starting rendering on the various slaves. Once that part starts I only see slight differences in the speed at which buckets finish, but that correlates to the various core speeds I have here. From that I assume the 'transfer' is done and no longer relevant. Is that what you mean?

My scenes are not generally geometry heavy, as I am normally doing product type stuff. That might be why (if we mean the same thing by it) transfer time are not really a major factor for me. It might take a few minutes for all 3 machines to start, but then the render phase from there will take 20 minutes, 60min, 20 hours etc. For me it's a fraction of the total time in almost every case.

So, I do get that having more cores (and less transfers) can make a difference, but I have a hard time seeing why it can't be quantified better than by terms like 'huge'

b

[Kyle_Grace]

So if you remove the 'variables' surely the model would be linear? In which case what's the point?

However, you need to transfer the scene to the slaves, then the irradiance map once it has been calculated and collated, and then each rendered bucket to the host. These take time. the more data you transfer over the network the slower it is. If you look at the vray log whilst rendering a heavy scene at a high resolution you'll see all the communication and data transfer that vray is doing. Other variables I imagine to be whether your using static or dynamic geometry? If you're using xrefs etc? Maybe Vlado or someone could know?

Maybe the reason nobody has ever 'modelled' the falloff is because it's specific to the time of work you do and how you go about doing it, hence even if you do find someone with data it wont necessarily be useful.

Like I said for me it comes down to cost, by the time you've paid for the use of someone's render farm, and added your own time to it, i'd rather just spend it on better quality kit (and go home on time for a change)

Sorry I can't be more helpful.

[simmsimaging]

It comes down to cost for me too - that was what started me on this thread in the first place

It's not so much about removing the variables but understanding where they actually come into play. You put it in a way that helps to simplify it though:

Do the 'variables' (which are endless) and aside from transfer times (which for me are quick, the log shows no other relevant transfers that I can see) affect the "base" render time per core only, or do they somehow affect how it scales across a DR render too. Either they do do or they don't. If they don't then it is a simply linear progression to figure out how much bang you get for your buck with adding slaves. If it does then there simply is no answer to the question. You just have to throw more money at it and hope for the best - which is not ideal and potentially very wasteful.

Anyway I think it is a simple enough thing to test if you have the farm for it. I just need to get a scene or two together and negotiate the cost and I'll have an answer (I hope!)

b

[sharpcube]

and wher do you buy a rendernode ? in europe belgium?

[MikeHampton]

My rules of thumb are:

1) Don't buy the latest and greatest processor/video card for either your main workstation or your renderfarm.. The real world performance of a $2K to $3K computer and an $8K computer simply doesn't justify the 10-20% speed increase you may or may not realize.

2) Although your main workstation may be a little suped up compared to your renderfarm (your render farm computers don't need killer video cards/etc.), try to keep your slave/renderfarm as close to your main workstation as you can in processor and ram. You don't want your workstation to be able to render something in 5 minutes that it takes one of your slaves an hour to do. And especially you don't want to be working on something that your workstation CAN render but your render slaves can not.

3) Then as far as how many to buy. I would buy as many as you can afford (This is assuming you can't afford to buy 1000 computers), and still add to it/replace some of them every year or two. How ever many slaves you buy, they will all need to be replaced/upgraded in 3-5 years. It's better to add a couple new computers every year or so and let the older computers drop out of your farm than to have to upgrade your entire farm at one time. So don't buy more than you can afford to replace in that time frame. Also realize that however many you buy, once you have more than about 10, if you want to add to it, that you need to increase the number you have by about 25% to 100% to realize much of a gain. IE.. if you have 100 computer renderfarm.. adding 5 more to it, you arn't going to realize much of a gain.


I have a 10 computer renderfarm where 2 of them have fallen apart and the other 8 are about useless. 5 years ago I wasn't throwing GI with millions of polygons at them. Who knows what I will be throwing at them in another 5 years. Luckly I think I will be able to add 10 new ones this year. I was asked if I wanted to get 20 instead of 10. Seems like I should jump at that.. but actually I asked if getting 20 meant that I wouldn't be able to ask for any more for another 5+ years? I was told that if I get 20 now.. don't even think about asking for any more for quite some time. I told IT that I would rather just get 10 now, with the plan that I would be able to get another 3-4 more new machines each year to add to it than to get all 20 at once. Because if I had 20 new machines at one time.. they are all aging at the same rate and in 5+ years.. they may not be able to handle what I would be needing them to do at that time and then all 20 of them need to be replaced at one time. Not adding a couple new machines each year to my current render farm is why I have 10 aging machines right now that are all falling apart on me at the same time.

If you are just trying to nail down EXACTLY how many to buy... why not just buy a couple of computers for your renderfarm and see how that works for you.. then if you still need more horsepower, add another one on in about 6 months.. then if you need more, add another in another 6 months and so on. When you have fewer than 10.. you can add 1 at a time and still see a significant performance increase. Once you get to about 10, I think you need to start adding more than 1 at a time to notice much of a benefit.

There is a saying that goes "If one person can dig a 3 foot deep hole in an hour, then 2 people ought to be able to do in in 30 minutes, right? Then 3 people should be able to do it in 20 minutes, right? Then 60 people ought to be able to do it in a minute, right? NO?!?! Well that's what the math says!!"

[simmsimaging]

That is a great guideline, thank you for that. I think that makes it clear and simple and helps clarify what some of the other guys have been saying. As a plan it makes a lot of sense.

Much appreciated.
b

[devin]

There is a saying that goes "If one person can dig a 3 foot deep hole in an hour, then 2 people ought to be able to do in in 30 minutes, right? Then 3 people should be able to do it in 20 minutes, right? Then 60 people ought to be able to do it in a minute, right? NO?!?! Well that's what the math says!!"

You've made some good points but my testing has shown that DR speed doesn’t increase in this way. The fact is that as you increase the number of machines in your DR farm your performance per DR node decreases significantly.

[rmejia]

There is a saying that goes "If one person can dig a 3 foot deep hole in an hour, then 2 people ought to be able to do in in 30 minutes, right? Then 3 people should be able to do it in 20 minutes, right? Then 60 people ought to be able to do it in a minute, right? NO?!?! Well that's what the math says!!"

Following the digger analogy, with 60 diggers digging a 3' hole, 57 diggers (buckets in the renderer) might be standing around with their shovels in hand... because they would not fit in such a small hole ... but 60 diggers digging a 60' whole, then they would be able to dig.

More computers on a small simple scene would be of no use after some point, they would only be able to do something on a large scene, so the larger the scene, the more computers to put in. Image size (like digger hole size) would be directly proportional to the amount of computers (diggers) that can work on it. If the output size is small, and only has 32 buckets, and there are 4 octa core computers, with one bucket per core, that would be the max, if there where 10 octa core computers in the farm, 6 of them would not even work on the scene, unless the bucket size was adjusted in which case:

Somewhat related to the time of the rendering is the size of the buckets. A lot of very small buckets (say 16x16) will render slower than the standard bucket size (64x64) due to overlap or something like that. I believe for speed you'd want to have as few buckets as possible, essentially one bucket per available processing core. This is assuming all of your machines have comparable speed-otherwise smaller buckets will balance the load better between the machines.

As far as scene complexity is involved, in a scene I have with about 13,410,282 Polys it takes a couple of minutes to load, then recieve the scene, and then start rendering on the slaves. Could take from 5 to 10 (maybe more) minutes depending on the slave specs, so DR can be pretty much useless in this case for something that takes 10 to 20 min to render. In this case the slave rendering initiation times can be a bag of hurt. The amount of slaves would have to be directly proportionate to render size and time, so it can greatly vary depending on scene size and complexity.

[cgrant3d]

IMHO while there are a ton of variables to take into account to find the exact perfect number of DR nodes your particular business needs the general formula can be quite simple.

For discussion sake lets assume the majority of time is spent rendering distributed tasks ie a 1 hour rendering isn't 45 minutes preparing geometry and 15 rendering its a good mix of GI/displacement/glossies that take 1 hr per frame. Lets assume all textures are stored on a file server and you're using a high speed full-duplex gigabit network (both of which are relatively cheap and help keep these calculations consistent). Lets also assume that each machine costs $3000.

1 workstation = 60 mins
+1 node = 30 mins (rendertime reduced 50%) + $3K
+2 nodes = 20 mins (rendertime further reduced 16%) +$6K
+3 nodes = 15 mins (rendertime further reduced 8%) + $9K
+4 nodes = 12 mins (rendertime further reduced 5%) + $12K
+5 nodes = 10 mins (rendertime further reduced 3%) + $15K

Now keep in mind this is ONLY taking into account DR needs for a single project. So if you're looking at this list and thinking of only buying 2 nodes you have to ask yourself whether you'll need to render backburner tasks AND need DR nodes simultaneously. Also - how many artists are there?

...

Here's how I might break it down for an arch-viz business model.

Typical day might be 1 high resolution still (60 mins) + 20 test renderings @ 2.5 mins each (averaged - some higher some lower) so a typical artist might be doing 90 mins of rendering every day

So - if you buy 3 render nodes thats $9K and then maybe invested $1K in network infrastructure upgrades (full-duplex gig instead of 100 lets say) you've got $10K into it and based on the percentages above (60 mins versus 15 mins) you could expect your daily time spent waiting for a render to reduce from 90 minutes to 22.5 mins so you've saved roughly an hour per day. Now you just have to figure out how much your billing rate is and we can figure out how long before you'd see a return on your investment.

For example - if your billing rate is $100/hr (makes the calculations easier, obviously your market prices / experiences vary and affect the hourly) so you might look at this investment as saving $100/day. Thus a $10K investment would take 100 fulltime work days to pay off or roughly 20 weeks.

...

So the million dollar question though is what will you do with the time saved? If you go overbudget on projects all the time then saving an hour / day won't help you. The trick would be using that hour per day to get work done faster, thus completing more projects per month and making more money - otherwise its not actually really ever paying for itself...

[devin]

As far as scene complexity is involved, in a scene I have with about 13,410,282 Polys it takes a couple of minutes to load, then recieve the scene, and then start rendering on the slaves. Could take from 5 to 10 (maybe more) minutes depending on the slave specs, so DR can be pretty much useless in this case for something that takes 10 to 20 min to render. In this case the slave rendering initiation times can be a bag of hurt. The amount of slaves would have to be directly proportionate to render size and time, so it can greatly vary depending on scene size and complexity.

Don't forget that most scenes aren't perfectly balanced, in some areas you'll have transparency and reflection and in others you'll have open sky. If you have an 8 core systems and size those to fill up the entire scene then some cores will have a harder time rendering than others which kills your efficiency.

1 workstation = 60 mins
+1 node = 30 mins (rendertime reduced 50%) + $3K
+2 nodes = 20 mins (rendertime further reduced 16%) +$6K
+3 nodes = 15 mins (rendertime further reduced 8%) + $9K
+4 nodes = 12 mins (rendertime further reduced 5%) + $12K
+5 nodes = 10 mins (rendertime further reduced 3%) + $15K
..

How did you arive at these numbers, and can you explain why the rendertime reduction isn't constant or would that require a doubbling of speed for every node over the last one?

[cgrant3d]

Don't forget that most scenes aren't perfectly balanced, in some areas you'll have transparency and reflection and in others you'll have open sky. If you have an 8 core systems and size those to fill up the entire scene then some cores will have a harder time rendering than others which kills your efficiency.

True but the numbers I'm giving are just the broad strokes to find a general baseline. There are of course plenty of instances where they would not make sense but what I'm trying to do here is provide a baseline where people can look at the scenario - see if it fits them and how the numbers add up. IMHO its not actually as important to see the specifics as it is to understand how adding additional nodes scale (see below).

How did you arive at these numbers, and can you explain why the rendertime reduction isn't constant or would that require a doubbling of speed for every node over the last one?

The reason the numbers aren't constant is that your overall renderfarm power gets larger and larger so each new CPU becomes a smaller and smaller addition.

Maybe it'll make more sense if we look at a renderfarm in GHz, and each node as the sum of all the processors inside it (quad 3GHz = 12GHz, dual quad = 24GHz).

1 workstation = 60 mins (24GHz)
+1 node = 30 mins (rendertime reduced 50%) + 24GHz(48 total)
+2 nodes = 20 mins (rendertime further reduced 16%) + 24GHz(72 total)
+3 nodes = 15 mins (rendertime further reduced 8%) + 24GHz(96 total)
+4 nodes = 12 mins (rendertime further reduced 5%) + 24GHz(120 total)
+5 nodes = 10 mins (rendertime further reduced 3%) + 24GHz(144 total)

The important part to see is the effect of adding another dual quad render node gets smaller and smaller as your renderfarm grows. Does that make sense?

[devin]

Yes that makes since, thanks.

[rmejia]

1 workstation = 60 mins
+1 node = 30 mins (rendertime reduced 50%) + $3K
+2 nodes = 20 mins (rendertime further reduced 16%) +$6K
+3 nodes = 15 mins (rendertime further reduced 8%) + $9K
+4 nodes = 12 mins (rendertime further reduced 5%) + $12K
+5 nodes = 10 mins (rendertime further reduced 3%) + $15K

Yep, makes sense.

It also correlates with devin's tests which would be:
+1 node = rendertime reduced by 41.5%
+2 nodes = rendertime further reduced 14.8%
+3 nodes = rendertime further reduced 7.7%
+4 nodes = rendertime further reduced 1%

After the 3rd additional computer the price vs performance ratio makes it pretty much useless to keep adding computers. So in conclusion the "ideal" amount of pc's rendering in DR would be around 3, with 2 being the best bang for the buck.