Building a renderfarm

[Lucetius]

Suurland,

If you have a scene in mind, (i.e. something similar to what you do to give you a good reference point.) I'd be happy to run a benchmark.

[suurland]

Lucetius,

Don't have a scene in mind, just wondering if you have some general numbers on which computers is the fastest. Maybe the "Performance index" from the backburner queue monitor would give a good picture.

/Thomas

[RErender]

Lucetius,

Don't have a scene in mind, just wondering if you have some general numbers on which computers is the fastest. Maybe the "Performance index" from the backburner queue monitor would give a good picture.

/Thomas

That number has very little to do with the actual speed of a system, not sure what all it factors in but often on our farm a 600 can have a higher rating than one of the 2.8's wich is just not possible on a horespower basis...

[ihavenofish]

my experience with render farms and PC's in general lead me to a few suggestions. i've had my own persnal "farms" ranging from a few to maybe 7 or 8 machines, and also worked at a place with a farm of near 200 machines. each scenario has its own quirks it seams.

first and foremost, make sure all the machines are the same, or at the least, the same type. this make it infinitely easier to administer. you could set up a ghost drive to rebuild any of the machines.

2: if you dont have a dedicated sys admin, or are not up to managing finiky pc's while trying to meet deadines, get someone else to build them. a reliable service partner that can get you help fast. this could be boxx, or you reliable local vendor, doesnt matter as long as you can lean on them on a sunday night when 3 machines tank hard disks

3: do not skimp on a few things... motherboards-if these go, you can be totally *ucked for days. get something nice, but not so rare you have to order them custom from taiwan... ram- the last thing you need is mysterious crashes... hard disks- small and cheap, but long life. HD speed seems not to matter terribly in a farm.... video cards- PAIN in the ass. use a high quality old school nvidia or ati card so you wont get driver issues.

4: cases are important. and yes, they can cost more than the rest of the PC. make sure you have adequate cooling (case and room) and that the power supplies are built like tanks. since we were not space limited, the megafarm at the place i used to work opted for standard tower cases and little regard for electricity consumption. the down side was that in the end, they installed a separate massive AC unit just for that room to keep it cold. its debatable i think whether they actually saved money on cheaper cases. they also have an entire department to watch over them and keep them up and running.

5: if your farm is getting big, over say 10 machines, make note of your network, especially if you are rendering massive frames and files. gigabit becomes a must at the backbone (though not necessarily on each machine)

6: monitoring and admining station. once you get above a certain amount of machines, the simple KVM approach begins to get bothersome. this needs a solution of some sort.

7: think of the futrue... if you are buying a whack of machines, you want something you wont toss in 1 year when you get the next batch. today, its seems that something with 64 bit support and room for more ram is the way to go. also agp 8x and/or pci express may become a feature point if gpu accelerated rendering takes hold.

8: multi cpu is also important, as in the end it can save costs not only in hardware, but in space, cooling, and software licenses.

9. software will become important on a farm, particularly the OS. those miniscule (or major) speed and relibility differences between win2k and and XP and the service packs along with setup will begin to mean something. you should take the time to find the best setup for the farm and then keep it. this is why its nice to have all the same boxes... you can immediately tell if one is mis behaving.

10. if you can, get a dedicated box with a raid array and psycho ethernet for saving frames to. the means you are free to crash your workstation at will and not worry about messing up the farm saving files. you may also want your actual max files and textures on this box too, or better yet, another fast network machine.

as for brand of cpu, i am an amd guy, but honestly, it doesnt matter. as long as you can rely on them. believe it or not, raw speed per machine isnt actually the stong point of having the farm(though its nice). you can always buy more

thats my few thoughts towards farms. sorry for not pointing out specific brands or models, but i honestly think as long as the person you buy it from backs it up with solid service, you wont need to care.

later

[suurland]

Lucetius,

Don't have a scene in mind, just wondering if you have some general numbers on which computers is the fastest. Maybe the "Performance index" from the backburner queue monitor would give a good picture.

/Thomas

That number has very little to do with the actual speed of a system, not sure what all it factors in but often on our farm a 600 can have a higher rating than one of the 2.8's wich is just not possible on a horespower basis...

If your slaves are rendering the same animation where each frame in average take the same amount of time the number is infact accurate, atleast it is here. Right now I'm rendering an animation on 8 slaves with 3 kinds of computers and the "performance index" is very accurate when I compare it with the rendertime on the actual machines.

The calculations:

Fastest system takes 87min per frame, the second faseste takes 102min/f and the slowest 173min/f.

Back burner gives me performance index 1 for the fastests of course, and 0,83 for the second fastest, and 0,47 for the slowest.

And if I calculate the rendertimes by hand I get: 87/102=0.85 and 87/173=0,5

So the deviation is 0,02 and 0,03.. I call that pretty damn accurate

/Thomas

[tammo]

here is a benchmark opteron VS xeon using max6 and vray:
http://www.hexus.net/content/reviews...lld19JRD04MjI=

xeons are 40 % faster.

[percydaman]

interesting article. would appear that the opterons are pretty much superior in just about everything BUT rendering in 3dsmax and vray.

[ihavenofish]

just a curious note about all this xeon vs opteron stuff and benchmarks.

i know someone who bought himself a nice new dual opteron 246, 2gb ram, 10k hard disk.. etc.

he set it up, installed XP which was bundled with service pack 2. he rendered off a scene (brazil, but thats irelevant to this point) and he found the machine to be almost 4 times SLOWER than his pentium 4 2.8ghz.

we were all puzzled. this was obviously not the right behavior. he went through a bit of settings and couldnt find anything to help. bios, ram, as well.

on a hunch, since i am a win2k guy and refuse to upgrade, i got him to install 2k sp4. no other changes, acpi enabled (which hurts performance significantly) and he set the scene to render again.

this time the machine was over 4 times FASTER than the p4. EIGHT times FASTER with win2k sp4 than with xp sp2. i suspect without acpi, there may even be a bigger increase in performance.

since i see such mad discrepencies between people with opterons and xeons here, i wonder is OS and set up has anything to do with it.

since people are basing multi thousand dollar purchases on these reports,
if anyone has the time (unfortunately i ony have 1 machine right now and its a p4 2.8 laptop with win2k) it would be interesting to see some peoples tests with their same system under the different OS's and service packs. ya never know, you might be able to get a speed increase by downgrading your operating system rather than upgrading your PC.

this just brings to mind my old dual pentium 2 350 rendering a fair bit faster than my dual pentium 3 450. both otherwise identical except the 350 had acpi disabled in 2k and enabled in the 450.

so, anyone bored?


later

[reza]

with respect to the comments above.....you should always build a render farm of computers with equal speed and ram ammount...

[tammo]

reza, that is of course true and makes administration easier. but in reality a render farm is growing from old workstations and new renderslaves. i never had problems with mixing machines.

regarding OS: no speed differences here between win2k, win server, win xp. tested cpus: athlon and p4, single processor only.

[percydaman]

thats weird. At home I use win2k, mostly because I just dont want to go out and purchase winxp when I think win2k works fine. I have winxp at work though and just end up tuning it to look and hopefully work like win2k anyways...

[ihavenofish]

yeah, on single intel machines, i tend to not notice any differences between os's.

the machines i have seen issues with all seem to be either dual processor, or single cpu machines with mad chips on via boards. i imagine its mostly related to the quirks between all the different architectures.

i have 3 sets of machines where i could show quantifiable results. one being this fellows dual opteron system showing an 8x difference from xpsp2 to 2ksp4. the others are my own dual pentium 2/3 systems. once i tweaked the os in the 450, its returned to a respectable speed. i forget, but i think it was near a 25-30% difference. the last was my amd 1200 system on a via chipset. i would get various discrepencies with different via drivers. one acpi was toasted, no issues, and a near double rendering speed increase.

i imaging its going to varie from machine to machine and how they got set up. i think maybe an initial first step here might be to get an ol benchmark scene (make it a 10-20 minute scanline scene so everyone can play) and make a list, grouping all the systems and guaging discrepencies. this way if 2 people with dual opteron 246's for example are getting way different render times, we can start picking at other hardware and operating system configurations. who knows, we may find a common thread and be able to help a few people get their investments up to speed.

it'd be one thing if every one with a xeon got 50% faster renders than with an opteron in a given scen/renderer, but the results i've seen here and most other places lead me to see people the the same machines getting way too different results. be nice to get to the bottom of it.

later