Building a renderfarm

[percydaman]

We are in the process of researching a 10 machine renderfarm here at work. Seeing that some people have had issues with some machines, ie hyperthreading and such, Im interested in anyones experiences and recomendations for machines that have been very stable with vray. The boss wants to get those little shuttles with like intel 3.0's or something, but told him about issues with hyperthreading. Any help or info would be greatly appreciated!

[Dynedain]

I built 6 shuttles - one of the better ones with the nforce3 boards - 3 of the 6 work extremely intermittently...maybee i just got a bad batch but i havent been happy

[plastic_]

i'd avoid shuttles...while they are cute it's annoying to upgrade or repair them. and they are quite noisy, because of the un-smart fan control...under load they quickly get hot and start whining.

[dapeter]

Why not move away from Intel and get AMD chips? I just got a dual-Opteron system at home and it's blazing fast. Tyan makes a nice dual-opteron 1U rackmountable server case - the Transport GX-28.

Remember - with Vray you get 10 render node licenses, but unlimited CPU licenses. So, if you outfit your renderfarm with dual-processor servers you can get 20 extra CPU's to render with. Plus, the Opteron's are ready for Win XP 64 (whenever they decide to release it, along with a 64-bit max and 64-bit Vray!).

[Giovanni]

I don't understand why U mount units are so expensive.

You can get a $500 Athlon 2 Ghz off the shelf box at Costco witch renders more than decently, completed with RCT 17"screen...Why a U1 server of same Mhz costs so much more? Is it the Mobo? The RAID, if any?

With $5000 you could get 10 CPUs...



regards,

gio

[Dynedain]

the extra engineering that goes into making them work well in terms of airflow and not overheat when stacked with hundreds of other machines - unlike that cheap shelf box

That cheap shelf box also has much worse components (chipset, ram, drives, etc) than you are going to find in machine built for stability with high-end components.

[Lucetius]

Percydaman,

I have done some extensive research / testing and will share my experinece / pains.

We currently have 21 computers that are used for rendering. They are comprised of; Athlon, Xeon, Pentium and Opteron CPU's. The Mix is as follows:

7 x Boxx 1u render slaves
1= Dual Xeon 2.4ghz, 1gig ram
4= Dual Athlon 2800, 1gig ram
2= Dual Athlon 2000, 1gig ram

3 x Home made 2u render slaves
3= Dual Athlon 2800, 2gig ram

3 x Boxx Workstations
3= Dual Athlon 2800, 4gig ram

2 x Home made workstations
1= Dual Athlon 1900, 2.5 gigs ram
1= Dual Opteron 250, 6gig ram (winxp 64)

6 x Shuttle boxes.
3=Intel 2ghz, 512meg ram
1=Intel 3.06ghz Extreeme, 1gig ram
1=Athlon 2000, 1ghz ram
1=Athlon 3000, 1ghz ram

My experience has been as follows:

We render mainly exterior Arch Vizualizations (retail) that are printed large format at high res. We mainly render at 7000x3500 pixels and need as much horse power we can get to crank these out in short order. (typical "FINAL" rendering time for us is 30min - 2 hour, 2-4 million poly scenes...most of the polys are high res cars.) All landscaping is done in PS.

If you can afford it, I recomend purchasing from Boxx. They will cost a bit more but your paying for service and reliability. The only problems we've had were with the 980xgl video cards and they over-nighted new ones to us without question. If your working on short deadlines, this can be critical. You'll also spend a lot less time trouble shooting problems. At some point you have to ask what your time is worth.

That being said, like the rest of the world, I have to live within a budget and have opted to build my own as needed. Generally, I find that I can build dual machine for $1,000 less, with better componants, than when I purchase from Boxx. There has been some headache getting them up and running but once on-line, they have all run flawless. I always make sure I purchase the highest quality parts I can get so as to remove any potential hardware issues. The shuttle boxes were originally purchased to keep email, word, excel, etc off of our workstations. I believe this has helped tremendously with stability and software confilcts. Of the 6 we own, we've had 1 power suppy fail which while shuttle fixed on their dime, however it took over 2 weeks. We also have a 2nd unit go offline and have not had a chance to diagnose the problem.

On the performance side, the AMD's have outperformed the Intel's on every project. I run performance tests with current projects whenever we get a new machine. The on-line reviews all say that Intel is better with MAX but that has not been the case for us. I can get more horse power for less or the same cost with AMD. This may not be the case for everyone??

Headaches, Trouble shooting, problems..... When we 1st started, we performed all our rendering in VIZ via backburner "region net render". We would submit the jobs out to the entire network and render across all the machines sucessfully. At the time, our scenes were probably maxing at around 800,000 polys and we would render only to 3000x3000pixels. We weren't running any GI, just the VIZ sun/sky setup. Rendering would take forever (several hours) but it was pretty painless and ran well even on the Shuttles with 512 meg ram (albiet a bit slow!!)

As our scenes got larger/more detailed and we outgrew viz, we eventually turned to MAX and Vray. While we gained an immense amount in the ability to do more faster, we also have a lot more problems with hardware. I can not get the shuttle boxes to render via Backburner or DR reliably. Some projects they work fine, others are a big pain. This may only be an issue of ram as I would say that for us today, any computer with less than 2gigs of ram is unreliable.

To wrap it up; we're a small company and as we continue to grow, I find that its more valuable to have my employees spend their time working vs fixing/trouble shooting. My recomendation is to spend the extra money up front as the reliablity later will = more productivity/billable hours/profit etc.

I appologise for the lengthy post/rambling. I hope this helps.

[nagboy]

Cheers,

You can get a complete mATX (microatx) machine with P4 3.2, 1GB ram, 40GB disc, onboard lan and onboard vga for around $500 here in Norway, id guess its even cheaper in the states.

10 of these machines will set you back a mere $5000, thats just about a highend workstation. and 10 x P4 3.2 kicks a** in a renderfarm.

and if you drop the 40gb disc and add a GB network card instead and set up a lan boot thingy server you are in administrative heaven and can add more boxes at will with zero config...

just my two cents...

nagboy

[percydaman]

thanks for the info guys. We're definitely looking at possible dual processor setups. Were just looking for specific mobos, etc that will be stable. If people are having issues with shuttles then maybe we wont go for those. I wish we could get the hyperthread issue worked out as its a huge consideration on what brands to buy and not buy.

any more specific info would be great. if we plan on building these ourselves, then mobo brands and such that have worked well would be very helpful.

thanks again.

[Syclone1]

Well I have built 4 Dual Xeons now with the Supermicro X5DAL-G mother boards and all have worked seamlessly with hyperthreading. The 1st one has been running steady since Nov 2003 and has renderd thousands of frames with HT enabled. Also from what I have heard from others using these boards have been rendering successfully with HT on. And its an ATX formfactor so you can put it into a regular case


-dave

[Dynedain]

I'm going to agree with Lucetius. My employer spent the extra cash to my me a BoXX for my latest workstation because the time wasted in building and supporting a home-built box (my previous workstation) ends up costing the company more than difference.

[dbuchhofer]

I'll add another voice for the 'Don't skimp on service' crowd. Being able to call up a vendor and have them overnight parts (If you're technically inclined..) or being able to ship the whole machine back and have a quick turnaround is an amazing savings in the course of my 3 years with 20 boxx's

They're starting to show their age, but 3 years of the solid computer hell that is rendering.. well.. Its been worth it here.

Cheers.

[suurland]

I'm facing some of the same problems too, about to add some more machines.. But I can't figure out which is the best.. Lucetius do you have any bechmarks/render times comparison on those machines, you seem to have a bit of everything.

/Thomas

[ene.xis]

beats me!
i m also bangin my head on the wall....
can t really decide
and the grahix is absoluttly confusin...everybody as its own opinion but few solid ground...
i m quite lost...but i ve to put it together an make a decision by monday...

[Da_elf]

i think i read somewhere that ILM are using racksaver now.
www.racksaver.com