We are in the process of researching a 10 machine renderfarm here at work. Seeing that some people have had issues with some machines, ie hyperthreading and such, Im interested in anyones experiences and recomendations for machines that have been very stable with vray. The boss wants to get those little shuttles with like intel 3.0’s or something, but told him about issues with hyperthreading. Any help or info would be greatly appreciated!
I built 6 shuttles - one of the better ones with the nforce3 boards - 3 of the 6 work extremely intermittently…maybee i just got a bad batch but i havent been happy
i’d avoid shuttles…while they are cute it’s annoying to upgrade or repair them. and they are quite noisy, because of the un-smart fan control…under load they quickly get hot and start whining.
Why not move away from Intel and get AMD chips? I just got a dual-Opteron system at home and it’s blazing fast. Tyan makes a nice dual-opteron 1U rackmountable server case - the Transport GX-28.
Remember - with Vray you get 10 render node licenses, but unlimited CPU licenses. So, if you outfit your renderfarm with dual-processor servers you can get 20 extra CPU’s to render with. Plus, the Opteron’s are ready for Win XP 64 (whenever they decide to release it, along with a 64-bit max and 64-bit Vray!).
I don’t understand why U mount units are so expensive.
You can get a $500 Athlon 2 Ghz off the shelf box at Costco witch renders more than decently, completed with RCT 17"screen…Why a U1 server of same Mhz costs so much more? Is it the Mobo? The RAID, if any?
With $5000 you could get 10 CPUs…
regards,
gio
the extra engineering that goes into making them work well in terms of airflow and not overheat when stacked with hundreds of other machines - unlike that cheap shelf box
That cheap shelf box also has much worse components (chipset, ram, drives, etc) than you are going to find in machine built for stability with high-end components.
Percydaman,
I have done some extensive research / testing and will share my experinece / pains.
We currently have 21 computers that are used for rendering. They are comprised of; Athlon, Xeon, Pentium and Opteron CPU’s. The Mix is as follows:
7 x Boxx 1u render slaves
1= Dual Xeon 2.4ghz, 1gig ram
4= Dual Athlon 2800, 1gig ram
2= Dual Athlon 2000, 1gig ram
3 x Home made 2u render slaves
3= Dual Athlon 2800, 2gig ram
3 x Boxx Workstations
3= Dual Athlon 2800, 4gig ram
2 x Home made workstations
1= Dual Athlon 1900, 2.5 gigs ram
1= Dual Opteron 250, 6gig ram (winxp 64)
6 x Shuttle boxes.
3=Intel 2ghz, 512meg ram
1=Intel 3.06ghz Extreeme, 1gig ram
1=Athlon 2000, 1ghz ram
1=Athlon 3000, 1ghz ram
My experience has been as follows:
We render mainly exterior Arch Vizualizations (retail) that are printed large format at high res. We mainly render at 7000x3500 pixels and need as much horse power we can get to crank these out in short order. (typical “FINAL” rendering time for us is 30min - 2 hour, 2-4 million poly scenes…most of the polys are high res cars.) All landscaping is done in PS.
If you can afford it, I recomend purchasing from Boxx. They will cost a bit more but your paying for service and reliability. The only problems we’ve had were with the 980xgl video cards and they over-nighted new ones to us without question. If your working on short deadlines, this can be critical. You’ll also spend a lot less time trouble shooting problems. At some point you have to ask what your time is worth.
That being said, like the rest of the world, I have to live within a budget and have opted to build my own as needed. Generally, I find that I can build dual machine for $1,000 less, with better componants, than when I purchase from Boxx. There has been some headache getting them up and running but once on-line, they have all run flawless. I always make sure I purchase the highest quality parts I can get so as to remove any potential hardware issues. The shuttle boxes were originally purchased to keep email, word, excel, etc off of our workstations. I believe this has helped tremendously with stability and software confilcts. Of the 6 we own, we’ve had 1 power suppy fail which while shuttle fixed on their dime, however it took over 2 weeks. We also have a 2nd unit go offline and have not had a chance to diagnose the problem.
On the performance side, the AMD’s have outperformed the Intel’s on every project. I run performance tests with current projects whenever we get a new machine. The on-line reviews all say that Intel is better with MAX but that has not been the case for us. I can get more horse power for less or the same cost with AMD. This may not be the case for everyone??
Headaches, Trouble shooting, problems… When we 1st started, we performed all our rendering in VIZ via backburner “region net render”. We would submit the jobs out to the entire network and render across all the machines sucessfully. At the time, our scenes were probably maxing at around 800,000 polys and we would render only to 3000x3000pixels. We weren’t running any GI, just the VIZ sun/sky setup. Rendering would take forever (several hours) but it was pretty painless and ran well even on the Shuttles with 512 meg ram (albiet a bit slow!!)
As our scenes got larger/more detailed and we outgrew viz, we eventually turned to MAX and Vray. While we gained an immense amount in the ability to do more faster, we also have a lot more problems with hardware. I can not get the shuttle boxes to render via Backburner or DR reliably. Some projects they work fine, others are a big pain. This may only be an issue of ram as I would say that for us today, any computer with less than 2gigs of ram is unreliable.
To wrap it up; we’re a small company and as we continue to grow, I find that its more valuable to have my employees spend their time working vs fixing/trouble shooting. My recomendation is to spend the extra money up front as the reliablity later will = more productivity/billable hours/profit etc.
I appologise for the lengthy post/rambling. I hope this helps.
Cheers,
You can get a complete mATX (microatx) machine with P4 3.2, 1GB ram, 40GB disc, onboard lan and onboard vga for around $500 here in Norway, id guess its even cheaper in the states.
10 of these machines will set you back a mere $5000, thats just about a highend workstation. and 10 x P4 3.2 kicks a** in a renderfarm.
and if you drop the 40gb disc and add a GB network card instead and set up a lan boot thingy server you are in administrative heaven and can add more boxes at will with zero config…
just my two cents…
nagboy
thanks for the info guys. We’re definitely looking at possible dual processor setups. Were just looking for specific mobos, etc that will be stable. If people are having issues with shuttles then maybe we wont go for those. I wish we could get the hyperthread issue worked out as its a huge consideration on what brands to buy and not buy.
any more specific info would be great. if we plan on building these ourselves, then mobo brands and such that have worked well would be very helpful. ![]()
thanks again.
Well I have built 4 Dual Xeons now with the Supermicro X5DAL-G mother boards and all have worked seamlessly with hyperthreading. The 1st one has been running steady since Nov 2003 and has renderd thousands of frames with HT enabled. Also from what I have heard from others using these boards have been rendering successfully with HT on. And its an ATX formfactor so you can put it into a regular case ![]()
-dave
I’m going to agree with Lucetius. My employer spent the extra cash to my me a BoXX for my latest workstation because the time wasted in building and supporting a home-built box (my previous workstation) ends up costing the company more than difference.
I’ll add another voice for the ‘Don’t skimp on service’ crowd. Being able to call up a vendor and have them overnight parts (If you’re technically inclined..) or being able to ship the whole machine back and have a quick turnaround is an amazing savings in the course of my 3 years with 20 boxx’s
They’re starting to show their age, but 3 years of the solid computer hell that is rendering.. well.. Its been worth it here.
Cheers.
I’m facing some of the same problems too, about to add some more machines.. But I can’t figure out which is the best.. Lucetius do you have any bechmarks/render times comparison on those machines, you seem to have a bit of everything.
/Thomas
beats me!
i m also bangin my head on the wall…
can t really decide
and the grahix is absoluttly confusin…everybody as its own opinion but few solid ground…
i m quite lost…but i ve to put it together an make a decision by monday…
i think i read somewhere that ILM are using racksaver now.
www.racksaver.com
Suurland,
If you have a scene in mind, (i.e. something similar to what you do to give you a good reference point.) I’d be happy to run a benchmark.
Lucetius,
Don’t have a scene in mind, just wondering if you have some general numbers on which computers is the fastest. Maybe the “Performance index” from the backburner queue monitor would give a good picture.
/Thomas
That number has very little to do with the actual speed of a system, not sure what all it factors in but often on our farm a 600 can have a higher rating than one of the 2.8’s wich is just not possible on a horespower basis…
my experience with render farms and PC’s in general lead me to a few suggestions. i’ve had my own persnal “farms” ranging from a few to maybe 7 or 8 machines, and also worked at a place with a farm of near 200 machines. each scenario has its own quirks it seams.
first and foremost, make sure all the machines are the same, or at the least, the same type. this make it infinitely easier to administer. you could set up a ghost drive to rebuild any of the machines.
2: if you dont have a dedicated sys admin, or are not up to managing finiky pc’s while trying to meet deadines, get someone else to build them. a reliable service partner that can get you help fast. this could be boxx, or you reliable local vendor, doesnt matter as long as you can lean on them on a sunday night when 3 machines tank hard disks ![]()
3: do not skimp on a few things… motherboards-if these go, you can be totally *ucked for days. get something nice, but not so rare you have to order them custom from taiwan… ram- the last thing you need is mysterious crashes… hard disks- small and cheap, but long life. HD speed seems not to matter terribly in a farm… video cards- PAIN in the ass. use a high quality old school nvidia or ati card so you wont get driver issues.
4: cases are important. and yes, they can cost more than the rest of the PC. make sure you have adequate cooling (case and room) and that the power supplies are built like tanks. since we were not space limited, the megafarm at the place i used to work opted for standard tower cases and little regard for electricity consumption. the down side was that in the end, they installed a separate massive AC unit just for that room to keep it cold. its debatable i think whether they actually saved money on cheaper cases. they also have an entire department to watch over them and keep them up and running.
5: if your farm is getting big, over say 10 machines, make note of your network, especially if you are rendering massive frames and files. gigabit becomes a must at the backbone (though not necessarily on each machine)
6: monitoring and admining station. once you get above a certain amount of machines, the simple KVM approach begins to get bothersome. this needs a solution of some sort.
7: think of the futrue… if you are buying a whack of machines, you want something you wont toss in 1 year when you get the next batch. today, its seems that something with 64 bit support and room for more ram is the way to go. also agp 8x and/or pci express may become a feature point if gpu accelerated rendering takes hold.
8: multi cpu is also important, as in the end it can save costs not only in hardware, but in space, cooling, and software licenses.
9. software will become important on a farm, particularly the OS. those miniscule (or major) speed and relibility differences between win2k and and XP and the service packs along with setup will begin to mean something. you should take the time to find the best setup for the farm and then keep it. this is why its nice to have all the same boxes… you can immediately tell if one is mis behaving.
10. if you can, get a dedicated box with a raid array and psycho ethernet for saving frames to. the means you are free to crash your workstation at will and not worry about messing up the farm saving files. you may also want your actual max files and textures on this box too, or better yet, another fast network machine.
as for brand of cpu, i am an amd guy, but honestly, it doesnt matter. as long as you can rely on them. believe it or not, raw speed per machine isnt actually the stong point of having the farm(though its nice). you can always buy more ![]()
thats my few thoughts towards farms. sorry for not pointing out specific brands or models, but i honestly think as long as the person you buy it from backs it up with solid service, you wont need to care.
later
Lucetius,
Don’t have a scene in mind, just wondering if you have some general numbers on which computers is the fastest. Maybe the “Performance index” from the backburner queue monitor would give a good picture.
/Thomas
That number has very little to do with the actual speed of a system, not sure what all it factors in but often on our farm a 600 can have a higher rating than one of the 2.8’s wich is just not possible on a horespower basis…
If your slaves are rendering the same animation where each frame in average take the same amount of time the number is infact accurate, atleast it is here. Right now I’m rendering an animation on 8 slaves with 3 kinds of computers and the “performance index” is very accurate when I compare it with the rendertime on the actual machines.
The calculations:
Fastest system takes 87min per frame, the second faseste takes 102min/f and the slowest 173min/f.
Back burner gives me performance index 1 for the fastests of course, and 0,83 for the second fastest, and 0,47 for the slowest.
And if I calculate the rendertimes by hand I get: 87/102=0.85 and 87/173=0,5
So the deviation is 0,02 and 0,03.. I call that pretty damn accurate ![]()
/Thomas