New rendering computers, V-Ray, render farm

[fasterG]

Hi,

This thread is going to be techy but I hope it's ok since it's all about rendering in V-Ray.
I've realized I need more rendering power, more cores, so I'm planning to buy a couple of extra computers. My mainstation now is a dual Xeon E5 2687W system which works great, with a Titan and 64 GB RAM.

The brand new dual Xeon E5-2699 v3 (18 cores*2) is simply too expensive. It was too good to be true.
So I'm looking for alternatives. Comparing bang vs buck it seems that a couple of fast i7's are better than dual Xeons.

Right now I'm choosing between a pair of single CPU systems consisting of:
i7 5960X (8 cores) or
i7 5930K (6 cores) or
i7 5820K (6 cores).
The latest being a lot cheaper even if the specs seems to be up there. Have to investigate that more.
32 GB per machine.

And here is where I need some advice:

1. Do the arguments above seem reasonable? Good choices?

2. Graphics cards. How important are they in a render farm? Do I need Quadros/Titans in all of them or is it ok with less advanced cards since rendering is all about CPU power?

3. Licenses and nodes. I have an ordinary V-Ray for Maya licence, not yet upgraded to 3.0. Does the licence include possibility to use more than one computer (how many?), or do I have to buy more licenses? Dongles on each computer? Maya installed on every computer?

4. Main workstation. I know Maya is great at using all available cores, but other programs like After Effects certainly is not. I'm doing all comp and motion in Ae, so one thought is maybe to switch main computer to one of the new ones (lets say switch my dual Xeon 2687W to an overclocked i7 5960X 8-core). It would probably be a lot faster in After Effects, but slower in Vray RT rendering. I guess it's a decision between which program to prioritize?

5. Windows 7. Still good, no reason to switch to Win 8?

Thanx!

[jstrob]

1. for sure the xeon v3 is too expensive but there are other xeon out there that are less expensive. I personnaly chose dual xeons for my render farm and I don't regret it. I never tried i7 but I prefer to have 2 cpu in a single computer. I put them in blades in a rack and they use less space, less exlectricity maybe and probably less heat. I say probably for all cause that 's the type of things that could have changed since I last check. the less computer I have the less I have to take care of for software installation etc.

2. You don,t need any graphic card for render nodes. except if you want to build a gpu based render farm which is completely something else. If you buy rackmount computer they already have an onbaord VGa which perfectly does the job.

3. I use max so I don't know for maya. And check with the sales at chaosgroup to know about their licenses.

4. It would be something to test but my guess would be the xeon will be faster with AE too. I think they improved it a lot in the latest versions in that regard. You also have to configure ae correctly and it's a bit counterintuitive. On the Adobe forum there is a thread about that.

5. yes don't switch to windows 8. I heard Win 7 is better for what we do.

[Neilg]

go for i7's, xeons are more powerful but nowhere near in line with the price difference. you're talking 4 times as much in cost for double the speed. if you need that in one box maybe it's worth it, but you're building a farm and the money has better uses.
you dont need gfx cards, but if you did get one for a workstation i'd get one of the new gtx 970 or 980's
you wouldn't notice a speed difference between the xeon or i7 in AE without getting out a stopwatch and doing side by side tests, and it wouldn't be much. i'd put money on it.
stay on windows 7.

[fasterG]

Ok, no extra graphic cards needed, and stay on Windows 7. Great!

i7 vs Xeon seems to be quite even then. Of course compact is good since we're a small studio, but not if the prize difference is TOO crazy. Should I use small/medium towers or rack mounted (racks are really noisy I've heard?)?

[Neilg]

i7 v xeon is just something you're going to have to think about what makes you more comfortable. of course if you upgrade to vray 3 you'd need fewer render licenses if you had xeons, but that still probably wouldnt offset the cost of them.

we built a bunch of i7 medium towers and stacked them over in a corner. they have a couple large fans in each and decent front->back airflow, they're pretty quiet. the switch in the same corner is louder than them.
I think thin racks are more expensive/noisy/difficult to cool, but I know you can get half desktop/half rack cases that take up 3 slots which you can fit normal pc parts in. no idea what they're actually like to use.
You pay a premium for racks because when you're getting 50 nodes, boxes aren't an option. For a few extra nodes you're better off with standard cases.

[jstrob]

The choice also depend on the electricity and space you have available.

When I started my render farm it was in a closet in a small apartment and the electricity supply was lacking. So I had no choice but to buy dual xeons in 1U rackmount to save space and power outlets. If you build a i7 render farm and one day you need more computing power and you have no space and your electricity box is fully loaded you will maybe regret you did not chose dual xeons. Now I have a lot morespace cause I bought a house with a basement. But still in a small rack I bought for 800$ I can put approx 30 1U with 2 cpu each. If you try to do the same with 60 i7 towers you would need a whole wharehouse for space and 60 outlets instead of 30.

You can see my render farm here:
http://www.strob.net/2014/04/07/comm...n-render-farm/

Also I think you can't have ECC memory in i7. It may be a point to considerate.

To know the exact price difference you will have to compare exact cpu models and in fact compare 2 entire builds cause prices change fast. And prices have big differences from store to store. I saw the same xeon cpus being sold with prices difference in the range of thousands of dollars.

For example I just bought 2 of these at tiger direct:
http://www.tigerdirect.ca/applicatio...=IE2-101794009

For 1907$(canadian dollars) each.

The same exact cpu is selling for 3213 $ CAD on amazon...
http://www.amazon.ca/Intel-BX80635E5...X80635E52680V2

And even better 3355.32$ CAD on hookbag... http://www.hookbag.ca/product/HEC0MK74Q/?p=2322

So you really have to look around for each piece that you buy. I did that and I bought my last workstation pieces from 3 differents stores and saved hundreds of dollars.

Another point to considerate for the workstation is that I heard that Phoenixfd can be faster on i7 compared to xeon. But I don't know how the comparison was made and I'm a bit skeptical about this. It's just because phoenix is not able to distribute the load evenly. But for vray it uses 100% of the power you give him so xeon is way faster.

you can also take a look at this chart to see how fast are the xeon compared to i7. not only xeon are always in advance in terms of speed but you can put 2 of them in same computer. So no wonder why they cost more. you can do more with them.
https://www.cpubenchmark.net/high_end_cpus.html

[fasterG]

Strob: Thanx for sharing. Yea two dual Xeons would definitely be more attractive and compact than 4 i7's, especially since I'm a small studio, but is it worth the prize difference? Maybe, I'll look into that again.
Is medium towers the way to go if rack mounted is not an option? Can I use a smaller size or will the airflow and cooling suffer?
Btw, why all the tinfoil around your farm? Is it unhealthy to be close? I was planning to keep my new computers in my work office

Neil: It seems that there is an extra cost to get 5 render nodes along with the 3.0 upgrade, but its not a major charge. Exactly, a bunch of i7 medium towers sounds nice. How does it work, are they active only in batch rendering or can you make them contribute even in RT rendering and scene/light setup?

Cheers!

[jstrob]

About thinfoil: I used it because it was the cheapest paper at the hardware store and I needed it to cover my soundproofing panels that were made of a kind of green and soft material. Those green sound proofing panels were deteriorating only by touching them so they needed to be covered. When I opened the door of my closet there was a very loud noise from a huge UPS I was using to bring power to my farm from a cloth dryer 240V outlet. And also from all the 1U rackmount computers. Those 1U use many very small and noisy fans. But as soon as I closed the door it was almost completely silent because the door and all the walls were covered with these sound proofing panels.

And obviously my closet needed a good ventilation. I was lucky there was an hole in one of the wall near the ceiling in my closet that was leading to an old and now unused chimney. It was a big enough empty space to redirect the heat using a fan mounted on the wall to this hole. It was my air outtake and I needed also an intake, so I used my cloth dryer machine pipe and I made a hole another lower hole this one near the floor that was bringing fresh air from outside. Since I live in canada it was really working well during winter when it was -40 celcius outside. during summer I was using a portable air conditioning unit specifically made for computer room (it needs only an air outtake and does not create much water).

About tower instead of rackmount I think there not a lot of other choices... For cooling I strongly suggest Noctua fans. They are the most silent I know and they work very well for years. And todays cpu are able to endure heat a lot so don't be too much concerned abvout overheating. But be concerened about noise. You don't want to build a render farm that will drive you nut because of the noise. And you may need a air conditioning unit once you get a lot of computers in a room. And if you live in a hot place heat will be more a problem for you than fo the cpu.

I heard that big companies with huge computer rooms calculated they save more money by letting the cpu run hotter. Maybe the cpu will last a little less if it's too hot but the electricty bill will be much lower. and anyway you have to change the cpu after a few years to increase the speed. but you still have to leave them below their max temp. for this use a utility like Core Temp.

[fasterG]

-40 Celsius is a good rendering environment : )

Yea I have a good tech support that will build the computers. I'm just wondering if mini towers or small form factor will work, or if mid towers are needed for safe cooling? With i7 that is.
Dual Xeons need at least mid tower to fit I guess!

[jstrob]

-40 Celsius is a good rendering environment : )

Yea I have a good tech support that will build the computers. I'm just wondering if mini towers or small form factor will work, or if mid towers are needed for safe cooling? With i7 that is.
Dual Xeons need at least mid tower to fit I guess!

In fact if you put any cpu in a smaller form factor you will need smaller fans that are noisier. Mine are in very small 1U rackmount so they need very small and noisy fans but I don't mind cause I put them in an enclosed and soundproofed room.

If you will have your render farm in the same room where you work, you would better choose the best and quietest fans available. The Noctua are a very good choice. Just look at their Db rating and choose the lowest. Choose them as cpu heatsink fan and also case fan. And then just choose the smaller case that can fit those.

You can see those on the pic below (the beige fans are noctua) I think I put too many of them in this case though. Only 1 fan on the heatsink and 2 at the top would have probably be enough. also on this build I removed the ram fan, they are completely useless and they became too noisy after only a few months. without them the temp is the same.

[Neilg]

In fact if you put any cpu in a smaller form factor you will need smaller fans that are noisier. Mine are in very small 1U rackmount so they need very small and noisy fans but I don't mind cause I put them in an enclosed and soundproofed room.

This also played a bit part in our decision to go with boxes - these are in the corner of our office, so super loud fans are very noticable. there's a shared server space in this building which we rent by the cubic foot (lol) but it's a mess so we just keep our super noisy main server in there. There's a couple win 95 servers ticking over in there so it's like parking a ferrari next to a burnt out shell in the rough part of town.

[fasterG]

A quick update:
I got the farm installed yesterday. There was some struggling to get the license working for the slaves, but finally its done. Distributed V-Ray 3.0 rendering for the win!

Since its a little urgent I continue with another thought.
Next project involves some kind of smoke and flames into tracked footage. Smoke growing up around a building.
I feel this is the right time to learn creating these kind of dynamics in 3d, instead of relying on post stuff.

My first idea was of course to continue the Chaosgroup road and buy Phoenix FD for Maya! But due to the lack of tutorials and other info about the Maya version, I'm now thinking of trying some Maya fluids instead.
A requirement is definitely still to render everything in V-Ray. I haven't investigated that deeply yet but:

* Can I still use V-Ray to render the Maya fluids? Easy setup, and possible without Phoenix?
* V-Ray 3.0 doc says: "New Phoenix FD Shaders with support for OpenVDB and Field3D". I guess that implies buying a Phoenix license as well?
* Is using Maya Fluids a good way for now, and update to Phoenix later? In other words, is Phoenix FD for Maya improving and getting updated? Any new version coming up? (The new Max version looks great!)

Thanx!

Edit:
After seeing some really cool Phoenix shots I think I'll give it a try.. a good learning resource is appreciated

[svetlozar.draganov]

*Can I still use V-Ray to render the Maya fluids? Easy setup, and possible without Phoenix?

Yes, it is possible to render Maya Fluids with V-Ray without Phoenix FD.

* V-Ray 3.0 doc says: "New Phoenix FD Shaders with support for OpenVDB and Field3D". I guess that implies buying a Phoenix license as well?

Phoenix FD requires license only for the actual simulation. You can render Phoenix FD caches with the Phoenix FD demo version as well as with VrayVolumeGrid node available in V-Ray 3.x without Phoenix FD license.

* Is using Maya Fluids a good way for now, and update to Phoenix later? In other words, is Phoenix FD for Maya improving and getting updated? Any new version coming up? (The new Max version looks great!)

Yes, Phoenix FD constantly improves. Most of the features available in Phoenix FD for 3DSMax are also available in the nightly builds of Phoenix FD for Maya.
It is true that the latest official build was released in 2012 but our internal builds update everyday.
We are not able to provide time frame for the next official version.

After seeing some really cool Phoenix shots I think I'll give it a try.. a good learning resource is appreciated

As a start you can visit Phoenix FD playlist in our YouTube Channel here:
http://www.youtube.com/playlist?list=PL9FF6FAF134201935
Most of the tutorials are for Phoenix FD for 3DMax but the same principles could be applied in Phoenix FD for Maya as well.

[t.petrov]

V-ray 3.0 ships with the same shader used by Phoenix FD and this shader is used for Maya fluids.
V-Ray 3.0 also adds a VRayVolumeGrid node that can be used to render OpenVDB and Field3D files.

[LL_Will_J]

When you say that ''V-ray 3.0 ships with the same shader used by Phoenix FD and this shader is used for Maya fluids.'' - how is this implemented? are all the Phoenix FD shader attributes read from the Maya fluid settings? Is there a way to find out what is supported and what isnt? For example fluid textures?

Also does this mean that the shading attributes in the VrayVolumeGrid are the same as the attributes in a Phoenix FD shader?

Thanks!

Will