Network rendering - new 32 core nodes issues

[chrisjacks21]

Right, having some strange issues here network rendering frames through backburner (not using DR).

We have just bought 2 new (and very expensive!) render nodes which are built around the brand new Xeon e5 chip, spec below:

Dual Xeon 8C E5-2680 2.7ghz (16 actual cores, 32 virtual in total) with 32GB ram.

Our latest workstations for comparison are single i7 3930 3.2ghz (6 actual cores, 12 virtual) with 32GB ram.

We had the new nodes installed last week and they have been working fine - albeit not quite as fast as we thought in comparision to the i7s, but still quicker by about 30/40%.

Today they have started playing up, whilst rendering 200 frames, the 2 new nodes progressively rendered each frame slower than the last, this continued and render times went through the roof - meanwhile, the i7 workstations rendered the same frames at a fairly consistent rate.

NOw on another test I have rendered today, the new nodes are rendering slower than some of our older workstations! Render times are sporadic.

Anyone have any ideas/experienced anything similar or maybe have the same new e5 processors?

Using max2010 and Vray 2.20.03

[super gnu]

heya chris.. sounds like an overheating issue.. the clock speed can be throttled if the chips hit their thermal limits. can you install cpu-z or realtemp or something and note the core temps/ clock speeds whilst rendering?

its the only thing i could think of that would lead to progressively reduced performance like that.


also, quite surprised that a pair of 8 cores at 2.7 are only 30-40% faster than a single six core at 3.2. however if the temps are limiting things on the 8 cores, then they will not turbo boost up to their highest overclock..and may even be permanently underclocked. whereas the i7 might end up running at 3.8 or something.

[chrisjacks21]

Hey Robin

THanks for that - cetainly will look into overheating - they are in an air-conditioned server room mind you.

This issue aside, I too was surprised they didn't run much faster than the i7s...

[chrisjacks21]

hi robin

just checked and they are not overheating, so back to sq 1!!! arrrghhhhh. any other thoughts??

[chrisjacks21]

anybody else using the E5-2680 chip?

[rivoli]

do they consistently make use of all threads in the tests you made, rendering at 100%, or is there any unused core/thread left?

[chrisjacks21]

do they consistently make use of all threads in the tests you made, rendering at 100%, or is there any unused core/thread left?

Hi - yes they are using all cores at 100%, there is plenty of RAM, but buckets are several times slower than i7.

[rivoli]

I thought that maybe, depending on the resolution of the test images, they could make use of all threads only at the beginning of the rendering and quickly being reduced to whatever number they could fit in the vfb ending up slower than other machines. but well, that was just a guess..
bucket for bucket I guess the i7 could be faster though, given the difference in clock speed.

[Inaxis]

Hi Chris!

Try to disable the hyper threading in BIOS/UFI.

[vlado]

We are now trying to get a configuration with two E5-2680 processors to see if there are any issues, but it will be a while before we get our hands on it. Until then I can't really help much; the progressive render time increase in particular is very odd.

Best regards,
Vlado

[chrisjacks21]

Hi Chris!

Try to disable the hyper threading in BIOS/UFI.

WIll give that a go inzxis - but surely will lose render power doing that?

We are now trying to get a configuration with two E5-2680 processors to see if there are any issues, but it will be a while before we get our hands on it. Until then I can't really help much; the progressive render time increase in particular is very odd.

Best regards,
Vlado

Hi Vlado - thanks...i hope i can get this sorted soon, spent a lot of money on these servers and deadline is looming!! will report back on turning off hyperthread...

[chrisjacks21]

So.....turning off hyper threading in bios has stablised the render nodes, getting consistent render times now - However, as you would expect, they are not rendering as quick as they should - they are now rendering pretty much same speed as the i7s.

Somebody has suggested that it could be an issue with windows 7 on the nodes affecting the cores (nodes are IBM x3550 M4 servers) and windows server 2008 might work better?

Would be good to know if anyone has used this processor or any 32 core processor with Vray???

[Dariusz Makowski (Dadal)]

Heya

Hard to say to be honest. Try running max on them and start render 1 by 1 by hand see how it goes. 1st of all try mental ray, then other rendering programs - to see if its vray or not vray related...

A side note 2x PC with 1x 8 core xeon will be faster than 2x xeon inside 1 PC.

[vlado]

In addition to what Dadal said, try simpler scenes where you only have regular geometry and brute force GI and/or area lights (no proxies, displacement, bitmaps etc - these features require some thread synchronization, which may be a bottleneck).

Best regards,
Vlado

[chrisjacks21]

In addition to what Dadal said, try simpler scenes where you only have regular geometry and brute force GI and/or area lights (no proxies, displacement, bitmaps etc - these features require some thread synchronization, which may be a bottleneck).

Best regards,
Vlado

Vlado - I did try a very simple scene earlier, although it did have bitmaps and GI was irr map on single frame (but no proxies, displace etc) still had same issues. Will try again as you and Dadal have suggested when I get chance, they are rendering frames for a deadline tomorrow currently! The main scene I have had the issues with, is a monster, really heavy on xrefs and vray proxies.