Network rendering - new 32 core nodes issues

[Dariusz Makowski (Dadal)]

Merge refs in ? This should be easy and quick as far as I remember...

[Rhys]

what version of windows are you using on the box? Can you see all cores. I remember an issue we had with windows needing professional or ultimate to see all cores and ram.

[vlado]

Something to consider perhaps - some users sent us their specifications for their 32-core nodes; when we asked our hardware guy to look at them, he noted that they used 1U cases and said there might not be enough room for proper cooling.

Best regards,
Vlado

[tricky]

A wild stab in the dark, but it isn't to do with having enough ram per core is it? If you have 16 (or 32) cores, is 32GB enough ram for them? That's only 2GB (or 1GB) per core.

[jacksc02]

Chris, when the frames are rendering are the CPUs at 100%?

[chrisjacks21]

what version of windows are you using on the box? Can you see all cores. I remember an issue we had with windows needing professional or ultimate to see all cores and ram.

Te nodes are running windows 7 and can see all cores and ram - we are currently installing windows server 2008 on one of them to see if it works any better.

Something to consider perhaps - some users sent us their specifications for their 32-core nodes; when we asked our hardware guy to look at them, he noted that they used 1U cases and said there might not be enough room for proper cooling.

Best regards,
Vlado

Hi Vlado - they are 1u rack mounted, but we did some temperature tests and they seemed to be running fine.

A wild stab in the dark, but it isn't to do with having enough ram per core is it? If you have 16 (or 32) cores, is 32GB enough ram for them? That's only 2GB (or 1GB) per core.

Thanks Tricky - but fairly sure its not the RAM thats topping out (we have run tests with small scenes that use little RAM)

Chris, when the frames are rendering are the CPUs at 100%?

Hey Chris - all CPU's show they are running at 100% during rendering - as mentioned above, we are banging on server 2008 to test - will report back shortly!

[vlado]

Ok then, let us know how it goes... we have ordered a similar configuration but it will be a while until I get it for tests.

Best regards,
Vlado

[Morbid Angel]

well we dont have 32 core machines, but 24 core (logical) and ran into the same problem with vray in maya. While the render time did not get higher from frame to frame, the render time was sporadic, on frames that should have been nearly same. Disabling HT solved that issue, however HT on i7's seem to perform just fine. I was unable to reproduce this with mental ray, but our setup is quite complicated, it would take lots of effort to do accurate test with another render.

[Hos]

We have 8 of the exact same machines, bought them the moment they where released, so far they seem like really expensive paperweights that have the added functionality of running benchmarks really well.

We are still in talks with Chaos support, but so far after testing so many things (literally turning on and off EVERY possible bios setting, every possible memory settings) and having tested on Windows 7, RHEL 6.2, Fedora 16 and CentOS 6.2, rendering in stable and nightly releases, we are not one bit closer to a solution.

We test using actual production scenes, so far, these new machines perform multitudes worse (like 2 to 8 times slower than a X5680, sometimes they even perform worse than our 6+ years older X5355 machines), which is a disaster for us, as we switched to VRay as our primary renderer a while back, and these machines where a huge investment for currently running and future rendering jobs.

As for render times, i'll post those tomorrow, but i can tell that VRay suffers the most from these new processors, around 400% slower than the same scene on a x5680, Mental Ray is bad as well, but around 20% slower than a X5680, Lightwave scenes i tested (nothing truly massive in scene size unfortunately, so it's hard to tell what they do on scenes the size we tested for VRay and MR, so keep that in mind) where around 50% faster than rendering on a X5680.

Coming week we'll probably have time again to look further into this, so we'll keep Chaos up to date on our findings.

Sven

p.s. posted this under our company forum login, as i couldn't remember my own forum login password.

[Morbid Angel]

We have 8 of the exact same machines, bought them the moment they where released, so far they seem like really expensive paperweights that have the added functionality of running benchmarks really well.

We are still in talks with Chaos support, but so far after testing so many things (literally turning on and off EVERY possible bios setting, every possible memory settings) and having tested on Windows 7, RHEL 6.2, Fedora 16 and CentOS 6.2, rendering in stable and nightly releases, we are not one bit closer to a solution.

We test using actual production scenes, so far, these new machines perform multitudes worse (like 2 to 8 times slower than a X5680, sometimes they even perform worse than our 6+ years older X5355 machines), which is a disaster for us, as we switched to VRay as our primary renderer a while back, and these machines where a huge investment for currently running and future rendering jobs.

As for render times, i'll post those tomorrow, but i can tell that VRay suffers the most from these new processors, around 400% slower than the same scene on a x5680, Mental Ray is bad as well, but around 20% slower than a X5680, Lightwave scenes i tested (nothing truly massive in scene size unfortunately, so it's hard to tell what they do on scenes the size we tested for VRay and MR, so keep that in mind) where around 50% faster than rendering on a X5680.

Coming week we'll probably have time again to look further into this, so we'll keep Chaos up to date on our findings.

Sven

p.s. posted this under our company forum login, as i couldn't remember my own forum login password.

Sorry to hear that. Let us know how it goes and if you finally have a solution.

[Dariusz Makowski (Dadal)]

Right, soo any chance for you to try mental ray/scanline render see if same issues happen too? It would be easier if we know if its vray or windows or other related.

[super gnu]

he did try mental ray, and lightwave.... also with windows and linux... says in his post

[vlado]

I should be getting the equivalent machine here later in the week so I'll have more info then.

In the meantime, have you tried running several render jobs at the same time on that machine (potentially through DR), pinned to different CPU nodes? Does that go any differently than running one render job on all nodes?

Best regards,
Vlado

[Hos]

I will try to test with multiple jobs on one machine.
In the last few days I did a whole range of benchmark tests on the new machines with vray. I tested scenes with a huge amount of textures, large amount of polygons, a lot of proxys, transform animation, really large scenes with small detail, high raytrace depth, etc.
None of those showed any big difference in rendertime between our old render machine (dual xeon X5680) and our new render blades (dual xeon E5 2680). In some of the tests the E5 2680 where about 30% faster and most of them the rendertime where the same. What is already not good when you take in account that the E5 2680 has 4 more cores and has a score of 24 in cinbech compared to a score of 14 with the X5680.
But in our production scenes the E5 blades rendered about 4 times slower then our older renderblades (X5680). When I tried to narrow the scene down until I had a really simple scene, I found out that the rendertime difference between our old(X5680) and our new(E52680) was only very large in these occasions:
-When I use a combination of a proxy, a lot of different shaders assigned to the proxy and with shaders with glossy reflections. Only with this combination the rendertime difference is this high. When I have a proxy with different shaders but without reflection, a normal object with different shaders and glossy reflections or just an proxy with a glossy shader the rendertime difference is not that high.
-And when setting the minimum dmc sampling with an higher number.
It seems a bit random, maybe a developer would see some logic in these results?

greetings,

Arjan

[vlado]

-When I use a combination of a proxy, a lot of different shaders assigned to the proxy and with shaders with glossy reflections. Only with this combination the rendertime difference is this high. When I have a proxy with different shaders but without reflection, a normal object with different shaders and glossy reflections or just an proxy with a glossy shader the rendertime difference is not that high.
-And when setting the minimum dmc sampling with an higher number.

It seems a bit random, maybe a developer would see some logic in these results?

Yes, in fact I had already guessed the problem and I believe I have a solution that would improve things, but I need to test it with the actual hardware.

Best regards,
Vlado