DR on 100 slaves

[jonahhawk]

When we first switched to Windows 7 I remember reading that the concurrent connection limit was raised from 10 to 20. There was also a registry setting which allowed setting the number to any number. Back then it didn't matter to us that much as we had fewer than 20 DR slaves. Now that we have around 230 of the firm's workstations on Backburner, we can use any of them when users are logged off. The BB server service is started automatically. We use a command line job to fire up the spawner and to take those machines off of BB while the use is using DR.

If we had both the spawner service and the backburner service running on all the slaves, is there a more elegant way to keep BB from using the slaves while they are used for DR? and visa versa; If BB is using a slave for an animation task or whatever, is there a way to prevent that same slave from being used with DR simultaneously?

I have since read that the latest builds of Window 7 have removed the simultaneous connection limit. So I've been having a lot of fun experimenting with DR to see how many buckets i can get on a single image. I can consistently get more than 500.

Sometimes the Ir precalc pass only seems to use a handful of slaves. I have 100 slaves assigned right now but only 20 or so did the precalc pass. When the render pass kicks in, there are 504 but not 800+ like I would expect. (100 slaves x 8cores).

What are the limits on DR? Do I remember correctly that it is limited by the number of licenses on our dongle?

[Morne]

10 DR nodes per 1 VRay License

[jonahhawk]

aha. Thank you for the reminder. We have 10 licenses. There must be a few others tied up so I'm not getting 800+ buckets. Not that I'm complaining. It's total overkill.

With that many buckets, there is a balance to the size of the buckets; too large and you're stuck waiting for 1% of the last buckets to finish. Too small and the server CPUs don't operate at 100%.

It is still odd to me that there are far fewer buckets on the precalc pass than on the render pass.

Anyone else have experience with DR on this scale?

[irwit]

aha. Thank you for the reminder. We have 10 licenses. There must be a few others tied up so I'm not getting 800+ buckets. Not that I'm complaining. It's total overkill.

With that many buckets, there is a balance to the size of the buckets; too large and you're stuck waiting for 1% of the last buckets to finish. Too small and the server CPUs don't operate at 100%.

It is still odd to me that there are far fewer buckets on the precalc pass than on the render pass.

Anyone else have experience with DR on this scale?

I didn't think you could use multiple machines to calculate pre passes?

[vlado]

I didn't think you could use multiple machines to calculate pre passes?

You can do that since a long time, no problem.

Best regards,
Vlado

[vlado]

It is still odd to me that there are far fewer buckets on the precalc pass than on the render pass.

It takes a while to transfer the scene to all the render slaves; it is entirely possible that by the time the scene is transferred to some of the slaves, the others have already finished calculating the irradiance map. With that many machines, using brute force as the primary engine makes a lot more sense.

Best regards,
Vlado

[tricky]

I had no idea that Windows 7 allowed more than 10 concurrant connections. This was one of the main reasons (a few years ago) that we invested in a server running Small Business Server so that we could have more connections at any one time (you needed to buy CAL licesnes to allow more connections than the standard 5).

Does this mean that when we come to retire our present server, we could replace it with a much simpler (and cheaper) 'server' that simply runs Windows 7 and not have any limit when we do DR and BB jobs?

[Dariusz Makowski (Dadal)]

Just a side note question... From long I was told - and I experimented that Distributed DR is a lot slower than Distributed Tile rendering.

Can u do a test on hard image with ur DR vs distributed tile rendering please? - u have to precalculate GI for tile rendering so it matches btw...

Let me know, thanks, bye.

[jonahhawk]

we do not precalc the GI. We use IR+BF and it matches fine. It's only LC that you have to save out to file.

@Tricky. Yes, I'm sorry to say, you could have been running a simple windows 7 box the whole time. A few years back, we asked IT about getting server OS for our workstations for the same reason. That would have been pretty embarrassing.

Over the weekend I consistently got 500 buckets from my workstation. The reason I didn't see 800+ was that a few people left max/vray open and were taking up licenses.

I'd also like to add that I was using slaves in four U.S. offices; Portland, Seattle, LA and DC. VRay DR is a very solid tool considering what I put it through this weekend.

I did see low CPU usage on the slave machines. This, I'm guessing, was do to small bucket size and the lag of my network connection, our WAN etc. My theory is that cores were finishing buckets faster than the network could get them all to me... maybe?

I only see 100% cpu usage with larger buckets. Larger buckets means less efficiency on the tail end of the rendering. i.e. waiting for ten buckets to finish when there are 500 others sitting idle.

Vlado, I may be way off on this but would it be possible (or useful) for the slave machine to cache the finished buckets locally somehow so that the CPU could move on and not have to wait for the network to deliver? (humbly, naively guessing here)

[jonahhawk]

Just a side note question... From long I was told - and I experimented that Distributed DR is a lot slower than Distributed Tile rendering.

Can u do a test on hard image with ur DR vs distributed tile rendering please? - u have to precalculate GI for tile rendering so it matches btw...

Let me know, thanks, bye.

Do you mean rendering to Strips via Backburner? We have trouble with strips as we like to use layered EXR, and now VRImg with ProEXR 1.7, no need to convert. Plus, with DR and IR+BF, we don't have to save a precalc'd file.

[Rhys]

@Tricky, Its annoying to think that you dont need to use windows server, of course the server version have lots more features, I dont know many 3d companies that actually run domain controllers on there server either.
SBS 2011 Essentials lets you connect 25 machines, with out the issue of cals. not a bad choice for studios, nice backup options. Always feel weird running windows 7 on a server to be honest!

[Dariusz Makowski (Dadal)]

Do you mean rendering to Strips via Backburner? We have trouble with strips as we like to use layered EXR, and now VRImg with ProEXR 1.7, no need to convert. Plus, with DR and IR+BF, we don't have to save a precalc'd file.

Yes thats what I mean... I hear from CGI artist where I work here that DBR on their tests were up at around 20% slower. So every 5 machines you get -1 machine on performance in case of 100 pcs thats quite a lot of lost speed...

Still would be nice if you could just run simple test. Tile rendering still can save you render passes but to separate files tho... a simple java script for photoshop could fix your issue of putting them in 1 file or just load in to stack function for PS...