How far does DR scale in terms of # of Nodes vs. Networkspeeds?

[instinct]

Hey there,

i was wondering if anyone had any experience when using DR with RT is stopping to scale due to network overhead in a 1GBit environment. As in
how many nodes can i add in before they do not add performance anymore.

Regards,
Thorsten

[Joelaff]

Not sure about RT, but Distributed rendering with regular VRay is limited to 10 nodes.

[super gnu]

thats 10 nodes per vray license.. if you have 10 licenses you can use 100 nodes.

i would suspect that network limitations would be very scene dependent.

if you wanna avoid the bottleneck you can either gang multiple gigabit ports together, or get 10 gb ethernet (costs a real bomb) or do what i did and go infiniband. 10 gb adapters cost about 40- 50 euros on ebay ( ripped out of old supers i guess.. the latest infiniband cards are over 40 gbit) -as opposed to 4-500 euros for 10gb ethernet. and you can get an infiniband switch for a few hundred if you look at the right time.

i only have 2 machines so i dont have a switch.. i do get almost instant start with dr whatever the scene... and RT is nice and responsive...

only downside is a total lack of support/ documentation or knowledgeable people... eve if you can find someone who knows infiniband, *nobody* uses it with windows!! -does work though.

[instinct]

thats 10 nodes per vray license.. if you have 10 licenses you can use 100 nodes.

i would suspect that network limitations would be very scene dependent.

if you wanna avoid the bottleneck you can either gang multiple gigabit ports together, or get 10 gb ethernet (costs a real bomb) or do what i did and go infiniband. 10 gb adapters cost about 40- 50 euros on ebay ( ripped out of old supers i guess.. the latest infiniband cards are over 40 gbit) -as opposed to 4-500 euros for 10gb ethernet. and you can get an infiniband switch for a few hundred if you look at the right time.

i only have 2 machines so i dont have a switch.. i do get almost instant start with dr whatever the scene... and RT is nice and responsive...

only downside is a total lack of support/ documentation or knowledgeable people... eve if you can find someone who knows infiniband, *nobody* uses it with windows!! -does work though.

Well 10G already gets REALLY expensive very quickly if you need to have more ports and need to integrated with existing infrastructure and security regulations hehe.
Anyways. I don't even see instant start locally. So i wonder if we are talking about different things. Do you mean instant start in terms of "starting to load" or as in "starting to render".

In addition i was not so much referring to transferring the scene to the nodes but managing the communication of what to render and getting image data back.

I am actually seeing really long load times even locally. What are the key factors for clicking render to getting image data? Can this be reduced by using proxies or alike? How does standalone handle that?

Regards,
Thorsten

[super gnu]

when i say "instant start" i mean it starts contributing almost immediately after the host starts.

have you checked how much bandwith you use when starting a dr render? are you maxing out your gigabit?

im waffling on here but i dont actually know the answer to your question :P . i was merely suggesting an alternative should you be in a position to choose. i get close to 600 meg a second when dr is initialising, so it can definitely use as much bandwith as you can throw at it for this step (up to the limits of your storage system i guess.. i dunno if anything is transferred directly from ram.. that would be neat.

when it actually gets going, i guess the overhead would depend on how fast buckets are being finished, and assuming compression, how much is in them.


wrt RT specifically i imagine its a lot down to your ray/bundle size etc.. make the bundles bigger might reduce network overhead as the machine spends longer working between sending data, if i understand correctly.

[vlado]

i was wondering if anyone had any experience when using DR with RT is stopping to scale due to network overhead in a 1GBit environment. As in how many nodes can i add in before they do not add performance anymore.

It depends on your scene actually; the more work there is to do, the more efficient the DR process will be. Also, you can adjust the bundle size/samples per pixel options to tune the bandwidth usage.

Currently one huge bottleneck is the transfer of the initial scene to all the render nodes - this grows exponentially with the number of nodes right now. It can be avoided to a large extent by using proxies.

Best regards,
Vlado

[instinct]

Cool. Thanks a lot Vlado. What about the "interactive" resolution. If performance is good enough will the switch to lower resolution disappear?

[vlado]

We might be better off with an option to turn it off...

Best regards,
Vlado