Vray - Optimal Render Farm Configuration

[eazyc10]

I have a large number of users that need to render multiple items on a regular basis. We currently have 4 dedicated servers running the DRspawner that render on demand. The team is currently running into issues where A) a spawner is not available and B) the performance is a little subpar. These are newer servers (2010) built for the specific purpose of rendering. Now my question is, I'm looking at adding more machines to the render farm and wondered if adding more machines via VMware on a single host would help out with the issue, or if I truely need to add more dedicated boxes for this.

I'm interested to see what other people have for larger render farm setups.

Eric

[super gnu]

i cant imagine running multiple VMs would help in the slightest. vray is already using all your cpu juice anyway. can you get dealine rendermanager for vray for rhino?! that might be able to manage jobs more efficiently than DR.

[super gnu]

i mean "deadline" no idea why i cant edit my post tonight.

[eazyc10]

I can definitely throw money at it, though I would need to know what benefits the render manager would give us. The current issues with DRSpawner is that there's minimal visibility to who's doing what on which servers as well as they're stating that sometimes the jobs simply don't use all the spawners even if they know no one's using it. Would the render manager solve this?

Eric

[Micha_cg]

Maybe interesting:

http://www.chaosgroup.com/forums/vbu...ighlight=slave

So I can see the process window of each slave at my master machine.

[eazyc10]

Thanks for the link - I'll check it out.

I think sometimes our issues (I apologize, i'm not the renderer nor do I check on this often) are with the spawner running but not actually accepting a job? Is this a common issue? I know that occasionally we'll have two drspawners running on a machine since we have it mapped on logon - could that be an issue?

[wcmansp]

We run a small renderfarm with this architecture -- DRSpawner running as a service (via Microsoft's srvany utility) and Deadline render management software controlling job submission -- but it's scalable. DRSpawner does have problems if jobs are submitted to it without enough time in between -- it either won't respond (so you lose that node's participation) or you may get some quality issues with the render. Using Deadline lets us queue submissions, so jobs are processed efficiently (and visibly -- everyone can tell where their job is), and, because it supports integration of tools like PsExec either as standalone command-line jobs or as pre- or post- job tasks, we can re-start the DRSpawner service on the clients automatically between each job (which we do). Because Deadline is available free for up to two Slaves, and each Slave, using the DRSpawner distributed rendering architecture, can manage 10 additional clients, it's a pretty cost-effective way to go. Deadline also offers a Deadline Toolbar for Rhino that can simplify submitting jobs; we've tweaked ours so that you only need to enter the output filename and type and press "Submit". We're supporting architecture students so we tried to make this as simple as we could for them and we're able to run it 24x7, self-serve (ie. on-demand), with submission from our machines or theirs, on campus or off. If this is something you're interested in I can provide more detail.

[eazyc10]

I'd definitely be interested on any more information you may have on this

Thanks,

We run a small renderfarm with this architecture -- DRSpawner running as a service (via Microsoft's srvany utility) and Deadline render management software controlling job submission -- but it's scalable. DRSpawner does have problems if jobs are submitted to it without enough time in between -- it either won't respond (so you lose that node's participation) or you may get some quality issues with the render. Using Deadline lets us queue submissions, so jobs are processed efficiently (and visibly -- everyone can tell where their job is), and, because it supports integration of tools like PsExec either as standalone command-line jobs or as pre- or post- job tasks, we can re-start the DRSpawner service on the clients automatically between each job (which we do). Because Deadline is available free for up to two Slaves, and each Slave, using the DRSpawner distributed rendering architecture, can manage 10 additional clients, it's a pretty cost-effective way to go. Deadline also offers a Deadline Toolbar for Rhino that can simplify submitting jobs; we've tweaked ours so that you only need to enter the output filename and type and press "Submit". We're supporting architecture students so we tried to make this as simple as we could for them and we're able to run it 24x7, self-serve (ie. on-demand), with submission from our machines or theirs, on campus or off. If this is something you're interested in I can provide more detail.

[TAUBMAN_COLLEGE_UM]

I'll write something up. Depending on your experience with Windows (e.g. running services, registry settings, command-line tools) and, to some degree, with Python (Deadline is largely openly-coded in Python, so highly-customizable) the write-up can be either more complete or more concise. Probably more than is good for a Forum post, in any event, so I'd plan to send you a Forum private message and you could comment back in this Forum item whether you found it useful. Likely by Monday.

[Micha_cg]

I'm interested to know more about Deadline and the advantages for VfR too. Would it allow to get crashed and restarted spawner back to the render process? Can be animation frames distributed as whole frame to the spawner and not at bucket level?

[TAUBMAN_COLLEGE_UM]

This process hasn't ended up crashing the spawners but, because we have DRSpawner running as a service, it does allow the spawners to be easily restarted if they have crashed. The way we've worked it in is that when each of the Deadline jobs starts up we execute what Deadline calls an "OnJobSubmitted" Event script (essentially a pre-job script) which, in this case, runs a series of PsExec commands to stop and restart the DRSpawner service on each of the client machines. It adds a few minutes to each job but it seems to keep the spawners (and their output) stable; if the service is running it stops and restarts it, if the service isn't running (has crashed) it ends up starting it. I haven't seen a case, while watching the V-Ray for Rhino progress window, where a "pinged out" spawner ever re-appeared but, since we synchronize our restarts with the beginning of each job, we haven't actually tested anything that would re-start a dropped spawner during a job to see what happened.

We haven't run animations through it; the V-Ray for Rhino work we get from students has been single-frame rendering.

[wcmansp]

My notes turned out to run about 7 pages. The intro section is below. If the Forum's Private Messages process will handle this much text I'll send a copy to Micha_cg and eazyc10 tomorrow. Their comments about its utility would be useful.


This is a description of a managed rendering environment that combines distributed rendering clients (V-Ray for Rhino’s DRSpawner) running as a service with a full-featured render management application (Deadline 5.1 from Thinkboxsoftware) as the user interface. There can be several advantages to this:

*The full-featured user interface of a professional product provides job submission, queuing and status, e-mail notification of completion/errors, single and cumulative job statistics

*Cost-effective operation, allowing the use of shared (ie. not dedicated) computers along with the ability to aggregate cores (a strength of distributed rendering).

*The ability to script (via PsTools commands wrapped in Python) additional controls on render clients to provide features such as Wake on LAN and powercfg changes (for green operation) and render service re-starts (to promote stable client participation and quality)

The areas involved setting this up can cross some expertise and it’s hard to assume what everyone knows. This is an attempt to provide enough detail that a person without a good deal of expertise (or time) could try it. I’ve tried to separate the basics from some of our specific solutions (which may not be generally applicable in other circumstances). There are “TIPS:” that provide more specific advice/comment on some of the choices. I hope it will be useful in some situations and, ideally, would generate suggestions or extensions.

[Micha_cg]

An other question I ask me - will Deadline allow me to work with an external renderfarm?

[eazyc10]

Thanks for the update - I truly look forward to seeing your write up and appreciate the time you've put into it. If you need to email me directly let me know and I can forward you my email address

Eric

[wcmansp]

It took 7 Private Messages to cover it; I'd probably want to do it another way in the future. I would appreciate any comments you or Micha have on whether you find it generally useful or that it just applies in limited situations.