I have a large number of users that need to render multiple items on a regular basis. We currently have 4 dedicated servers running the DRspawner that render on demand. The team is currently running into issues where A) a spawner is not available and B) the performance is a little subpar. These are newer servers (2010) built for the specific purpose of rendering. Now my question is, I’m looking at adding more machines to the render farm and wondered if adding more machines via VMware on a single host would help out with the issue, or if I truely need to add more dedicated boxes for this.
I’m interested to see what other people have for larger render farm setups.
i cant imagine running multiple VMs would help in the slightest. vray is already using all your cpu juice anyway. can you get dealine rendermanager for vray for rhino?! that might be able to manage jobs more efficiently than DR.
I can definitely throw money at it, though I would need to know what benefits the render manager would give us. The current issues with DRSpawner is that there’s minimal visibility to who’s doing what on which servers as well as they’re stating that sometimes the jobs simply don’t use all the spawners even if they know no one’s using it. Would the render manager solve this?
I think sometimes our issues (I apologize, i’m not the renderer nor do I check on this often) are with the spawner running but not actually accepting a job? Is this a common issue? I know that occasionally we’ll have two drspawners running on a machine since we have it mapped on logon - could that be an issue?
We run a small renderfarm with this architecture -- DRSpawner running as a service (via Microsoft’s srvany utility) and Deadline render management software controlling job submission -- but it’s scalable. DRSpawner does have problems if jobs are submitted to it without enough time in between -- it either won’t respond (so you lose that node’s participation) or you may get some quality issues with the render. Using Deadline lets us queue submissions, so jobs are processed efficiently (and visibly -- everyone can tell where their job is), and, because it supports integration of tools like PsExec either as standalone command-line jobs or as pre- or post- job tasks, we can re-start the DRSpawner service on the clients automatically between each job (which we do). Because Deadline is available free for up to two Slaves, and each Slave, using the DRSpawner distributed rendering architecture, can manage 10 additional clients, it’s a pretty cost-effective way to go. Deadline also offers a Deadline Toolbar for Rhino that can simplify submitting jobs; we’ve tweaked ours so that you only need to enter the output filename and type and press “Submit”. We’re supporting architecture students so we tried to make this as simple as we could for them and we’re able to run it 24x7, self-serve (ie. on-demand), with submission from our machines or theirs, on campus or off. If this is something you’re interested in I can provide more detail.
I’ll write something up. Depending on your experience with Windows (e.g. running services, registry settings, command-line tools) and, to some degree, with Python (Deadline is largely openly-coded in Python, so highly-customizable) the write-up can be either more complete or more concise. Probably more than is good for a Forum post, in any event, so I’d plan to send you a Forum private message and you could comment back in this Forum item whether you found it useful. Likely by Monday.
I’m interested to know more about Deadline and the advantages for VfR too. Would it allow to get crashed and restarted spawner back to the render process? Can be animation frames distributed as whole frame to the spawner and not at bucket level?
This process hasn’t ended up crashing the spawners but, because we have DRSpawner running as a service, it does allow the spawners to be easily restarted if they have crashed. The way we’ve worked it in is that when each of the Deadline jobs starts up we execute what Deadline calls an “OnJobSubmitted” Event script (essentially a pre-job script) which, in this case, runs a series of PsExec commands to stop and restart the DRSpawner service on each of the client machines. It adds a few minutes to each job but it seems to keep the spawners (and their output) stable; if the service is running it stops and restarts it, if the service isn’t running (has crashed) it ends up starting it. I haven’t seen a case, while watching the V-Ray for Rhino progress window, where a “pinged out” spawner ever re-appeared but, since we synchronize our restarts with the beginning of each job, we haven’t actually tested anything that would re-start a dropped spawner during a job to see what happened.
We haven’t run animations through it; the V-Ray for Rhino work we get from students has been single-frame rendering.
My notes turned out to run about 7 pages. The intro section is below. If the Forum’s Private Messages process will handle this much text I’ll send a copy to Micha_cg and eazyc10 tomorrow. Their comments about its utility would be useful.
This is a description of a managed rendering environment that combines distributed rendering clients (V-Ray for Rhino’s DRSpawner) running as a service with a full-featured render management application (Deadline 5.1 from Thinkboxsoftware) as the user interface. There can be several advantages to this:
*The full-featured user interface of a professional product provides job submission, queuing and status, e-mail notification of completion/errors, single and cumulative job statistics
*Cost-effective operation, allowing the use of shared (ie. not dedicated) computers along with the ability to aggregate cores (a strength of distributed rendering).
*The ability to script (via PsTools commands wrapped in Python) additional controls on render clients to provide features such as Wake on LAN and powercfg changes (for green operation) and render service re-starts (to promote stable client participation and quality)
The areas involved setting this up can cross some expertise and it’s hard to assume what everyone knows. This is an attempt to provide enough detail that a person without a good deal of expertise (or time) could try it. I’ve tried to separate the basics from some of our specific solutions (which may not be generally applicable in other circumstances). There are “TIPS:” that provide more specific advice/comment on some of the choices. I hope it will be useful in some situations and, ideally, would generate suggestions or extensions.
Thanks for the update - I truly look forward to seeing your write up and appreciate the time you’ve put into it. If you need to email me directly let me know and I can forward you my email address
It took 7 Private Messages to cover it; I’d probably want to do it another way in the future. I would appreciate any comments you or Micha have on whether you find it generally useful or that it just applies in limited situations.
Just in the mement I found at the deadline FAQ pages:
"Does Rhino need to be licensed on each render node? "
“Yes.”
That’s to bad, an additional Rhino license for each computer of a render farm - that’s to much for me. It’s a pity that my request for “rendering in Rhino demo mode” never got a chance.
That’s true for the standard Deadline configuration but not for this kind of architecture. In its standard configuration, the way the Deadline documentation describes it, there is a Deadline Slave running on every render node. When that Slave picks up a job from the Repository it is scripted to start the application requested in the job (in this case, Rhino) and execute the appropriate rendering commands on that node. In the standard configuration, then, every node has to have the application. In this distributed rendering architecture there is still a copy of Rhino on the Deadline Slave, but now each Slave can now use up to 10 additional rendering client machines that just need DRSpawner running. These 10 machines don’t need to run Rhino and they aren’t running the Deadline Slave software, just DRSpawner. At least on the V-Ray for Rhino version we use (1.05.29, EDU) you can install the DRSpawner client (and the materials and RDK) on these machines without installing “V-Ray for Rhino” (which wouldn’t install unless it could find Rhino) and DRSpawner doesn’t require a V-Ray for Rhino license. So, while you do need a copy of Rhino wherever you run the Deadline Slave software, you don’t need that many Slaves; each Slave can use 10 machines that aren’t Slaves (and don’t need Rhino installed). You need at least 1 Rhino copy for each 10 machines in your farm, but you don’t need a Rhino copy on each machine.
Micha_cg: Yes. The idea is that there is a machine with Rhino running V-Ray for Rhino in Distributed Rendering mode talking to its DRSpawner clients that now, with the addition of Deadline, can receive/spool/queue incoming jobs, provide job status and notifications back to users, and keep statistics. Because the Deadline Slave can also execute jobs that are essentially Windows command line tasks, the Slave can do things like start and stop the DRSpawners on the client machines (as part of each job, if desired), implement Wake on LAN and powercfg for running in a power savings mode, or implement scheduled changes.
To your earlier question, Deadline does offer one type of job export -- it will transfer a job from one Deadline Repository to another. We have tested this within our organization to transfer certain jobs from our renderfarm (which is only running one Slave) to another unit’s renderfarm which is configured in the more traditional multiple Slave on dedicated Boxx machines configuration. They don’t run the distributed rendering architecture so a job can take longer but they do offer more Slaves. The question about whether we can export to an external (cloud?) system is one we have considered as we think about how we would expand capacity. If it can be done -- ie. if you can design a system that manually takes a group of files from one machine and copies them into a cloud-based rendering solution -- it can probably be implemented in Deadline because of the amount of Python code exposed to work with. We’ll probably look into this over the summer.
eazyc10: Let me know if you have questions and, certainly, how it turns out. The notes were a summary of what we’d found in working with this over about a two year period. I didn’t include all the things we didn’t end up doing, or changed our mind on after we tried them, but your situation may be different.