Render managing with DR: Vray.exe query + pre-render/post-render python

[Fredrik Averpil]

Hi,

Our current scenario:
- Our Current render manager handles job queues and priorities + notifies upon completion etc.
- V-Ray for Maya DR is completely standalone, does not talk at all with the render manager.

This causes us to, in order to run a DR render on all our machines, go into our render manager, pause all current and queued jobs and tell all artists not to submit any new jobs to the farm – all this to prevent the render manager to start parallell jobs side by side with the DR job.


Here's a wish list scenario:
- Vray.exe takes query commands, and can return values such as "status: idle" or "status: busy/rendering".
- Upon a DR initiation, vray.exe can read a pre-render a user-defined and a post-render python script, if it exists.

In this scenario, we could make Maya - V-Ray - Render manager to communicate, extremely rudimentary, using Python and prevent running parallell jobs.

My question: Is this already possible to do today, or is any new features required for this to work?
If this does not work today, I think a lot of people would benefit and want this feature high up on the wish list of new features – at least, I know I do

[pechart]

Just stumbled upon this and as I'm currently working on the same issue i'd like to give the possibility to query the server a +1!

[chuckie7413]

Also very interested on this.

We have to do the same thing in regards to making sure no other renders are happening when wanting to do DR rendering.

Cheers,

Richard

[paulDryzen]

Just create a custom job on your renderfarm which is starting the slave.
Do not start the slave by default to make this work.

Thats how I do it to be able to control which machines are blocked for DR rendering.

Paul

[pechart]

Good idea. Should work for batch jobs. But what about clients used for local DR jobs, started by the artists workstation?

[paulDryzen]

My workflow is like this.

Artist enables DR in Maya.
Artist sends a job to the renderfarm for a spcific renderslave like rendernode10
Artist uses the rendernode10 for DR rendern
After Dr rendering is done Artist kills the job from the renderfarm and rendernode10 is free again for the normal rendering of jobs.

For DR rendering during batch I don´t have a sollution because I don´t use this feature. For batch I only use one machine per frame. This is faster than blocking more machines for just one frame because a lot of time is wasted for data transfer etc.
For highres stillframes I use the method above.

Hope this helps...

Paul

[pechart]

Thanks for your suggestions.

Maybe i want a little bit too much here But we recently had a job for which several high res (8kx6k) stills had to be rendered. And of course we'd like to submit them to the farm on the weekend. Our farm is not that big, so we have to work out a solution that handles those projects. But we'll see if we can work something out. Or we switch to another workflow, maybe.

[t.petrov]

Hi,

I'm not sure I understand what is the exact problem here.
What I've comprehended from you posts is that you want to share
your render nodes between a render farm manager and the artist's
workstations. Am I correct?

/Teodor

[pechart]

Yes, that's the plan. But for that I need to query the status of the vray renderslave in some way, so that the renderfarm manager can say 'Nope, this machine is already in use!', when an artist is using that rendernode. So no other jobs (not only vray, also mr, nuke, afx, etc.) will be pushed to that machine.
But for this the renderfarm needs to get the status of the machine. Maybe it's possible in other ways, but I thought a query would be nice. Or maybe I'm just thinking to complicated .

[Fredrik Averpil]

Hi,

I'm not sure I understand what is the exact problem here.
What I've comprehended from you posts is that you want to share
your render nodes between a render farm manager and the artist's
workstations. Am I correct?

/Teodor

First off, I'd like to emphasize that DR is an absolutely fantastic feature, and a life saver, not to be seen on most other render packages. Unfortunately, it's current implementation hinders us from using it the way we all really would use it – if we could.

The problem: You cannot have "regular" farm jobs being on queue on the render farm mixed together with DR jobs (since you will end up with nodes rendering DR at the same time as a regular job and you'll run out of RAM etc). This is a crucial feature which could determine whether a deadline is met, whether money will be on the bank and is ultimately the biggest issue we can deal with; a broken render farm.
The reason behind this: The DR server is talking directly to the DR clients. It doesn't communicate with 3rd party render management software in any way.
The unfortunate workaround, which we all have to live with today:: Tell everyone in your studio you need them to stop using the render farm (or certain nodes such as workstations), since you need to perform a DR render on them.
The unaffordable workaround, which may only work in theory: Have a dedicated V-Ray farm. This isn't financially viable for us, as we need to be able to throw different jobs onto all machines we have, right about 24/7.

You would have to go and build a house of cards worth of scripts (even wrapping DR in your in-house launcher) in order to send the appropriate error codes around today, in the current state of DR, which in our studio we've found not to be worth doing since we can't guarantee the DR's behavior won't change down the line (we don't have control over the source code etc).

A constructive proposal to try solving this:
If the DR server could send customized commands, it would be able to submit a faux render job to the specific node (think of it as a python script just performing a loop which is not intensive on the cpu or memory) and the render node would become "busy/rendering". Whenever the DR server releases the node, it will also change an environment variable which the faux render job (python script) reads and exits the loop, returns a code which indicates that the render was completed and sets the node free to take on another job.

If the DR client could act (optionally) like mayabatch.exe/nuke.exe etc it would be even simpler to set DR up to work parallel to "regular" jobs etc;
- load up
- attempt to receive a render (or timeout and exit after X minute/s)
- accept render from DR server and commence crunching
- return an error code when finished: error/canceled/completed
- exit

From our point of view, it would also be very important to make DR work from an arbitrary install location. But now we're getting complicated...

[t.petrov]

Fredrik:
Have you tried something like:

Create a V-Ray launcher script (mel/python) in Maya. Probably you can use the pre-render callback, too.
When the script is executed it:
- queries the Render farm for free machines for DR.
- fills the list of IPs in the DR settings of V-Ray
- starts a V-Ray render inside Maya.
- when the render finishes the script tells the Render farm that the machines are free to use for other computations

[Fredrik Averpil]

- queries the Render farm for free machines for DR.

Ok, we can do that. But what if only one machine is free right now (all the other machines are finishing the previous job and are soon ready to begin crunching this DR job)?

- fills the list of IPs in the DR settings of V-Ray

Ok, we can do that. Worst case though, no machines are filled in except the one starting up Maya which won't really do us any good because then it won't be a "DR job".

- starts a V-Ray render inside Maya.

Ok, we can do that. So this means one machine on the farm will look like it's performing a Maya render job. But what if more machines were ready for render?
You mean I should disable them in the render farm manager (meaning taking them offline) via the script?

- when the render finishes the script tells the Render farm that the machines are free to use for other computations

Ok, we can do that too. This means I will now enable the render nodes again (take them online).


There are some flaws to the above:
1. In between many jobs running on the farm, at one point, there will most likely just be one single computer to start crunching on a new (potentially a DR) job. This means that in DR's current shape and form, there won't be any DR really.
2. How can an artist cancel a running DR job (which in this scenario is running on many machines)?

Again, just to not sound like I am all negative DR is a life-saver, but it can be improved and become even more valuable.

[t.petrov]

There are some flaws to the above:
1. In between many jobs running on the farm, at one point, there will most likely just be one single computer to start crunching on a new (potentially a DR) job. This means that in DR's current shape and form, there won't be any DR really.

I'm not sure I understand this.

2. How can an artist cancel a running DR job (which in this scenario is running on many machines)?

Stopping the render from inside Maya, stops the rendering on all DR server machines (vray.exe is running, but should do nothing).

Again, just to not sound like I am all negative DR is a life-saver, but it can be improved and become even more valuable.

Don't worry, we value user feedback very much, because it help us improve V-Ray.

Currently I'm testing some possible workflows with DR and I'll post back, when I've finished my investigation. (if I don't post back in couple of days, don't hesitate to bother/urge me).

/Teodor

[t.petrov]

I've finished my tests.

What I've found is that:
DR servers started after the DR rendering has begun, should automatically join the DR rendering automatically.
This should work with older V-Ray for Maya releases if the workstation OS is Windows.
On Linux and OSX it should work with revision greater or equal to 18764.

So you can add all possible IPs in the DR Settings in Maya and then start as many nodes as you have available.
If another node gets idle you can start the V-Ray standalone on it and it will automatically join the DR.
After the DR Rendering has finished, you can kill all the V-Ray standalone processes on the nodes. And start
again, when new rendering is started.
Unfortunately this setup won't work if you have two workstations starting DR Renderings, because the nodes will
participate at random in the two renderings.

Currently in 3dsMax we have an option to limit the number of nodes used, so you can add all the node IPs, but
V-Ray will use only N of them. I've logged an issue in our tracker to implement the same option in Maya, too.
It is relatively easy issue to implement, so we can add it very quickly.

I'll log another issue about implementing an executable which can query the status of V-Ray standalone running
in server mode. This one is relatively harder to do and it will take more time.
Can you tell me what kind of information do you expect from such a status check?

Unfortunately at the moment it turns out that you cannot add new nodes to active rendering dynamically.
There is another option: We can add an API inside Maya, so you can add IPs to the list of the DR hosts during
rendering, but I'm not sure how easy it will be. Probably it will be easier than making the status query tool.

/Teodor