(No Change Required, Closed) [HOU-1693] [HOU-1673] Batch rendering workflow needs to be fixed

[corestudio]

Basically we should be able to automatically use Houdini engines and have vray render with standalone without having to create .vrscene files. In anything complex, it takes longer to generate the .vrscene files than the actual render. On short sequences it's not big deal, but we have a 2500 frame sequence that's close to unrenderable. We've stripped out most of the scene and are rendering it with redshift, which just works, but the elements that need to be v-ray are slowing things to a crawl. It's so painful. As it stands we just have to take up Houdini GUI licenses and split the shot between machines, which totally sucks.

I don't mind the option for those that want it, but it shouldn't be the only way.

Please change this behavior. It's an uphill battle against Redshift as it is, and having to take this step is not helping. Like at all.

[Gosho.Genchev]

Hello,

if it is the case that GUI licenses are used instead of Engine ones, we'll definitely be looking into this. As far as I know, we are not doing anything special to force such behavior. We'll check with SideFX right away.

The VRScene export should not be slower than the render itself under any circumstances. What V-Ray build are you using ? A month or so ago we found a way to optimize fragments export - VRScenes were getting quite large before this. Another possibility is VDBs in the scene - we bake them if you change the voxel information after loading them from disk, or if those were never baked - there's no way around this. I can't think of any other reasons to cause this - any help to reproduce this so it can be fixed would be much appreciated.

All the best!

[corestudio]

Thanks for the reply. So you mean I should just be able to do a normal Deadline submit and have it work? My memory is that just doing a Render submit (vs Export and Render) didn't work, therefore we had to have the GUI running. That's what I meant. We're using a nightly from a couple days ago.

[Gosho.Genchev]

If we are talking about Deadline in particular, the support team received a question on this a few days ago and I gave them a rather lengthy explanation. I will paste it here, and I apologize if you were the one to query them and you're receiving the same response:

The old workflow using a jobPreLoad python script described here (https://docs.chaosgroup.com/display/...line+and+V-Ray) was necessary because the Deadline ROP node in Houdini had no support for V-Ray.
Submitting jobs from Houdini was basically impossible at that point.
When submitting as described in the page above, the hip file is opened with Hython (houdini's python distribution) - this consumes a Houdini or Hbatch license for every single slave on the network
that is being used during rendering. The jobPreLoad was used to append the required environment variables so when Hython is called, V-Ray is also loaded.
This essentially means that any time you render a hip file this way, you will need both an Hbatch and a V-Ray Render Node license.
It is essentially the same as opening a hip file on your local machine (through hython or hbatch)* and hitting the render button.

We added the V-Ray Deadline ROP node to allow the submission of VRScenes.
It works in one of three ways:
1. Export Job / Render job - this will submit the hip file to the farm, Hython will be called on one of the slave machines to export a VRScene, and a V-Ray Standalone job will be triggered to render the VRScene:
-> Benefits: The export of VRScenes takes, in general, significantly less time than rendering. An Hbatch license will be used only during the export of the VRScene. The V-Ray Renderering is a separate job so the Hbatch license will be returned by the slave when the VRScene is exported.
-> Drawbacks: you will still need additional Hbatch/Houdini licenses. Those will be quickly returned after VRScene export, though.
2. Export Locally / Render job - this will export the VRScene from the artist's machine and submit it to Deadline. A V-Ray Standalone job will be triggered to render the VRScene:
-> Benefits: no additional Hbatch/Houdini licenses are required for the farm. Only a V-Ray Render Node license is used by the slaves
-> Drawbacks: the artist machine is doing the VRScene export. If that's a heavy scene over a large frame range, it could take a while
3. Render job - this is the same as using the jobPreLoad script as described at the start. The hip file is submitted, hython is called, V-Ray is used to render the scene.
-> Benefits: does not depend on VRScene export
-> Drawbacks: consumes both Hbatch and V-Ray Render Node licenses.

I hope this clarifies how things are supposed to work - if any of this is behaving differently for you, please inform us so we know to check.

[corestudio]

Yeah that was me The thing is we're ONLY able to get Export Locally/Render to work. Doing Render or Export/Render doesn't. I'll post the errors we get in the Deadline log. So if I just wanted to go around deadline, is it possible to just use the Command line to render?

[Gosho.Genchev]

Yes, you have the option of using hbatch or hython. For the purpose of rendering a hip file which is already set up and you're not looking to modify it further with code, both will give you the same: an hbatch/engine license will be consumed to run the process, and a V-Ray Render Node license will be consumed to render the scene.
Both of those processes run the same as Houdini so your setup needs to allow them to find V-Ray - whether you're using the json file in the packages folder of Houdini, or the environment file in the documents folder makes no difference.

Here's a video of doing this with hython: https://drive.google.com/file/d/13Ba...ew?usp=sharing

As to the V-Ray Deadline ROP - please do share the logs with us if possible. We've been getting feedback from others as well and this is the first I hear of those modes not working - it is possible a certain type of setup is not covered by the code.

Best regards!

[corestudio]

I have three logs. All submitted to Deadline via the vrayDeadline ROP. All submitted to the same machine (in fact it's the workstation that it's being submitted from). Only the Export Locally / Render Job results in frames on disk. Can I email you directly so as to not post the logs here? They have client/job name in them. Thanks.

[Gosho.Genchev]

Yes, please do send them over e-mail!

Thank you!

[corestudio]

ok I've emailed them to you

[corestudio]

thanks for the link to the hython workflow. It works! But I feel like I'm back in the 90's rendering my Maya scenes with mentalray I suppose this is what the Render setting in the V-Ray Deadline ROP is supposed to be doing, but it's just not working for us as the logs I sent show. I'd happily add Engine licenses if it worked.

Does V-Ray work ok with HQueue? I'm tempted to set it up just to push this shot through the door.

Yes, you have the option of using hbatch or hython. For the purpose of rendering a hip file which is already set up and you're not looking to modify it further with code, both will give you the same: an hbatch/engine license will be consumed to run the process, and a V-Ray Render Node license will be consumed to render the scene.
Both of those processes run the same as Houdini so your setup needs to allow them to find V-Ray - whether you're using the json file in the packages folder of Houdini, or the environment file in the documents folder makes no difference.

Here's a video of doing this with hython: https://drive.google.com/file/d/13Ba...ew?usp=sharing

As to the V-Ray Deadline ROP - please do share the logs with us if possible. We've been getting feedback from others as well and this is the first I hear of those modes not working - it is possible a certain type of setup is not covered by the code.

Best regards!

[Gosho.Genchev]

Yes, it does but requires modifying the python files provided by SideFX. We have a docs page on this here: https://docs.chaosgroup.com/display/...ueue+and+V-Ray

[Gosho.Genchev]

Hey, corestudio ,

looks like I'm going to need more information. Those errors in the log are not generated by our code at all - I suspect the slave machines are unable to find V-Ray.
How is your environment set up ? Each slave should be able to find the V-Ray files - either using the packages/json file or an environment file in the documents/houdini folder.

I've configured the following things:

1. Monitor -> Tools -> Configure Plugins -> Houdini -> Hython 17.5 executable to point to the Houdini 17.5.460 installation folder/bin/hython.exe
2. Monitor -> Tools -> Configure Plugins -> V-Ray -> V-Ray Executable to point to the V-Ray for Houdini folder /appsdk/bin/vray.exe
3. I've installed Houdini 17.5.460 in the same place for all slave machines
4. I've placed the V-Ray for Houdini folder in the same place for all slave machines
5. I've added the attached environment files in the documents/houdini folder for all salve machines


As to the licenses - after a job is submitted, hython is called which correctly consumes a Houdini Engine license, and correctly releases it once the VRScene is exported. Screenshot:

[corestudio]

Thanks I'll take a look. Pretty sure we've set it all like that, perhaps a conflict with Redshift somehow (which is working and checking out engines). One question though, we're using 18 not 17.5. Have you tested with 18?

[Gosho.Genchev]

I'll check tomorrow, tried with H18 first but realized my Deadline has no support for it, googled and found a thread with recent replies which left me with the impression it's not yet supported.

If you guys are using it, I assume that's not the case. Will update Deadline tomorrow and come back with info.

All the best!

[Gosho.Genchev]

Hey corestudio ,

I grabbed the latest Deadline and tested with Houdini 18.0.348 - initially thought everything is fine there as well but decided to render some heavier scenes and hit the same issue as you - it's not related to Houdini or Deadline version - Deadline is failing the jobs because of export errors when generating the VRScenes. If you export a scene locally and see an error message in the console, submitting the hip file will not work. We've contacted Deadline for a potential solution - the script that executes the rendering itself is part of the Deadline distribution and from the looks of it, there isn't much we can do to remedy the problem on our end - the code there is simply stating "fail the job if there's errors in the log".

My suggestion would be to enable Renderer ROP -> Options -> UI -> Log Window and check for text highlighted in red before submitting any jobs. Can you please check if your failing jobs throw any errors in the log window ?

I've logged the problem as HOU-1693 and will inform you once there's progress on this.

All the best!