Maya 2016.5 with Vray 3.4 and Deadline problem

[SonyBoy]

This is a long shot but just in case anyone happens to be using Maya 2016.5 with Vray 3.4 and Deadline 8.017.

We recently upgraded to Maya 2016.5 and our previously working farm is now experiencing a lot of failed frames that we can't trace the cause of. We have other software running successfully on the farm (Nuke, C4D among others), so it seems to be a Maya and/or Vray problem.

Anyone have similar issues or is aware of any known problems?

[Morbid Angel]

I've not used 2016.5 a lot on the farm but when I did it worked fine for the most part. The errors might be related to the new / old render layer system as far as I remember, deadline 8 did not support the new render layers so you have to use the old ones.

[SonyBoy]

We are using the legacy render layers -- our assets pre-date the new Render Setup system anyway.

[Morbid Angel]

so then what kind of issues are you having can you provide a few examples?

[SonyBoy]

It's basically two problems 1) Render nodes picking up packets but immediately failing with a fatal error, and 2) Render nodes picking up packets and then just sitting there and not rendering until we notice and requeue the chunk. This does not seem to be limited to particular "bad" nodes and happens at random.

We are rendering scenes that have been created in older versions of Maya (probably 2016) which as usual are opening with errors (but seem to work and otherwise render fine), so it could be related to that. However I've also tried saving out a scene as an .ma file and fixing the header, but this didn't clear up the issue. Right now we're not sure what the problem is so just babysitting renders and requeueing where necessary.

[Morbid Angel]

Can you paste the last few lines of your deadline progress log from any machines which either failed or stuck? So basically some frames do get through the farm but not always?

[SonyBoy]

Unfortunately this is the problem -- the feedback isn't very helpful at all:

//////////////////////////

================================================== =====
Error
================================================== =====
Error: Monitored managed process "MayaBatch" has exited or been terminated.
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

================================================== =====
Type
================================================== =====
RenderPluginException

================================================== =====
Stack Trace
================================================== =====
at Deadline.Plugins.SandboxedPlugin.a(DeadlineMessage A_0)
at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

================================================== =====
Log
================================================== =====
2017-04-04 11:17:02: BEGIN - RENTAL008\setup
2017-04-04 11:17:02: 0: Executing plugin command of type 'Initialize Plugin'
2017-04-04 11:17:02: 0: INFO: Executing plugin script 'C:\Users\setup\AppData\Local\Thinkbox\Deadline8\s lave\rental008\plugins\58e3e2eaa8f9e538c4a0e269\Ma yaBatch.py'
2017-04-04 11:17:03: 0: INFO: About: Maya Batch Plugin for Deadline
2017-04-04 11:17:03: 0: INFO: The job's environment will be merged with the current environment before rendering
2017-04-04 11:17:03: 0: Done executing plugin command of type 'Initialize Plugin'
2017-04-04 11:17:03: 0: Start Job timeout is disabled.
2017-04-04 11:17:03: 0: Task timeout is disabled.
2017-04-04 11:17:03: 0: Loaded job: S020_V12_SD - PRO_UTILITY_PASSES - S020_RenderCam (58e3e2eaa8f9e538c4a0e269)
2017-04-04 11:17:03: 0: Skipping T: because it is already mapped
2017-04-04 11:17:03: 0: Executing plugin command of type 'Start Job'
2017-04-04 11:17:03: 0: INFO: Rendering with Maya Version 2016.5
2017-04-04 11:17:03: 0: INFO: Setting MAYA_DISABLE_CIP (ADSK Customer Involvement Program) environment variable to 1 for this session
2017-04-04 11:17:03: 0: INFO: Setting MAYA_DISABLE_CER (ADSK Customer Error Reporting) environment variable to 1 for this session
2017-04-04 11:17:03: 0: INFO: Setting MAYA_DISABLE_CLIC_IPM (ADSK In Product Messaging) environment variable to 1 for this session
2017-04-04 11:17:03: 0: INFO: Setting MAYA_ENABLE_LEGACY_RENDER_LAYERS environment variable to 1 for this session
2017-04-04 11:17:03: 0: INFO: Enforcing 64 bit build of Maya
2017-04-04 11:17:03: 0: INFO: Setting PYTHONHOME to C:\Program Files\Autodesk\Maya2016.5\Python for this session
2017-04-04 11:17:03: 0: INFO: Starting monitored managed process MayaBatch
2017-04-04 11:17:03: 0: INFO: Stdout Redirection Enabled: True
2017-04-04 11:17:03: 0: INFO: Stdout Handling Enabled: True
2017-04-04 11:17:03: 0: INFO: Popup Handling Enabled: True
2017-04-04 11:17:03: 0: INFO: QT Popup Handling Enabled: False
2017-04-04 11:17:03: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2017-04-04 11:17:03: 0: INFO: Using Process Tree: True
2017-04-04 11:17:03: 0: INFO: Hiding DOS Window: True
2017-04-04 11:17:03: 0: INFO: Creating New Console: False
2017-04-04 11:17:03: 0: INFO: Running as user: setup
2017-04-04 11:17:03: 0: INFO: Enforcing 64 bit build of Maya
2017-04-04 11:17:03: 0: INFO: Executable: "C:\Program Files\Autodesk\Maya2016.5\bin\MayaBatch.exe"
2017-04-04 11:17:03: 0: INFO: Argument: -prompt -proj "T:/PS_PROD/BEST_PLACE_TO_PLAY/BPTP_PSFAMILY_SIZZLE/CG/A_PROJ/MAYA"
2017-04-04 11:17:03: 0: INFO: Full Command: "C:\Program Files\Autodesk\Maya2016.5\bin\MayaBatch.exe" -prompt -proj "T:/PS_PROD/BEST_PLACE_TO_PLAY/BPTP_PSFAMILY_SIZZLE/CG/A_PROJ/MAYA"
2017-04-04 11:17:03: 0: INFO: Startup Directory: "C:\Program Files\Autodesk\Maya2016.5\bin"
2017-04-04 11:17:03: 0: INFO: Process Priority: BelowNormal
2017-04-04 11:17:03: 0: INFO: Process Affinity: default
2017-04-04 11:17:03: 0: INFO: Process is now running
2017-04-04 11:17:03: 0: Done executing plugin command of type 'Start Job'
2017-04-04 11:17:03: 0: Plugin rendering frame(s): 1133
2017-04-04 11:17:03: 0: Executing plugin command of type 'Render Task'
2017-04-04 11:17:03: 0: INFO: Waiting until maya is ready to go
2017-04-04 11:17:27: 0: STDOUT: Stack trace:
2017-04-04 11:17:27: 0: STDOUT: Fatal Error. Attempting to save in C:/Users/setup/AppData/Local/Temp/setup.20170404.1117.ma
2017-04-04 11:17:28: 0: WARNING: Monitored managed process MayaBatch is no longer running
2017-04-04 11:17:28: 0: Done executing plugin command of type 'Render Task'

[Morbid Angel]

Yeah this means it crashed right at the opening of the scene.

Can you try going to your local workstation directory via cmd C:\Program Files\Autodesk\Maya2016.5\bin\ and type mayabatch.exe -v, it will return build number such as:

C:\Program Files\Autodesk\Maya2016.5\bin>mayabatch.exe -v
Maya 2016 Extension 2 SP1, Cut Number 201605191025

then go to your render slave and do the same, the numbers should match, if they don't something is off on maya install...

Also I would suggest making sure vray is deployed properly with the same build number for workstation / slave.

[SonyBoy]

This will have to be a job for our IT guy, but I'll forward the info. Thanks.