What render controller do you use?

[gfmarsh]

Hello. I am looking around for options into render controller software. I currently use Deadline with Maya 2017, but I find the system to be extremely finicky and fussy most of the time, where it seems to reject files that otherwise render fine locally or through distributed rendering. These are not file path errors, or network drive errors as they seem to be related to structures in the file itself that Deadline has issues with. I get issues like frames that get completed but are not, equivalent nodes that won't render at all where as others do. I know there are other companies that make render controller systems but I am not finding much on which ones are good. Like the interface of Deadline, I just can't stand how apparently fussy it is to use for anything but the most basic of files. I have gone through V-Ray support and they have been really great to work with, but they too have run out of answers about Deadline in my case.

[Morbid Angel]

Sorry to hear for your troubles. We use deadline and 95% of the time it works great. What kind of issues in particular do you have, when you say it can't deal with file structure what do you mean? Do you have some log examples of what happens?
About other controllers, I don't know if there are any better ones which work out of the box. Lots of them require you to write submitters, extensions etc. I've used "rush" for some years and it was great but required lots of scripting to get it going.

[yolov]

Like Dmitry said, if there's any specific issues that can be tied to V-Ray, we should definitely try and work them out. Any logs and/or scenes are helpful, along with any additional information that could help us pin down potential issues.

Are you having trouble with particular scenes? Do you have issues with some of your projects, but not with others? When you get an issue - what is it exactly - does it crash, do assets go missing. I'm not familiar with your discussions with our support team, but if you PM me the ticket numbers you have opened with us, I could have a look.

[gfmarsh]

Thanks for your replies...

It's sort of a long story, but we are working with files that are Xgen based so I am not sure how stable that workflow is with Deadline. It did work relatively well before, but not without hiccups as I recall. I really want to thank Mihail from Chaos Group as he has been awesome getting things fixed as best he could with the live support he did twice. My workstation seems to have issues staying visible on the network under an assigned static IPs so we switched over to a Reserved DHCP for everything. That alone took awhile to do as there are 10 nodes.

Mihail indicated that he checked the scene file on their system and it rendered correctly, at least that what I think he said from our chat. I can send it again to be sure. This particular file once had an Xgen character in it but the the character was deleted in order to render additional plant elements. It seems to render this particular file (without XGen) is okay when you switch off the Mayabatch check box, although this is not an acceptable solution as sometimes Xgen (if you need it) will get left out and the render time doubles. I have encountered this before again where it fails to render but if you send it to Distributed Render or render it locally it will render fine. I thought initially that cleaning up the file of old Renderman nodes (present from a co-workers machine) was making a difference but I am not totally convinced although it SEEMS to run into less problems if I do clean the file. In any case it will skip the first round frames in both cleaned and uncleaned files and state that they are completed. The uncleaned file will eventually start skipping frames and report them as completed. I have two particular nodes that don't want to render this file at all. All this seems to be file specific to most of the files in the project I have but I will check more of them. It could be related to the reference file, but I always import it before I send it to render. It's important to note that if I setup a simple scene with Vray dome light globally reflecting a texture to the same project network folder all of my machines render no problem. It could be more file specific as I will be rendering more as I test. I had suggested to Mihail that perhaps I should look at setting the host up on a separate workstation that only runs as a host, one that doesn't have network problems. Whatever the issue is, there doesn't appear to be a simple fix.

The nodes that skip (fail) that give a completion signal on Deadline display this in the log:

2018-03-22 18:08:43: 0: Start Job timeout is disabled.
2018-03-22 18:08:43: 0: Task timeout is disabled.
2018-03-22 18:08:43: 0: Plugin rendering frame(s): 141
2018-03-22 18:08:44: 0: Executing plugin command of type 'Render Task'
2018-03-22 18:08:44: 0: INFO: Waiting until maya is ready to go
2018-03-22 18:08:44: 0: STDOUT: mel: READY FOR INPUT
2018-03-22 18:08:45: 0: INFO: >This is a Render Job
2018-03-22 18:08:45: 0: INFO: Rendering with vray
2018-03-22 18:08:45: 0: INFO: Rendering to network drive
2018-03-22 18:08:45: 0: INFO: Creating melscript to execute render
2018-03-22 18:08:45: 0: INFO: Auto memory detection for VRay disabled
2018-03-22 18:08:45: 0: INFO: Executing script: C:\Users\gfmar\AppData\Local\Temp\tmpF61.tmp
2018-03-22 18:08:45: 0: INFO: Waiting for script to finish
2018-03-22 18:08:45: 0: STDOUT: mel: Loading scene: V:/3d/DH_Shots/DawnHorse01/scenes/Rail_Transformation_08R.ma
2018-03-22 18:08:45: 0: STDOUT: Error: file: C:/Users/gfmar/AppData/Local/Temp/tmpF61.tmp line 47: Cannot load scene "V:/3d/DH_Shots/DawnHorse01/scenes/Rail_Transformation_08R.ma". Please check the scene path, then try opening the scene on the machine which ran this job to troubleshoot the problem.
2018-03-22 18:08:45: 0: Done executing plugin command of type 'Render Task'


Sound confusing? But that's about the long and the short of it. I've been using Maya and Vray for about 7 months now with Deadline for about 4 months, so there's been a bit of a learning curve.

Thanks,
Greg

[Morbid Angel]

Yeah I'd have to take a look at that scene, additionally full log. The issue based on your description as I understand is following: deadline with terminate a frame if it encounters a certain condition like render man node (which would be unknown node) I had this happen before. If it completes the frame, it really means that it did not know what to do with it. The present message would mean that it could not load the maya file under mayabatch mode for whatever reason, and there are several reasons why this happens. I had case where I get a scene from a Mac maya, and it would be referenced into my scene. Working locally opens/renders fine. Submitting to deadline produces this exact error. Reason being, some sort of compatibility issue between mac and win. So solution would be to open initial mac scene and save over it under windows maya.

Regarding xgen, its not very well implemented in maya to begin with. So unless you cache the xgen description it will not render even if the fur is not dynamic. There maybe several issues at once happening to your scene and this is the reason why you should provide a scene for us to take a look at. Also, what windows and deadline version are you using?

[gfmarsh]

Hi Dmitry,

Thanks for your reply. That's interesting that you have had had issues with Renderman nodes as well. The Xgen was working before with the odd hiccup with the Renderman garbage in the files. I do export patches, and have groom baked the descriptions. I am not a network expert, but I am wondering if this is something much more basic ie network. The problems sort of stem back two weeks ago to when I added a 4 node server to the renderfarm running Windows Server 2012 R2. I accidentally installed V-Ray Standalone instead of V-Ray for Maya 2017. Which I then corrected. Anyway, I replaced two switches that ran 6 nodes together including the host workstation with a brand new 16-Port Gigabit Switch Netgear Prosafe switch. The system seems correctly wired, otherwise I wouldn't connect at all. I am using CAT5 cable throughout and I can't see CAT 6 being needed, but one thing Mihail mentioned that he thought the network was bottlenecking at times. Tonight I installed the old switches and go the same results (even tried wiring my workstation directly to the router), so there goes that theory out the door. I have tried to run files that didn't have Xgen (although are clouded with Renderman nodes) and this issue is widespread now as they won't render properly either. I am really stumped but I do think it perhaps point to maybe a networking issue with my workstation, but I am willing to believe in anything including Contrails causing the issue at this point. I can send you some file but they like won't show you much. Let me know where I can send as they are copyright protected. Thanks

[Morbid Angel]

gfnarsh, please email it to silhouetteimages@gmail.com we can chat from there. I doubt its the network switch (network related).

[gfmarsh]

Hi Dmitry,

I will send you a file. Thanks.

With exception of one of the server nodes (all of the new server nodes that previously ran fairly well although skipped) I can render the same scene in 4K (this one is non Xgen) through Distributed Rendering in Vray which I am doing now as I've given up using Deadline for this project, unless some miracle cure is discovered. I have spent a week and half trying to get this sorted. All my previous testing on Deadline were at 2K or lower. Most of my nodes are fairly highend. The Deadline techs are curious about this as well.

I am not sure why this one server node won't engage with DR, but at least I can render with DR Vray though with the rest of them. Thank god.

I was calculating my licenses. So theoretically since I have 10 nodes + 1 workstation, I could render with 11 nodes with DR and still work with my workstation doing IPR, etc?

Thanks
Greg

[gfmarsh]

Thanks for looking into this Dmitry and for all your help. Upgrading the versions did not seem to help in this instance, and for whatever reason caused loading of the Xgen to seemingly trip up on some file errors that have been with this file for quite sometime (although they did render before with the errors through Deadline). Maya 2017.5 and 2018.2 would take a very long time to load the file in the viewport render IPR as they seem to hang up on some XUV file asset errors. I switched back to Maya 2017 and installed 2016 which handled my file much better (with the errors). One thing I noticed was that Vray 3.60.04 would not render the Xgen fur anymore, but I don't know if this is error related or not. It wasn't until I switched back to 3.60.03 that the fur would render.

I also got some help today with a some techs and developers at Deadline. They noticed an error node on another test file that once was removed began rendering properly without the skipping of frames issue. Since these errors are somewhat minor, as the project would batch render locally and in DR, they wrote a patch for Deadline that would actually cause it to ignore these types of errors and push the job through. So there, I take back my doubts about Deadline, as they were good at finding the problem and providing a solution to files that have become somewhat sloppy with all the different people working on it in different locations. Says something about the value of shared central project resources I guess. The whole thing was pretty confusing as I went, but it makes much more sense to me about the importance of getting rid of these errors in the file path. We still have some .XUV errors we don't know how to delete as those Xgen items aren't used anymore but I have asked Autodesk to tell me what to do. Just happy I have a solution.

Attached is the patch for the Deadline Repository Mayabatch Plugin (v10.0.12.1) which makes Deadline ignore errors and got rid of my skipping of frame or false completion of frames. It should go in the in the Repository folder under Plugins/Mayabatch where you replace the one that is there. Also this other mel script will detect missing assets in your file and list them in the script editor. Somewhat similar to the File Path Editor I suppose, but everything is one one list.


DETECT MISSING NODES, FILES:

string $fileNodes[] = `ls -type file`;

for( $fileNode in $fileNodes )
{
string $fileAttr = $fileNode + ".fileTextureName";
string $curFile = `getAttr $fileAttr`;
print($fileNode + " - " + $curFile + "\n");
}

[shaio]

Hi,
sorry for the delay. The reason for the fur not showing is because you need to do "File > Export Patches for Batch Render" in the xGen window for version 3.60.04. It will export the abc files and then the fur will render. This wasn't necessary with 3.60.03 and that is the reason why it's rendering fine there. The issue is known and our devs will be working on it. I'll get back to you as soon as it I have some more information.

[gfmarsh]

No. That doesn't seem to be it. It still won't render in the Viewport IPR after doing the patches. Normally you only have to do that when you batch render anyhow, not for the viewport.

[shaio]

Thanks for the feedback, I'll double check it and let you know if there is new information.

[shaio]

I've double checked and the fur renders fine if you Export the patches for batch in 3.60.04 and save the scene. I've also exported the collections and descriptions again just in case and again saved the scene. I know that the Export Patches for Batch Render is only used for batch rendering but we broke it in 3.60.04 as I mentioned in my previous post and is in our to do list for fixing. Sorry about this.
If you still aren't able to render it in IPR/Viewport IPR after those steps please give me some more info for the steps you use before starting the render in Viewport IPR so I can try to help you.

[shaio]

Hi,
the issue is fixed in the latest stable V-Ray for Maya 3.60.05. There is no need to Export patches for batch after any change. Let me know if you have any issues.

[stezza]

RenderPal, is great!
Used for over 10 years nows never had an issue.